Hydrogenaudio Forums

Hydrogenaudio Forum => Listening Tests => Topic started by: guruboolez on 21 August, 2005, 02:33:47 PM

Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 21 August, 2005, 02:33:47 PM
[span style='font-size:14pt;line-height:100%']Preliminary notes[/span]


Last summer I performed a blind listening comparison (http://www.hydrogenaudio.org/forums/index.php?showtopic=23355) between three different audio formats, all set for ~175 kbps encodings. The purpose of the test was to investigate about encoding quality with classical music (and only classical) and to see which format would be the most efficient (i.e. the closest to transparency at lowest bitrate possible) for this kind of music. As jumping-off place for bitrate I took MPC –standard preset which was indisputably recognized as the best encoding solution outputing at 175...190 kbps on average. And indeed, the test ended on musepack superiority. MPC was even superior to Vorbis and MP3 at presets presenting higher bitrate (~195 kbps for LAME, ~185 for Vorbis against ~175 for musepack). Consequently, MPC encodings appeared to sound better and to be smaller at the same time. Amazed by the existent gap between all contenders I conclude my specific test with these words: “I didn’t think that MPC –standard was so in advance”.

My vacation are now quite over. I performed during my free time a big checkup of lossy quality at 80 kbps (http://www.hydrogenaudio.org/forums/index.php?showtopic=35438) and 96 kbps (this one has to be translated in english  ), and it’s too late now to complete the 128 kbps I planned to do in the same silly conditions (150+35 tested samples). But I used my small remaining time to do again the listening test at 175 kbps I did last year, with the same 18 samples and the same hardware.


[span style='font-size:14pt;line-height:100%']Why doing again the same test?[/span]


As a result of constant evolution of most audio encoders I consider my previous results as really outdated. I recall that Vorbis encodings were done with MEGAMIX I (hybrid encoder melting aoTuV beta 2, Garf Tuned 2 and Quantum Knot tunings). This encoder didn't subsist for a long time... and doesn’t exist anymore; it was replaced by MEGAMIX II, then official 1.1 with Impulse Trigger Profile + Impulse Noisetune switches, which was finally followed by aoTuV beta 3 and beta 4. The same goes for LAME: 3.97 alpha 3 was tested, and during this time LAME developers have submitted eight new versions of this alpha and a few other ones (lame_may, lame_june...)! MPC has also changed: from 1.14 beta to 1.15 alpha which is now considered as safe to use.
As a consequence of this evolution, problems audible last years (kind of ringing for LAME, or noise and coarseness for Vorbis) may be corrected or at least be lowered. The first purpose of my test is therefore to check the outcomes of recent tunings done for high bitrate settings.

There’s also a second point which stimulated me to do again the test and this point is called AAC. I haven’t tested AAC last year for technical and moral reasons. Technically, iTunes encoder couldn’t be set to ~175 kbps; Apple's AAC encoder wasn't also gapless and is for my purpose unsuitable for my conception of artefact-free encodings. I also felt as dishonest the inclusion of Nero AAC: it had recognized issues with classical first and a new encoder supposed to solve these problems was announced as imminent. Some readers suggest me to include faac as competitor, but I felt as unfair to test an encoder which was probably not the state of the art of AAC format and to compare it to the most advanced implementation of other formats (MEGAMIX and LAME 3.97).
I never regret my choice. But this absence of AAC frustrated my curiosity for a long time, because I had strictly no idea about comparative performance of this format with other contenders. That’s why I decided to absolutely include AAC this time. WMAPro will also be tested this time if possible.

The purpose of my test is therefore to obtain a fresh photography of the current performance of all modern lossy formats with classical music using the most advanced implementations for each of them.


[span style='font-size:14pt;line-height:100%']I. Choosing the encoders[/span]


My purpose being to test most advanced encoders the choice of format hasn't to be controversial for most of them:

MP3: LAME 3.97 alpha 11. Release date: July 2005. Note: --vbr-new encoding mode.
MPC: mppenc 1.15v. Release date: march 2005.
Vorbis: aoTuV beta 4. Release date: June 2005, updated in July 2005 (merged with SVN 1.1.1).
WMAPro: no choice here: it's 9.1 or nothing. Release date: during 2004.

Choosing the good AAC encoder is much harder:
Apple AAC: There's still no VBR mode with iTunes. Consequently it's currently impossible to use Apple's AAC encoder unless other contenders will output an average bitrate close to either 160 kbps or 192 kbps. It's unlikely...
Nero Digital AAC: the most advanced VBR AAC encoder and therefore better placed to represent the AAC format. Big problem: should I use the 'high' and defaulted encoder or rather the 'fast' one which is really better at lower bitrate with classical music? The first one is still recommended by all Nero's developers and it's a valid reason to choose it instead of something they don't consider as stable enough (Garf (http://www.hydrogenaudio.org/forums/index.php?showtopic=32370&view=findpost&p=282207),  JohnV (http://www.hydrogenaudio.org/forums/index.php?showtopic=29925&view=findpost&p=258778) and Ivan Dimokovic (http://www.hydrogenaudio.org/forums/index.php?showtopic=29925&view=findpost&p=258894)). But the situation has maybe changed since their recommendation; I wouldn't also discard too quickly the possibility of using an encoder working better for the difficulties proper to the musical genre I'll test. The debate could be endless if a trivial but objective argument hadn't close the debate: the average bitrate of VBR mode of both encoders (see below).
faac AAC: testing faac might also be interesting. And even for fun, it would give me the possibility to oppose four different open-source implementations of four different formats  But such friendly comparison has a price: increasing the onerousness of the test which is anything but easy at this bitrate...


[span style='font-size:14pt;line-height:100%']II. Targeting a bitrate[/span]


The purpose of my test is not to see what encoders could do with xxx kbps for each sample; I don't plan to force each encoding reaching a precise bitrate. My purpose is to stay close to the real usage of a vast majority of listeners (if not all...): using for every encoding one fixed setting which should statistically corresponds on average to the desired bitrate. That's why it's really fundamental to precisely know the average bitrate corresponding to a defined preset. And there's only one way to get it: encoding several tracks or albums.
Last year, I used as reference ~20 classical (+3 non-classical) albums. This year, I decided to be more methodical. I’m now using 150 different tracks (I mean full tracks) coming from 150 different CD in order to increase the variety of encoded tracks. It’s important to note that I didn’t choose randomly those tracks. I meticulously worked to get a representative microcosm of my full classical library, balanced between different grand ensemble (vocal, orchestral, chamber, soloist recording). This collection is nothing more than the 150 full tracks from which I’ve extracted 150 short samples in order to build a “catalogue raisonné” of musical situations occurring with classical music (see this test (http://%5bURL=http://www.hydrogenaudio.org/forums/index.php?showtopic=35438)).

I genuinely expect from this methodically constructed library to be a highly representative panel of my classical collection. My assumption could be verified by checking the average bitrate of the entire collection encoded with WavPack -fx5 (all my >1000 CD digital library is encoded with this preset): 642 kbps for the selection of 150 tracks against 635 kbps for a complete set of more than 15000 tracks. The deviation is inferior to 1%!  Statistics are really magical.


[span style='font-size:14pt;line-height:100%']III. Observing bitrates[/span]


I started with MPC which must give the reference bitrate. All other competitors have to be set in order to get a similar value.


MPC: --quality 5 corresponds precisely to 184,54 kbps. This is higher from what I expected first (~175 kbps). The 150 reference tracks are maybe not as representative as supposed. I also tried 1.14 (used last year) with the same preset and --xlevel: 176,28 kbps, much closer to the native average bitrate of --standard profile and reassuring me about the representativity of my collection of tracks. The bitrate has therefore inflated by 4.7% from 1.14 to 1.15v with classical.
=> I'll therefore try to get from all other encoders a setting which outputs to [span style='font-size:14pt;line-height:100%']184,5 kbps ±2% (180,5...188,1 kbps).[/u][/b][/span]

MP3: I first tried -V2 --vbr-new, which corresponds to the former --preset fast standard. Average bitrate is 181,79 kbps. Now, this value is lower from what I estimated last year (and that's why I tested -V3 in addition to -V2)... Indeed, 3.97alpha3 -V2 would output to 192,99 kbps. Nice gain (-5.80%). Obviously LAME developpers also worked on efficiency. Gain is great enough that LAME --preset standard could now be fairly compared to MPC --standard. But I recall another time that it only applies for classical (I suppose that bitrate is higher with other musical suffering from sb21 issue).

Vorbis: aoTuV beta 4 -q6,00 leads to 181,48 kbps. This is lower than what I expected, and it's also lower than MPC --standard bitrate. I get 186,99 kbps for the old MEGAMIX I. Bitrate has therefore be lowered with latest aoTuV.
-q6,00 could therefore be directly compared to MPC --standard and LAME --preset fast standard (for classical music).

WMAPro: VBR75 leads to 150,24 kbps. The next available preset is VBR90 and it leads to 203,96 kbps. Both are very far for the range I fixed and consequently WMAPro can't compete in this test.

Nero Digital AAC: Like LAME and WMAPro Nero Digital doesn't offer any precise VBR scale but seven presets. -internet leads to ~142 kbps for both 'high' and 'fast' encoders. -streaming high corresponds to 176,14 kbps and -streaming fast to 193,33 kbps. Consequently none of them is inside the fixed range; the closest one is -streaming high and is therefore the less unacceptable solution (I recall that the 'high' encoder is still the recommended one).

faac AAC: this is the only encoder able to fit into the fixed bitrate range (thanks to the precise VBR scale alla vorbis & mpc). AAC faac –q 175 leads to 180,92 kbps. This –q setting won’t probably correspond to 180 kbps with other musical genre and that’s the occasion to recall another time that the whole test is specific to classical music and nothing else.


Recapitulative table

Code: [Select]
          bitrate_2004   bitrate_2005     evolution in kbps   ...in %

MPC          176,28         184,54            +8,26 kbps      +4,69 %
MP3          192,99         181,79           -11,20 kbps      -5,80 %  
Vorbis       186,99         181,48            -5,51 kbps      -2,95 %
AAC faac   not tested       180,92              --              --
AAC Nero   not tested       176,14              --              --


=> faac, LAME, aoTuV are very close each others (difference is inferior to 0,9 kbps!). MPC presents a higher bitrate (+3 kbps) and Nero Digital a lower one (-5 kbps). The gap between the extreme is worrying: approximately 5% corresponding to 8 kbps. That's not a huge difference but these eight missing kbps may lead to a significant difference in quality. I could discard Nero Digital for this test but I would consider this choice as a mistake. For my own curiosity I'm also very impatient to see how would perform an advanced implementation of AAC in comparison to other formats, even if bitrate are not fully comparable.

=> As a consequence I decided to test both Nero Digital AAC and faac AAC, and I will consider Nero Digital presence as a "bonus" interesting to watch rather than an entire competitor. That's why my final diagramme (plots) will graphically separate Nero AAC results from other contenders. I hope this will avoid unecessary debate about any kind of unfairness based on bitrate disparity.



SUMMARY

Are going to be test:

AAC: faac 1.24.1. Release date: end 2004 (?). Setting: -q175
AAC: Nero Digital aacenc32 v.3.2.0.15. Release date: June 2005. Setting: -streaming (high/default encoder).
MP3: LAME 3.97 alpha 11. Release date: July 2005. Setting: -V2 --vbr-new
MPC: mppenc 1.15v. Release date: march 2005. Setting: --quality 5
Vorbis: aoTuV beta 4 based on 1.1.1. Release date: July 2005. Setting: -q6,00



[span style='font-size:14pt;line-height:100%']IV. Additional information[/span]


I performed all my last listening tests on a Creative Audigy 2 soundcard, which resamples everything to 48000 KHz. Some people consider that internal resampling (transparent in my opinion) is treating unfairly musepack and would biased any listening test. To cut the controversial short, I installed my (better) Terratec DMX6Fire 24/96 which doesn't resample 44100 KHz files (I'm not using it anymore for daily listening because of interference with my VIA chipset).

HARDWARE & SOFTWARE SETTINGS:

soundcard: Terratec DMX6Fire 24/96
headphone: BeyerDynamic DT-531
amp: Onkyo MT-5
software player: Java ABC/HR 0.5 beta 5.
software decoder: foobar2000 0.83 (in order to automatically get files free of offset and to solve my incompatibility issues occuring with Vorbis (http://www.hydrogenaudio.org/forums/index.php?showtopic=32128&view=findpost&p=308056)).

TESTING PRINCIPLES:

ABX phase : To limit the listening fatigue and to end the test before I left my appartment, I restricted the ABX tests to the most transparent encodings (note > 4.00).
Number of trials : eight trials as a minimum. I recall that schnofler's ABC/HR software doesn't reveal to score until the test is closed by the user (and it also can't be resume). Therefore the number of trails hasn't to be fixed: as long as score is hidden the pval isn't ruined. That's why I add more trials when I suspect bad results. I never exceed 16 trials: if something is really transparent I didn't persecute the encoding
Notation : My notation was very severe last year, with a full dynamic range of notation (a lot of notes were inferior to 2.0). That's why I decided to add 10 points to each score (in order to disconnect the notation from the usual corresponding scale). This year, I tried to respect the ITU scale. When a difference is audible but not really annoying, the notation is at least equal to 4.0 and my hairs must stand on end to allow a notation inferior to 2.0 (from "annoying" to "very annoying"). Notation is still severe (I keep in mind that all encodings were set at 180 kbps) and that's why results I get here can't absolutely not be compared to other listening tests I done, especially those performed for low bitrate settings. By the way, there are no anchors in this test (high anchor is of course unecessary here).
Samples: Same as last year. See this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=23355).
Gain: I hadn't modify the gain of any file. All were played at their original volume.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 21 August, 2005, 02:34:25 PM
[span style=\'font-size:16pt;line-height:100%\']V. Results and detailed comments[/span]



(https://hydrogenaud.io/imgcache.php?id=17efde6fca5bc06be830e9f3888c9aac" rel="cached" data-warn="External image, click to view at original size" data-url="http://audiotests.free.fr/tests/2005.08/HQ180/HQ180results.png)



[span style=\'font-size:13pt;line-height:100%\']Sample 01: Krall
Short description: the only non-classical sample (Jazz). Cymbals, drums and voice.
Possible problems: smearing on cymbals and drums, distorted cymbals.
replaygain_sample_gain: = -5.50 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]

Code: [Select]
ABC/HR for Java, Version 0.5a, 17 août 2005
Testname:

Tester: guruboolez

1L = E:\SUMMER TESTS 2005\HQ180\01_krall [AAC faac].wav
2L = E:\SUMMER TESTS 2005\HQ180\01_krall [AAC Nero Digital].wav
3L = E:\SUMMER TESTS 2005\HQ180\01_krall [MPC 1.15v].wav
4R = E:\SUMMER TESTS 2005\HQ180\01_krall [vorbis aoTuV].wav
5L = E:\SUMMER TESTS 2005\HQ180\01_krall [MP3 LAME].wav

---------------------------------------
General Comments: This is what I wrote for 4R, before cancelling my notation after a bad ABX test:

"Last file I have to find. After many trials, I found one small distorted passage [3.86 - 5.67]"

failed on ABX -> 5.0 / 5.0
---------------------------------------
1L File: E:\SUMMER TESTS 2005\HQ180\01_krall [AAC faac].wav
1L Rating: 3.0
1L Comment: pre-echo is a bit worrying; it seems to distort the cymbal
---------------------------------------
2L File: E:\SUMMER TESTS 2005\HQ180\01_krall [AAC Nero Digital].wav
2L Rating: 2.3
2L Comment: pre-echo is also very perceptible ; cymbals on the beginning are even more distorted than previous file. They sound false, unatural... something weird.
---------------------------------------
3L File: E:\SUMMER TESTS 2005\HQ180\01_krall [MPC 1.15v].wav
3L Rating: 4.0
3L Comment: cymbals sound false, distorted. No pre-echo this time
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\01_krall [MP3 LAME].wav
5L Rating: 2.7
5L Comment: smearing is really perceptible (probably the worse of the serie) ; cymbals are also suffering from false sounding.
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\01_krall [vorbis aoTuV].wav
   7 out of 12, pval = 0.387


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\01_krall [vorbis aoTuV].wav
Playback Range: 03.867 to 05.671
   3:39:07 PM f 0/1 pval = 1.0
   3:39:10 PM p 1/2 pval = 0.75
   3:39:12 PM p 2/3 pval = 0.5
   3:39:15 PM f 2/4 pval = 0.687
   3:39:18 PM p 3/5 pval = 0.5
   3:39:20 PM p 4/6 pval = 0.343
   3:40:02 PM p 5/7 pval = 0.226
   3:40:04 PM f 5/8 pval = 0.363
   3:40:07 PM f 5/9 pval = 0.5
   3:40:10 PM f 5/10 pval = 0.623
   3:40:15 PM p 6/11 pval = 0.5
   3:40:19 PM p 7/12 pval = 0.387

Cymbals are still a problem for lossy encoders at this bitrate: often smeared and sometimes distorted. Nero Digital is the worst¹, followed by LAME which suffers from strongest pre-echo; then faac. Contrary to other encodings MPC doesn’t have any smearing issue, but cymbals don't sound true. aoTuV is the best: I really had to insist in order to unmask the encoding, but I totally missed the ABX phase.
¹ [span style=\'font-size:8pt;line-height:100%\']Nero Digital allocate much less bits (150 kbps) than competitors, all at more than 200 kbps.[/span]



[span style=\'font-size:13pt;line-height:100%\']Sample 02: Fuga
Short description: harpsichord.
Possible problems: pre-echo, out of tune (tremolos, vibrating notes).
replaygain_sample_gain: +14.97 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 17 août 2005
Testname:

Tester: guruboolez

1R = E:\SUMMER TESTS 2005\HQ180\02_fuga [AAC faac].wav
2L = E:\SUMMER TESTS 2005\HQ180\02_fuga [MP3 LAME].wav
3L = E:\SUMMER TESTS 2005\HQ180\02_fuga [MPC 1.15v].wav
4R = E:\SUMMER TESTS 2005\HQ180\02_fuga [AAC Nero Digital].wav
5R = E:\SUMMER TESTS 2005\HQ180\02_fuga [vorbis aoTuV].wav

---------------------------------------
General Comments:
---------------------------------------
1R File: E:\SUMMER TESTS 2005\HQ180\02_fuga [AAC faac].wav
1R Rating: 3.0
1R Comment: smearing on most harpsichord note.
One is audibly distorted (tremolo between 8.52 - 10.00]
---------------------------------------
2L File: E:\SUMMER TESTS 2005\HQ180\02_fuga [MP3 LAME].wav
2L Rating: 2.5
2L Comment: harpsichord notes are smeared, and sometimes distorted. I suspect LAME, used to have this kind of issue with this instrument.
---------------------------------------
3L File: E:\SUMMER TESTS 2005\HQ180\02_fuga [MPC 1.15v].wav
3L Rating: 4.5
3L Comment: no smearing, no 'tremolo' distortion. Very good, excepted on one note [~9.00].

AFTER ABX: slight smearing is also audible
---------------------------------------
4R File: E:\SUMMER TESTS 2005\HQ180\02_fuga [AAC Nero Digital].wav
4R Rating: 2.5
4R Comment: very good encoding; smearing is ultra-slight ; most often there's no distortions. But from ~6.00 to the end, distortions are perceptible, sometimes annoying, and smearing becomes audible too [see between 11.18 - 14.11!].
---------------------------------------
5R File: E:\SUMMER TESTS 2005\HQ180\02_fuga [vorbis aoTuV].wav
5R Rating: 4.5
5R Comment: slight smearing (?)
Hard to ABX
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\02_fuga [MPC 1.15v].wav
   15 out of 16, pval < 0.001
Original vs E:\SUMMER TESTS 2005\HQ180\02_fuga [vorbis aoTuV].wav
   16 out of 24, pval = 0.075


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\02_fuga [MPC 1.15v].wav
Playback Range: 07.915 to 12.024
   3:57:03 PM p 1/1 pval = 0.5
   3:57:07 PM p 2/2 pval = 0.25
   3:57:11 PM p 3/3 pval = 0.125
   3:57:15 PM p 4/4 pval = 0.062
   3:57:19 PM f 4/5 pval = 0.187
   3:57:25 PM p 5/6 pval = 0.109
   3:57:29 PM p 6/7 pval = 0.062
   3:57:34 PM p 7/8 pval = 0.035
   3:57:39 PM p 8/9 pval = 0.019
   3:57:42 PM p 9/10 pval = 0.01
   3:57:46 PM p 10/11 pval = 0.0050
   3:57:50 PM p 11/12 pval = 0.0030
   3:57:56 PM p 12/13 pval = 0.0010
   3:58:00 PM p 13/14 pval < 0.001
   3:58:03 PM p 14/15 pval < 0.001
   3:58:07 PM p 15/16 pval < 0.001

Original vs E:\SUMMER TESTS 2005\HQ180\02_fuga [vorbis aoTuV].wav
Playback Range: 07.463 to 14.285
   4:02:43 PM p 1/1 pval = 0.5
   4:02:48 PM f 1/2 pval = 0.75
   4:02:59 PM f 1/3 pval = 0.875
   4:03:04 PM p 2/4 pval = 0.687
   4:03:08 PM p 3/5 pval = 0.5
   4:03:12 PM p 4/6 pval = 0.343
   4:03:19 PM p 5/7 pval = 0.226
Playback Range: 10.290 to 13.645
   4:03:29 PM p 6/8 pval = 0.144
   4:03:33 PM p 7/9 pval = 0.089
   4:03:36 PM p 8/10 pval = 0.054
   4:03:38 PM p 9/11 pval = 0.032
   4:03:41 PM f 9/12 pval = 0.072
   4:03:44 PM p 10/13 pval = 0.046
   4:03:47 PM f 10/14 pval = 0.089
   4:03:50 PM p 11/15 pval = 0.059
   4:04:02 PM f 11/16 pval = 0.105
   4:04:05 PM f 11/17 pval = 0.166
   4:04:12 PM p 12/18 pval = 0.118
   4:04:15 PM f 12/19 pval = 0.179
   4:04:18 PM p 13/20 pval = 0.131
   4:04:20 PM p 14/21 pval = 0.094
   4:04:23 PM p 15/22 pval = 0.066
   4:04:25 PM f 15/23 pval = 0.105
   4:04:28 PM p 16/24 pval = 0.075

Not a surprise for me: MP3 and especially LAME has still weakness with this instrument, which is distorted by the encoding (notes are trembling) and also has smearing issue. Nero Digital is excellent on the beginning despite of smallest bitrate (163 kbps); but quality suddenly drops on the second half. Faac is a bit better on average. Both aoTuV and MPC are excellent here, with only a very subtle smearing. MPC¹ was easier to ABX due to a distortion occurring on one short moment; for aoTuV, I changed the tested range during the ABX phase in order to catch an easier segment for comparison. Pval is > 0,05 < 0,10 on overall but < 0,05 on the second range I tested.
¹ [span style=\'font-size:8pt;line-height:100%\']very high bitrate (230 kbps) for MPC, used to bloat the bitrate with solo harpsichord.[/span]



[span style=\'font-size:13pt;line-height:100%\']Sample 03: Mahler
Short description: mixed chorus (without instrument). Very tonal (no attacks).
Possible problems: kind of ringing, distortions on sibilant consonant (“s”).
replaygain_sample_gain: +0.87 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 17 août 2005
Testname:

Tester: guruboolez

1R = E:\SUMMER TESTS 2005\HQ180\03_mahler [MP3 LAME].wav
2R = E:\SUMMER TESTS 2005\HQ180\03_mahler [AAC faac].wav
3L = E:\SUMMER TESTS 2005\HQ180\03_mahler [MPC 1.15v].wav
4L = E:\SUMMER TESTS 2005\HQ180\03_mahler [AAC Nero Digital].wav
5L = E:\SUMMER TESTS 2005\HQ180\03_mahler [vorbis aoTuV].wav

---------------------------------------
General Comments:
---------------------------------------
1R File: E:\SUMMER TESTS 2005\HQ180\03_mahler [MP3 LAME].wav
1R Rating: 3.5
1R Comment: distortions on voice are audible, but not as worrying than 2R or 3L.
---------------------------------------
2R File: E:\SUMMER TESTS 2005\HQ180\03_mahler [AAC faac].wav
2R Rating: 2.0
2R Comment: voices are distorted, with strange 'pop' during the song. I suspect faac, which has this bug (pop/warbling audible on tonal moment) for a long time.
example: 4.16 - 6.07
---------------------------------------
3L File: E:\SUMMER TESTS 2005\HQ180\03_mahler [MPC 1.15v].wav
3L Rating: 2.5
3L Comment: There's something ugly in the voice, unstable, wrong. It's not natural
---------------------------------------
4L File: E:\SUMMER TESTS 2005\HQ180\03_mahler [AAC Nero Digital].wav
4L Rating: 4.5
4L Comment: This one was close to be perfect. It was the encoding I unmasked. But there's a small pasage which sounded distorted, not very hard to ABX.
On overall seems to be the less wounded by distortions.
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\03_mahler [vorbis aoTuV].wav
5L Rating: 3.8
5L Comment: It seems that some 'vocal matter' was removed. Audible for example during 20.56 - 25.16. It distorts the voice, but in a different manner as 3L, and closer to 2R. Slightly better than 1R.
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\03_mahler [AAC Nero Digital].wav
   8 out of 8, pval = 0.0030


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\03_mahler [AAC Nero Digital].wav
Playback Range: 22.289 to 25.568
   4:22:37 PM p 1/1 pval = 0.5
   4:22:41 PM p 2/2 pval = 0.25
   4:22:46 PM p 3/3 pval = 0.125
   4:22:50 PM p 4/4 pval = 0.062
   4:22:56 PM p 5/5 pval = 0.031
   4:23:01 PM p 6/6 pval = 0.015
   4:23:06 PM p 7/7 pval = 0.0070
   4:23:10 PM p 8/8 pval = 0.0030

I recognized faac, which is the only encoder to my knowledge betraying some warbling issues on this kind of (tonal) samples. In addition to this problem voice are distorted (maybe a consequence of low bitrate allocation¹). MPC has serious issue here, already noticed in the past. Voices are not natural, a bit ugly and also unstable. Less annoying are distortions audible with LAME and aoTuV. This last one seems to remove some noise/matter. Nero Digital is the best, close to perfection with problem occurring on a very small portion of the sample (and ABXed without difficulties).
¹ [span style=\'font-size:8pt;line-height:100%\']157 kbps which corresponds to the faac lowest bitrate of the whole test.[/span]


[span style=\'font-size:13pt;line-height:100%\']Sample 04: Weihnachts-Oratorium (Oratorio de Noël)
Short description: exulting orchestra (period instruments), with brass, percussions and mixed chorus.
Possible problems: loss in details, distorted on voices or instruments (especially brass).
replaygain_sample_gain: -2.62 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 17 août 2005
Testname:

Tester: guruboolez

1L = E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [AAC Nero Digital].wav
2L = E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [MP3 LAME].wav
3L = E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [MPC 1.15v].wav
4R = E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [vorbis aoTuV].wav
5R = E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [AAC faac].wav

---------------------------------------
General Comments:
---------------------------------------
1L File: E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [AAC Nero Digital].wav
1L Rating: 4.0
1L Comment: Slight distortion on chorus, exactly as if vocal matter was removed (it reminds me distortions I perceived with previous sample).
---------------------------------------
2L File: E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [MP3 LAME].wav
2L Rating: 3.2
2L Comment: orchestra is distorted. Annoying.
---------------------------------------
3L File: E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [MPC 1.15v].wav
3L Rating: 4.3
3L Comment: Some details are missing (not annoying). After ABX phase: brass are distorted, a bit wrong
---------------------------------------
4R File: E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [vorbis aoTuV].wav
4R Rating: 4.5
4R Comment: AFTER ABX: I've just lost my previous comment, I've wrote for the reference...
What I've just heard: fatness, coarseness. Would it be Vorbis? Are this problems (typical for this encoder) still audible at this bitrate?
The distortions is nevertheless really subtle (so subtle that I missed it during the ABC/HR phase). It's less annoying in my opinion that the distortion heard with previous file ('wrong color').
---------------------------------------
5R File: E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [AAC faac].wav
5R Rating: 3.0
5R Comment: distortions (a bit more annoying [on brass] than 2R)
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [vorbis aoTuV].wav
   8 out of 8, pval = 0.0030
Original vs E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [MPC 1.15v].wav
   8 out of 8, pval = 0.0030


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [vorbis aoTuV].wav
Playback Range: 00.769 to 02.102
   4:38:40 PM p 1/1 pval = 0.5
   4:38:45 PM p 2/2 pval = 0.25
   4:38:51 PM p 3/3 pval = 0.125
   4:38:56 PM p 4/4 pval = 0.062
   4:39:01 PM p 5/5 pval = 0.031
   4:39:06 PM p 6/6 pval = 0.015
   4:39:13 PM p 7/7 pval = 0.0070
   4:39:20 PM p 8/8 pval = 0.0030

Original vs E:\SUMMER TESTS 2005\HQ180\04_Oratorio Noël [MPC 1.15v].wav
Playback Range: 00.769 to 02.102
   4:37:47 PM p 1/1 pval = 0.5
   4:37:52 PM p 2/2 pval = 0.25
   4:37:57 PM p 3/3 pval = 0.125
   4:38:01 PM p 4/4 pval = 0.062
   4:38:06 PM p 5/5 pval = 0.031
   4:38:15 PM p 6/6 pval = 0.015
   4:38:17 PM p 7/7 pval = 0.0070
   4:38:20 PM p 8/8 pval = 0.0030

faac and LAME are the worse and both have the same kind of distortions altering the orchestral part. Nero AAC was one step better, with minor problems on chorus part. I noticed something I associated to noise reduction. MPC and aoTuV are very good. Brass is slightly distorted with MPC which also removes some details in the background. Vorbis is even better than MPC. So good that I first rated the reference, but the notation was automatically cancelled by ABC/HR after a positive ABX test. It’s important to note that I’ve suspected Vorbis: for the first time I heard coarseness/fatness used to affect this format.



[span style=\'font-size:13pt;line-height:100%\']Sample 05: Dom Bedos
Short description: organ, with long continuous tonal notes
Possible problems: kind of ringing (constant tonal part being fluctuating).
replaygain_sample_gain: +5.10 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 17 août 2005
Testname:

Tester: guruboolez

1L = E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [AAC faac].wav
2L = E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [MP3 LAME].wav
3R = E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [vorbis aoTuV].wav
4L = E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [MPC 1.15v].wav
5L = E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [AAC Nero Digital].wav

---------------------------------------
General Comments:
---------------------------------------
1L File: E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [AAC faac].wav
1L Rating: 2.7
1L Comment: Distortions are perceptible, not really deranging first but more irritating on the second part.
---------------------------------------
2L File: E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [MP3 LAME].wav
2L Rating: 2.7
2L Comment: beginning: very subtle distortion
end (last note): distortion is clearly more annoying
---------------------------------------
3R File: E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [vorbis aoTuV].wav
3R Rating: 4.0
3R Comment: last note has something strange, unconstant, slightly raucous.
---------------------------------------
4L File: E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [MPC 1.15v].wav
4L Rating: 3.7
4L Comment: First part is excellent, but second part (especially last note) is slightly distorted (a bit more irritating than previous file)
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [AAC Nero Digital].wav
5L Rating: 2.5
5L Comment: The most (immediately) shoking. There are weird disotrtions on tonal moments. Ugly. Last tonal note is nevertheless better, but still distorted.
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [vorbis aoTuV].wav
   8 out of 8, pval = 0.0030
Original vs E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [MPC 1.15v].wav
   13 out of 16, pval = 0.01


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [vorbis aoTuV].wav
Playback Range: 08.051 to 10.063
   9:57:12 PM p 1/1 pval = 0.5
   9:57:19 PM p 2/2 pval = 0.25
   9:57:23 PM p 3/3 pval = 0.125
   9:57:26 PM p 4/4 pval = 0.062
   9:57:29 PM p 5/5 pval = 0.031
   9:57:32 PM p 6/6 pval = 0.015
   9:57:35 PM p 7/7 pval = 0.0070
   9:57:38 PM p 8/8 pval = 0.0030

Original vs E:\SUMMER TESTS 2005\HQ180\05_Dom Bedos [MPC 1.15v].wav
Playback Range: 05.344 to 06.524
   9:54:41 PM f 0/1 pval = 1.0
   9:54:44 PM p 1/2 pval = 0.75
   9:54:48 PM p 2/3 pval = 0.5
   9:54:51 PM p 3/4 pval = 0.312
   9:54:55 PM f 3/5 pval = 0.5
   9:54:58 PM p 4/6 pval = 0.343
   9:55:02 PM p 5/7 pval = 0.226
   9:55:05 PM p 6/8 pval = 0.144
Playback Range: 09.300 to 11.475
   9:55:17 PM p 7/9 pval = 0.089
   9:55:21 PM p 8/10 pval = 0.054
   9:55:27 PM f 8/11 pval = 0.113
   9:55:30 PM p 9/12 pval = 0.072
   9:55:33 PM p 10/13 pval = 0.046
   9:55:36 PM p 11/14 pval = 0.028
   9:55:48 PM p 12/15 pval = 0.017
   9:55:51 PM p 13/16 pval = 0.01

Nero Digital appears as the worst encoding. Distortions are ugly. The encoding of the problematic last note was better and it preserves Nero AAC from a biting notation¹. Not far from Nero Digital: faac² and LAME³. Both offers a really good sound on beginning but quality drops for each on the end of the sample. Again, MPC and aoTuV are on top. Both are excellent on the first half of this sample. But second half and especially last note is apparently much harder to encode for all encoders. MPC is here distorted. Last year, I heard something wrong but failed on ABX phase; this time, ABX was successful. aoTuV presents a different form of distortion, less annoying: something raucous, unconstant and hard to describe – subtle but not too hard to ABX. Like often with very tonal part encoded by MPC: bloated bitrate.
¹ [span style=\'font-size:8pt;line-height:100%\']This is exactly the kind of sample for which Nero ‘fast’ encoder is perfect (but with bloated bitrate as other side of the coin: 233 kbps for this sample with 'fast' encoder and -streaming preset against 165 kbps for the tested encoding!)[/span]
² [span style=\'font-size:8pt;line-height:100%\']faac bitrate is very low: 150 kbps. It’s 80 kbps less than MPC![/span]
³ [span style=\'font-size:8pt;line-height:100%\']LAME quality is now much better than the one tested one year ago (ugly distortions are ringing are gone).[/span]



[span style=\'font-size:13pt;line-height:100%\']Sample 06: Platée
Short description: orchestra representing a rainstorm (period instruments).
Possible problems: background is detailed due to the presence of instruments ensuring continuo; loss in details is expected
replaygain_sample_gain: -0.07 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 17 août 2005
Testname:

Tester: guruboolez

1R = E:\SUMMER TESTS 2005\HQ180\06_Platée [AAC Nero Digital].wav
2L = E:\SUMMER TESTS 2005\HQ180\06_Platée [vorbis aoTuV].wav
3L = E:\SUMMER TESTS 2005\HQ180\06_Platée [AAC faac].wav
4L = E:\SUMMER TESTS 2005\HQ180\06_Platée [MP3 LAME].wav
5R = E:\SUMMER TESTS 2005\HQ180\06_Platée [MPC 1.15v].wav

---------------------------------------
General Comments:
---------------------------------------
1R File: E:\SUMMER TESTS 2005\HQ180\06_Platée [AAC Nero Digital].wav
1R Rating: 4.0
1R Comment: harpsichord on background is slightly imprecise, subtly softened. Not irritating. Slightly distorted between 3.83 - 5.40
---------------------------------------
2L File: E:\SUMMER TESTS 2005\HQ180\06_Platée [vorbis aoTuV].wav
2L Rating: 3.5
2L Comment: Details are also missing; harpsichord edges are softened - a bit more than previous file. Not irritating, just imprecise.
---------------------------------------
3L File: E:\SUMMER TESTS 2005\HQ180\06_Platée [AAC faac].wav
3L Rating: 2.5
3L Comment: This one is now distorted. The harpsichord in background sounds false and not only imprecise. Irritating.
---------------------------------------
4L File: E:\SUMMER TESTS 2005\HQ180\06_Platée [MP3 LAME].wav
4L Rating: 3.8
4L Comment: Again, I've lost my previous comments, because I've rated first the reference...
I give 4.5 first, but after the ABX phase I have to reconsider this note. First I've only heard a subtle loss in details, but now that I'm listening the good file and after a positive ABX tests this encoding appears as less enjoying: apart missing details (like other encodings) there's also audible distortions on harpsichord.
---------------------------------------
5R File: E:\SUMMER TESTS 2005\HQ180\06_Platée [MPC 1.15v].wav
5R Rating: 4.3
5R Comment: Softened - harpsichord edges are a bit vague.
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\06_Platée [MP3 LAME].wav
   14 out of 16, pval = 0.0020


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\06_Platée [MP3 LAME].wav
Playback Range: 01.609 to 03.684
   10:11:50 PM p 1/1 pval = 0.5
   10:11:54 PM p 2/2 pval = 0.25
   10:12:02 PM f 2/3 pval = 0.5
   10:12:05 PM f 2/4 pval = 0.687
   10:12:10 PM p 3/5 pval = 0.5
   10:12:14 PM p 4/6 pval = 0.343
   10:12:29 PM p 5/7 pval = 0.226
   10:12:33 PM p 6/8 pval = 0.144
   10:12:38 PM p 7/9 pval = 0.089
   10:12:42 PM p 8/10 pval = 0.054
   10:12:46 PM p 9/11 pval = 0.032
   10:12:49 PM p 10/12 pval = 0.019
   10:12:54 PM p 11/13 pval = 0.011
   10:13:02 PM p 12/14 pval = 0.0060
   10:13:06 PM p 13/15 pval = 0.0030
   10:13:09 PM p 14/16 pval = 0.0020

faac presents the most distorted sound on orchestra again (see sample_04, and later sample_10 & sample_15). Details are either softened or distorted. aoTuV occupies here an unusual last but one place. Sound is too imprecise, and edges of harpsichord are softened. LAME sounds in a similar manner, offering a bit more precisions than aoTuV but also slight distortions on continuo. I discovered the problem during ABX phase, and first rated the reference instead of encoded file. Nero Digital is very good, with a subtle loss in details and located distortions. It’s a remarkable performance for such low bitrate encoding¹. Same goes for MPC but without distortions this time: just a slight smoothing of details in background.
¹ [span style=\'font-size:8pt;line-height:100%\']161 kbps, whereas all other contenders have a bitrate comprise between 194 [aoTuV] and 215 kbps [LAME]![/span]


[span style=\'font-size:13pt;line-height:100%\']Sample 07: Marche Royale (00.00 – 12.00)
Short description: Chamber orchestra. First part of a sample divided in two. Here: drums, violin, trumpet, cello, clarinet.
Possible problems: pre-echo (drums) and usual distortions on instruments like violin or clarinet.
replaygain_sample_gain: +1.02 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 17 août 2005
Testname:

Tester: guruboolez

1R = E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MP3 LAME].wav
2R = E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MPC 1.15v].wav
3R = E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC faac].wav
4L = E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [vorbis aoTuV].wav
5L = E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC Nero Digital].wav

---------------------------------------
General Comments: The evaluation is based on the first twelve seconds of this sample: drums, violin, trumpet, cello, clarinet.
The second part (cymbals mostly) will be the subject of the next test
---------------------------------------
1R File: E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MP3 LAME].wav
1R Rating: 3.6
1R Comment: drums are smeared
---------------------------------------
2R File: E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MPC 1.15v].wav
2R Rating: 3.8
2R Comment: Clearly smeared. Drums are not totally clean. A bit hollowed.
---------------------------------------
3R File: E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC faac].wav
3R Rating: 3.5
3R Comment: smearing
---------------------------------------
4L File: E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [vorbis aoTuV].wav
4L Rating: 4.0
4L Comment: Few pre-echo but drums are 'noisy' from within.
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC Nero Digital].wav
5L Rating: 2.5
5L Comment: worse pre-echo; drums are also distorted.
---------------------------------------

ABX Results:

Disappointing performance of Nero Digital which presents the worst pre-echo performance. The same issue also appeared on sample_01. faac has the same situation, but annoyance is clearly lower. The encoder is followed by LAME, which seems to be very slightly better, and then MPC, also (slightly) smeared with other minor problems (hollowed drums). aoTuV is the best, with very few pre-echo but rather a distortion coming from within the drums (boosted noise).


[span style=\'font-size:13pt;line-height:100%\']Sample 08: Marche Royale (12.00 – 29.00)
Short description: Chamber orchestra – 2nd part. Here: cymbals (orchestral ones, different from those heard in sample_O1) are introduced; I focused my rating on these.
Possible problems: smearing and distortions (swoosh).
replaygain_sample_gain: +1.02 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 17 août 2005
Testname:

Tester: guruboolez

1L = E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [vorbis aoTuV].wav
2R = E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC Nero Digital].wav
3L = E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC faac].wav
4R = E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MP3 LAME].wav
5R = E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MPC 1.15v].wav

---------------------------------------
General Comments: I tried to ABX every file. At one moment I was so confused that I couln't tell which file was distorted, which one was smeared, etc... That's why I started a complete ABX checkup. I only failed for the first one. I'd still say that 1L is the encoded one, but for this test, a bad ABX test imply 5.0 as notation.
---------------------------------------
2R File: E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC Nero Digital].wav
2R Rating: 3.5
2R Comment: Smeared. Edges are really softened. Cymbals sound a bit wrong
---------------------------------------
3L File: E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC faac].wav
3L Rating: 2.5
3L Comment: Slight smearing, but much more annoying are the distortions on cymbals. Brrrrr...
---------------------------------------
4R File: E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MP3 LAME].wav
4R Rating: 2.1
4R Comment: Cymbals are the most distorted with this encoding
---------------------------------------
5R File: E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MPC 1.15v].wav
5R Rating: 4.5
5R Comment: Distortions are very small. It was the hardest to ABX, apart the first sample for which I failed.
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC Nero Digital].wav
   8 out of 8, pval = 0.0030
Original vs E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC faac].wav
   7 out of 8, pval = 0.035
Original vs E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MPC 1.15v].wav
   13 out of 16, pval = 0.01
Original vs E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MP3 LAME].wav
   8 out of 8, pval = 0.0030
Original vs E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [vorbis aoTuV].wav
   8 out of 16, pval = 0.598


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC Nero Digital].wav
Playback Range: 11.963 to 15.198
   10:51:30 PM p 1/1 pval = 0.5
   10:51:35 PM p 2/2 pval = 0.25
   10:51:39 PM p 3/3 pval = 0.125
   10:51:43 PM p 4/4 pval = 0.062
   10:51:47 PM p 5/5 pval = 0.031
   10:51:51 PM p 6/6 pval = 0.015
   10:51:57 PM p 7/7 pval = 0.0070
   10:52:01 PM p 8/8 pval = 0.0030

Original vs E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [AAC faac].wav
Playback Range: 11.963 to 15.198
   10:52:40 PM f 0/1 pval = 1.0
   10:53:01 PM p 1/2 pval = 0.75
   10:53:05 PM p 2/3 pval = 0.5
   10:53:09 PM p 3/4 pval = 0.312
   10:53:13 PM p 4/5 pval = 0.187
   10:53:17 PM p 5/6 pval = 0.109
   10:53:21 PM p 6/7 pval = 0.062
   10:53:26 PM p 7/8 pval = 0.035

Original vs E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MPC 1.15v].wav
Playback Range: 11.963 to 15.198
   10:54:51 PM p 1/1 pval = 0.5
   10:54:59 PM p 2/2 pval = 0.25
   10:55:08 PM p 3/3 pval = 0.125
   10:55:16 PM p 4/4 pval = 0.062
   10:55:21 PM p 5/5 pval = 0.031
   10:55:27 PM p 6/6 pval = 0.015
   10:55:32 PM f 6/7 pval = 0.062
   10:55:41 PM p 7/8 pval = 0.035
   10:55:48 PM f 7/9 pval = 0.089
   10:55:53 PM p 8/10 pval = 0.054
   10:55:57 PM f 8/11 pval = 0.113
   10:56:01 PM p 9/12 pval = 0.072
   10:56:09 PM p 10/13 pval = 0.046
   10:56:13 PM p 11/14 pval = 0.028
   10:56:18 PM p 12/15 pval = 0.017
   10:56:22 PM p 13/16 pval = 0.01

Original vs E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [MP3 LAME].wav
Playback Range: 11.963 to 15.198
   10:53:49 PM p 1/1 pval = 0.5
   10:53:53 PM p 2/2 pval = 0.25
   10:53:57 PM p 3/3 pval = 0.125
   10:54:01 PM p 4/4 pval = 0.062
   10:54:05 PM p 5/5 pval = 0.031
   10:54:08 PM p 6/6 pval = 0.015
   10:54:11 PM p 7/7 pval = 0.0070
   10:54:15 PM p 8/8 pval = 0.0030

Original vs E:\SUMMER TESTS 2005\HQ180\07-08_Marche Royale [vorbis aoTuV].wav
Playback Range: 11.963 to 15.198
   10:49:40 PM f 0/1 pval = 1.0
   10:49:43 PM f 0/2 pval = 1.0
   10:49:45 PM p 1/3 pval = 0.875
   10:49:48 PM p 2/4 pval = 0.687
   10:49:50 PM f 2/5 pval = 0.812
   10:49:53 PM p 3/6 pval = 0.656
   10:50:01 PM p 4/7 pval = 0.5
   10:50:03 PM f 4/8 pval = 0.636
   10:50:06 PM p 5/9 pval = 0.5
   10:50:11 PM p 6/10 pval = 0.376
   10:50:19 PM f 6/11 pval = 0.5
   10:50:22 PM f 6/12 pval = 0.612
   10:50:24 PM p 7/13 pval = 0.5
   10:50:27 PM f 7/14 pval = 0.604
   10:50:30 PM p 8/15 pval = 0.5
   10:50:32 PM f 8/16 pval = 0.598

First, short remark: I ABXed everything here, because my feeling became so confuse and imprecise on ABCHR phase that I wasn’t unable anymore to structure the notation and the hierarchy.
LAME is the worse; it wins its worst note for the entire test with this instrument (cymbals). Distorted and really unpleasant. faac also presents worrying distortions (and audible smearing). Nero Digital is much less annoying but smearing as well as distortions are still moderately perceptible. Much better is MPC, harder to ABX (13/16) and barely distorted. I noticed a very small difference for aoTuV during ABCHR phase but I couldn’t confirm it on ABX module (8/16) and I therefore manually cancelled the notation (4.8) I give him first.


[span style=\'font-size:13pt;line-height:100%\']Sample 09: Orion II
Short description: trombones –one of the most difficult brass instrument to encode for transform encoders- here meticulously recorded. It corresponds to usual “micro-attacks” problems.
Possible problems: micro-attacks encoded with noise or -in worst case- distorted.
replaygain_sample_gain: -4.80 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 17 août 2005
Testname:

Tester: guruboolez

1R = E:\SUMMER TESTS 2005\HQ180\09_Orion II [AAC faac].wav
2L = E:\SUMMER TESTS 2005\HQ180\09_Orion II [MPC 1.15v].wav
3L = E:\SUMMER TESTS 2005\HQ180\09_Orion II [MP3 LAME].wav
4L = E:\SUMMER TESTS 2005\HQ180\09_Orion II [vorbis aoTuV].wav
5L = E:\SUMMER TESTS 2005\HQ180\09_Orion II [AAC Nero Digital].wav

---------------------------------------
General Comments:
---------------------------------------
1R File: E:\SUMMER TESTS 2005\HQ180\09_Orion II [AAC faac].wav
1R Rating: 3.3
1R Comment: There's noise covering the microattacks which are slightly blurred.
---------------------------------------
2L File: E:\SUMMER TESTS 2005\HQ180\09_Orion II [MPC 1.15v].wav
2L Rating: 3.8
2L Comment: Noise between micro-attacks. Not really annoying, but clearly perceptible.
AFTER ABX: there's a clear 'pshhhhh-artefact' that identifies this encoding when compared directly to 4L.
---------------------------------------
3L File: E:\SUMMER TESTS 2005\HQ180\09_Orion II [MP3 LAME].wav
3L Rating: 3.0
3L Comment: Noise is higher than 1R. Noise reachs a peak somewhere between 3.90 - 6.67.
---------------------------------------
4L File: E:\SUMMER TESTS 2005\HQ180\09_Orion II [vorbis aoTuV].wav
4L Rating: 4.0
4L Comment: Blurring and noise; better than 1R & 3R, very similar to 2L.
AFTER ABX: this one is cleaner compared to 2L. No artefact here, just a slight constant noise.
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\09_Orion II [AAC Nero Digital].wav
5L Rating: 1.0
5L Comment: Very ugly distortion. Micro-attacks are slaughtered. It's very annoying, and 1.0 isn't severe at all.
---------------------------------------

ABX Results:
E:\SUMMER TESTS 2005\HQ180\09_Orion II [MPC 1.15v].wav vs E:\SUMMER TESTS 2005\HQ180\09_Orion II [vorbis aoTuV].wav
   12 out of 12, pval < 0.001
Original vs E:\SUMMER TESTS 2005\HQ180\09_Orion II [MPC 1.15v].wav
   14 out of 16, pval = 0.0020
Original vs E:\SUMMER TESTS 2005\HQ180\09_Orion II [vorbis aoTuV].wav
   13 out of 16, pval = 0.01


---- Detailed ABX results ----
E:\SUMMER TESTS 2005\HQ180\09_Orion II [MPC 1.15v].wav vs E:\SUMMER TESTS 2005\HQ180\09_Orion II [vorbis aoTuV].wav
Playback Range: 03.875 to 05.639
   11:31:21 PM p 1/1 pval = 0.5
   11:31:29 PM p 2/2 pval = 0.25
   11:31:32 PM p 3/3 pval = 0.125
   11:31:35 PM p 4/4 pval = 0.062
   11:31:38 PM p 5/5 pval = 0.031
   11:31:41 PM p 6/6 pval = 0.015
   11:31:43 PM p 7/7 pval = 0.0070
   11:31:46 PM p 8/8 pval = 0.0030
   11:31:49 PM p 9/9 pval = 0.0010
   11:31:52 PM p 10/10 pval < 0.001
   11:31:55 PM p 11/11 pval < 0.001
   11:31:58 PM p 12/12 pval < 0.001

Original vs E:\SUMMER TESTS 2005\HQ180\09_Orion II [MPC 1.15v].wav
Playback Range: 03.904 to 06.150
   11:26:14 PM p 1/1 pval = 0.5
   11:26:18 PM f 1/2 pval = 0.75
   11:26:22 PM p 2/3 pval = 0.5
   11:26:25 PM p 3/4 pval = 0.312
   11:26:28 PM p 4/5 pval = 0.187
   11:26:30 PM p 5/6 pval = 0.109
   11:26:34 PM p 6/7 pval = 0.062
   11:26:38 PM p 7/8 pval = 0.035
   11:26:42 PM p 8/9 pval = 0.019
   11:26:45 PM p 9/10 pval = 0.01
   11:26:49 PM p 10/11 pval = 0.0050
   11:26:53 PM p 11/12 pval = 0.0030
   11:26:58 PM p 12/13 pval = 0.0010
   11:27:02 PM f 12/14 pval = 0.0060
   11:27:06 PM p 13/15 pval = 0.0030
   11:27:10 PM p 14/16 pval = 0.0020

Original vs E:\SUMMER TESTS 2005\HQ180\09_Orion II [vorbis aoTuV].wav
Playback Range: 03.904 to 06.150
   11:28:21 PM p 1/1 pval = 0.5
   11:28:24 PM p 2/2 pval = 0.25
   11:28:27 PM f 2/3 pval = 0.5
   11:28:30 PM p 3/4 pval = 0.312
   11:28:33 PM p 4/5 pval = 0.187
   11:28:39 PM p 5/6 pval = 0.109
   11:28:42 PM p 6/7 pval = 0.062
   11:28:49 PM p 7/8 pval = 0.035
   11:28:51 PM f 7/9 pval = 0.089
   11:29:15 PM p 8/10 pval = 0.054
   11:29:21 PM p 9/11 pval = 0.032
   11:29:27 PM p 10/12 pval = 0.019
   11:29:31 PM p 11/13 pval = 0.011
   11:29:34 PM p 12/14 pval = 0.0060
   11:29:40 PM p 13/15 pval = 0.0030
   11:29:44 PM f 13/16 pval = 0.01

Nero Digital is very bad. Artefact is so terrible that I gave (thoughtful decision) the lowest note (1.0) to this sample¹. LAME and faac are much better and they only present noise as defect. MPC is even better (few noise) but presents a short artefact I identified during the ABX phase (14/16). aoTuV sounded similar to MPC but after several listenings it appears as slightly cleaner and without artefact².
¹ [span style=\'font-size:8pt;line-height:100%\']This is the only notation inferior to 2.0 for the entire test.[/span]
² [span style=\'font-size:8pt;line-height:100%\']Direct blind comparison between MPC and aoTuV was easy: 12/12 pval < 0.001. It should be worth noticing that Vorbis has made clear progress in this area (micro-attacks, used to be smeared at this preset) compared to last test including MEGAMIX.[/span]




[span style=\'font-size:13pt;line-height:100%\']Sample 10: “Dover, giustizia”
Short description: mezzo-soprano voice accompanied by orchestra.
Possible problems: voice can be distorted and instrument may also be softened.
replaygain_sample_gain: -4.73 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 17 août 2005
Testname:

Tester: guruboolez

1R = E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [vorbis aoTuV].wav
2L = E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [MPC 1.15v].wav
3L = E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [MP3 LAME].wav
4L = E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [AAC faac].wav
5L = E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [AAC Nero Digital].wav

---------------------------------------
General Comments:
---------------------------------------
2L File: E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [MPC 1.15v].wav
2L Rating: 3.8
2L Comment: Distortions on voice, clearly audible and easy to ABX. Subtle details in instrumental part are softened.
---------------------------------------
3L File: E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [MP3 LAME].wav
3L Rating: 4.5
3L Comment: Slight loss in details and few distortions. Very good.
---------------------------------------
4L File: E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [AAC faac].wav
4L Rating: 3.5
4L Comment: Not perfect. Some problems are a bit annoying (distortions on voice).
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [AAC Nero Digital].wav
5L Rating: 4.2
5L Comment: Very good. Subtle details are out (harpsichord in continuo), but it needs a direct comparison to be perceived. Slight distortions on harpsichord at the very end.
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [MP3 LAME].wav
   9 out of 12, pval = 0.072
Original vs E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [MPC 1.15v].wav
   10 out of 12, pval = 0.019
Original vs E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [vorbis aoTuV].wav
   5 out of 12, pval = 0.806


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [MP3 LAME].wav
Playback Range: 04.419 to 06.694
   11:48:13 PM p 1/1 pval = 0.5
   11:48:16 PM p 2/2 pval = 0.25
   11:48:19 PM f 2/3 pval = 0.5
   11:48:29 PM p 3/4 pval = 0.312
   11:48:34 PM f 3/5 pval = 0.5
   11:48:37 PM p 4/6 pval = 0.343
   11:48:41 PM p 5/7 pval = 0.226
   11:48:47 PM p 6/8 pval = 0.144
   11:48:57 PM p 7/9 pval = 0.089
   11:49:01 PM p 8/10 pval = 0.054
   11:49:13 PM p 9/11 pval = 0.032
   11:49:16 PM f 9/12 pval = 0.072

Original vs E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [MPC 1.15v].wav
Playback Range: 04.419 to 06.694
   11:47:20 PM f 0/1 pval = 1.0
   11:47:24 PM p 1/2 pval = 0.75
   11:47:28 PM f 1/3 pval = 0.875
   11:47:32 PM p 2/4 pval = 0.687
   11:47:35 PM p 3/5 pval = 0.5
   11:47:38 PM p 4/6 pval = 0.343
   11:47:41 PM p 5/7 pval = 0.226
   11:47:45 PM p 6/8 pval = 0.144
   11:47:48 PM p 7/9 pval = 0.089
   11:47:51 PM p 8/10 pval = 0.054
   11:47:54 PM p 9/11 pval = 0.032
   11:47:56 PM p 10/12 pval = 0.019

Original vs E:\SUMMER TESTS 2005\HQ180\10_Dover, giustizia [vorbis aoTuV].wav
Playback Range: 04.419 to 06.694
   11:45:55 PM f 0/1 pval = 1.0
   11:45:58 PM f 0/2 pval = 1.0
   11:46:02 PM p 1/3 pval = 0.875
   11:46:06 PM f 1/4 pval = 0.937
   11:46:24 PM p 2/5 pval = 0.812
   11:46:27 PM f 2/6 pval = 0.89
   11:46:30 PM p 3/7 pval = 0.773
   11:46:44 PM f 3/8 pval = 0.855
   11:46:48 PM f 3/9 pval = 0.91
   11:46:52 PM f 3/10 pval = 0.945
   11:46:56 PM p 4/11 pval = 0.886
   11:47:01 PM p 5/12 pval = 0.806

faac gets the last place, but with a flattering notation corresponding to a small distortion on voice (3.5). This is the highest note I gave for a worst encoding. Obviously, this sample doesn’t contain anything excessively difficult for ours five competitors. Unusual place for MPC which presents distortion on voice and which smooth subtle details in the background. Then comes Nero Digital which doesn’t distort voice but which softens some details in orchestral background. But Nero Digital also clearly distorts the harpsichord on the very end. LAME is similar to Nero Digital but without the distortions noticed before on harpsichord. aoTuV is transparent to my ears (5/12) despite of its low bitrate (164 kbps).


[span style=\'font-size:13pt;line-height:100%\']Sample 11: Trumpet Voluntar
Short description: mezzo trumpet with organ – the recording is very noisy on loud moments only (maybe a consequence of the organ?).
Possible problems: trumpet sometimes trigger artefacts; noise may cause ringing.
replaygain_sample_gain: -4.98 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 18 août 2005
Testname:

Tester: guruboolez

1L = E:\SUMMER TESTS 2005\HQ180\11_trumpet [AAC Nero Digital].wav
2R = E:\SUMMER TESTS 2005\HQ180\11_trumpet [MP3 LAME].wav
3R = E:\SUMMER TESTS 2005\HQ180\11_trumpet [vorbis aoTuV].wav
4L = E:\SUMMER TESTS 2005\HQ180\11_trumpet [MPC 1.15v].wav
5R = E:\SUMMER TESTS 2005\HQ180\11_trumpet [AAC faac].wav

---------------------------------------
General Comments:
---------------------------------------
1L File: E:\SUMMER TESTS 2005\HQ180\11_trumpet [AAC Nero Digital].wav
1L Rating: 3.5
1L Comment: Trumpet is slightly distorted. Not bad.
---------------------------------------
2R File: E:\SUMMER TESTS 2005\HQ180\11_trumpet [MP3 LAME].wav
2R Rating: 4.5
2R Comment: Excellent encoding. Distortion is very low (I still wonder how I get this score during ABX phase...)
---------------------------------------
3R File: E:\SUMMER TESTS 2005\HQ180\11_trumpet [vorbis aoTuV].wav
3R Rating: 4.7
3R Comment: No distortion, not irregularities, but a subtle additionnal noise.
---------------------------------------
4L File: E:\SUMMER TESTS 2005\HQ180\11_trumpet [MPC 1.15v].wav
4L Rating: 3.8
4L Comment: Trumpet is irregular, like an additionnal vibrato.
---------------------------------------
5R File: E:\SUMMER TESTS 2005\HQ180\11_trumpet [AAC faac].wav
5R Rating: 3.0
5R Comment: This one is the most immediately distorted. Trumpet has something coarse, and sound a bit false. Nothing bad, but audible and slightly annoying.
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\11_trumpet [MPC 1.15v].wav
   9 out of 10, pval = 0.01
Original vs E:\SUMMER TESTS 2005\HQ180\11_trumpet [AAC faac].wav
   10 out of 10, pval < 0.001
Original vs E:\SUMMER TESTS 2005\HQ180\11_trumpet [AAC Nero Digital].wav
   9 out of 10, pval = 0.01
Original vs E:\SUMMER TESTS 2005\HQ180\11_trumpet [MP3 LAME].wav
   8 out of 10, pval = 0.054
Original vs E:\SUMMER TESTS 2005\HQ180\11_trumpet [vorbis aoTuV].wav
   8 out of 10, pval = 0.054


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\11_trumpet [MPC 1.15v].wav
Playback Range: 01.245 to 03.052
   12:25:01 AM p 1/1 pval = 0.5
   12:25:06 AM p 2/2 pval = 0.25
   12:25:14 AM p 3/3 pval = 0.125
   12:25:18 AM p 4/4 pval = 0.062
   12:25:21 AM p 5/5 pval = 0.031
   12:25:24 AM p 6/6 pval = 0.015
   12:25:27 AM p 7/7 pval = 0.0070
   12:25:32 AM p 8/8 pval = 0.0030
   12:25:35 AM f 8/9 pval = 0.019
   12:25:38 AM p 9/10 pval = 0.01

Original vs E:\SUMMER TESTS 2005\HQ180\11_trumpet [AAC faac].wav
Playback Range: 01.245 to 03.052
   12:26:04 AM p 1/1 pval = 0.5
   12:26:07 AM p 2/2 pval = 0.25
   12:26:10 AM p 3/3 pval = 0.125
   12:26:13 AM p 4/4 pval = 0.062
   12:26:16 AM p 5/5 pval = 0.031
   12:26:19 AM p 6/6 pval = 0.015
   12:26:21 AM p 7/7 pval = 0.0070
   12:26:24 AM p 8/8 pval = 0.0030
   12:26:27 AM p 9/9 pval = 0.0010
   12:26:30 AM p 10/10 pval < 0.001

Original vs E:\SUMMER TESTS 2005\HQ180\11_trumpet [AAC Nero Digital].wav
Playback Range: 01.245 to 03.052
   12:21:10 AM p 1/1 pval = 0.5
   12:21:20 AM p 2/2 pval = 0.25
   12:21:30 AM p 3/3 pval = 0.125
   12:21:34 AM p 4/4 pval = 0.062
   12:21:38 AM p 5/5 pval = 0.031
   12:21:50 AM p 6/6 pval = 0.015
   12:21:54 AM p 7/7 pval = 0.0070
   12:21:59 AM f 7/8 pval = 0.035
   12:22:03 AM p 8/9 pval = 0.019
   12:22:06 AM p 9/10 pval = 0.01

Original vs E:\SUMMER TESTS 2005\HQ180\11_trumpet [MP3 LAME].wav
Playback Range: 01.245 to 03.052
   12:22:30 AM p 1/1 pval = 0.5
   12:22:33 AM f 1/2 pval = 0.75
   12:22:36 AM f 1/3 pval = 0.875
   12:22:42 AM p 2/4 pval = 0.687
   12:22:48 AM p 3/5 pval = 0.5
   12:22:53 AM p 4/6 pval = 0.343
   12:22:58 AM p 5/7 pval = 0.226
   12:23:03 AM p 6/8 pval = 0.144
   12:23:09 AM p 7/9 pval = 0.089
   12:23:14 AM p 8/10 pval = 0.054

Original vs E:\SUMMER TESTS 2005\HQ180\11_trumpet [vorbis aoTuV].wav
Playback Range: 01.245 to 03.052
   12:23:45 AM p 1/1 pval = 0.5
   12:23:51 AM p 2/2 pval = 0.25
   12:23:55 AM p 3/3 pval = 0.125
   12:24:00 AM p 4/4 pval = 0.062
   12:24:05 AM f 4/5 pval = 0.187
   12:24:12 AM p 5/6 pval = 0.109
   12:24:16 AM f 5/7 pval = 0.226
   12:24:32 AM p 6/8 pval = 0.144
   12:24:37 AM p 7/9 pval = 0.089
   12:24:43 AM p 8/10 pval = 0.054

The inconstant noise of the reference file is very disturbing (to be honest, I don’t like this sample). I therefore ABXed everything in order to be sure to not being betray by the weird reference sound.
faac is the worse and offers distortions (trumpet) that were immediately perceptible. Nero Digital and then MPC both present slight problems with trumpet; MPC encodes trumpets with some irregularities (phenomenon already noticed in chorus/sample_03). LAME and aoTuV are really excellent. I heard a very subtle distortion with LAME on ABC/HR ABXed with 8/10. aoTuV has no distortion but it adds a small amount of noise (also noticed with sample_07).
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 21 August, 2005, 02:35:11 PM
[span style=\'font-size:13pt;line-height:100%\']Sample 12: Liebestod
Short description: Soprano voice and full orchestra (modern instruments) – submitted by harashin.
Possible problems: same as sample_10. Violins (in orchestra) are often difficult to encode and may introduce another encoding difficulty.
replaygain_sample_gain: -3.57 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 18 août 2005
Testname:

Tester: guruboolez

1L = E:\SUMMER TESTS 2005\HQ180\12_Liebestod [AAC faac].wav
2L = E:\SUMMER TESTS 2005\HQ180\12_Liebestod [MPC 1.15v].wav
3L = E:\SUMMER TESTS 2005\HQ180\12_Liebestod [MP3 LAME].wav
4L = E:\SUMMER TESTS 2005\HQ180\12_Liebestod [vorbis aoTuV].wav
5L = E:\SUMMER TESTS 2005\HQ180\12_Liebestod [AAC Nero Digital].wav

---------------------------------------
General Comments:
---------------------------------------
1L File: E:\SUMMER TESTS 2005\HQ180\12_Liebestod [AAC faac].wav
1L Rating: 3.5
1L Comment: Audible distortion on voice; strings are not totally transparent neither.
---------------------------------------
2L File: E:\SUMMER TESTS 2005\HQ180\12_Liebestod [MPC 1.15v].wav
2L Rating: 4.0
2L Comment: Irregularities audible with both orchestra and soprano. ABXed during ~13.00 - ~18.00. Voice has something hollowed, not really pleasant, but  this problem is confined to short part of this sample.
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\12_Liebestod [AAC Nero Digital].wav
5L Rating: 3.0
5L Comment: Voice is distorted, like 1L. Most annoying are the violins (orchestra: 23.00-27.00), not really pleasant.
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\12_Liebestod [MPC 1.15v].wav
   7 out of 8, pval = 0.035
Original vs E:\SUMMER TESTS 2005\HQ180\12_Liebestod [MP3 LAME].wav
   8 out of 16, pval = 0.598
Original vs E:\SUMMER TESTS 2005\HQ180\12_Liebestod [AAC faac].wav
   8 out of 8, pval = 0.0030
Original vs E:\SUMMER TESTS 2005\HQ180\12_Liebestod [vorbis aoTuV].wav
   10 out of 16, pval = 0.227
Original vs E:\SUMMER TESTS 2005\HQ180\12_Liebestod [AAC Nero Digital].wav
   8 out of 8, pval = 0.0030


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\12_Liebestod [MPC 1.15v].wav
Playback Range: 14.167 to 20.529
   1:04:38 AM p 1/1 pval = 0.5
   1:04:41 AM p 2/2 pval = 0.25
   1:04:45 AM p 3/3 pval = 0.125
   1:04:49 AM p 4/4 pval = 0.062
   1:04:55 AM p 5/5 pval = 0.031
   1:04:59 AM p 6/6 pval = 0.015
   1:05:04 AM p 7/7 pval = 0.0070
   1:05:08 AM f 7/8 pval = 0.035

Original vs E:\SUMMER TESTS 2005\HQ180\12_Liebestod [MP3 LAME].wav
Playback Range: 21.919 to 29.992
   1:05:43 AM f 0/1 pval = 1.0
   1:05:47 AM f 0/2 pval = 1.0
   1:05:52 AM p 1/3 pval = 0.875
   1:05:56 AM f 1/4 pval = 0.937
   1:06:04 AM p 2/5 pval = 0.812
   1:06:07 AM p 3/6 pval = 0.656
   1:06:11 AM p 4/7 pval = 0.5
   1:06:18 AM f 4/8 pval = 0.636
   1:06:21 AM p 5/9 pval = 0.5
   1:06:26 AM f 5/10 pval = 0.623
   1:06:29 AM p 6/11 pval = 0.5
   1:06:44 AM f 6/12 pval = 0.612
   1:06:50 AM p 7/13 pval = 0.5
   1:06:55 AM f 7/14 pval = 0.604
   1:07:03 AM p 8/15 pval = 0.5
   1:07:23 AM f 8/16 pval = 0.598

Original vs E:\SUMMER TESTS 2005\HQ180\12_Liebestod [AAC faac].wav
Playback Range: 22.372 to 29.992
   1:03:36 AM p 1/1 pval = 0.5
   1:03:40 AM p 2/2 pval = 0.25
   1:03:44 AM p 3/3 pval = 0.125
   1:03:48 AM p 4/4 pval = 0.062
   1:03:57 AM p 5/5 pval = 0.031
   1:04:02 AM p 6/6 pval = 0.015
   1:04:05 AM p 7/7 pval = 0.0070
   1:04:08 AM p 8/8 pval = 0.0030

Original vs E:\SUMMER TESTS 2005\HQ180\12_Liebestod [vorbis aoTuV].wav
Playback Range: 21.919 to 29.992
   1:07:35 AM f 0/1 pval = 1.0
   1:07:41 AM p 1/2 pval = 0.75
   1:07:47 AM f 1/3 pval = 0.875
Playback Range: 12.082 to 20.796
   1:08:10 AM p 2/4 pval = 0.687
   1:08:16 AM f 2/5 pval = 0.812
   1:08:23 AM p 3/6 pval = 0.656
   1:08:29 AM p 4/7 pval = 0.5
   1:08:35 AM p 5/8 pval = 0.363
   1:08:41 AM p 6/9 pval = 0.253
   1:08:48 AM p 7/10 pval = 0.171
   1:08:55 AM f 7/11 pval = 0.274
   1:09:03 AM f 7/12 pval = 0.387
   1:09:08 AM p 8/13 pval = 0.29
   1:09:16 AM f 8/14 pval = 0.395
   1:09:21 AM p 9/15 pval = 0.303
   1:09:27 AM p 10/16 pval = 0.227

Original vs E:\SUMMER TESTS 2005\HQ180\12_Liebestod [AAC Nero Digital].wav
Playback Range: 17.161 to 22.186
   1:09:55 AM p 1/1 pval = 0.5
   1:10:00 AM p 2/2 pval = 0.25
   1:10:05 AM p 3/3 pval = 0.125
   1:10:10 AM p 4/4 pval = 0.062
   1:10:15 AM p 5/5 pval = 0.031
   1:10:19 AM p 6/6 pval = 0.015
   1:10:22 AM p 7/7 pval = 0.0070
   1:10:26 AM p 8/8 pval = 0.0030

Nero Digital is the worst and pay for distortions occurring in orchestral moments¹. faac offers similar performance but with less distortions on violins. MPC is better and presents some irregularities especially on voice (see sample_10 for similar problem). Note: last year I unmasked MPC on ABCHR phase but missed the ABX test. Has the encoding quality regressed or did my hearing improved a bit?
Two files are transparent to my ears (a record for this test): LAME (8/16) and aoTuV (10/16). Note: there were a clear ringing audible with alpha 3 of LAME; now it’s totally gone.
¹ [span style=\'font-size:8pt;line-height:100%\']I already noticed similar distortions in the past with violins/chorus with older version of Nero AAC; the new ‘fast’ encoder is used to perform better. The same problem is audible in similar conditions with sample_18.[/span]



[span style=\'font-size:13pt;line-height:100%\']Sample 13: LadyMacbeth
Short description: full and hysterical orchestra (modern instruments) with a lot of percussions and cymbals – submitted by harashin.
Possible problems: pre-echo is likely; distortions on other instruments (especially cymbals and brass) may also occur.
replaygain_sample_gain: -5.27 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 18 août 2005
Testname:

Tester: guruboolez

1R = E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [vorbis aoTuV].wav
2R = E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [AAC faac].wav
3R = E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [AAC Nero Digital].wav
4L = E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [MPC 1.15v].wav
5L = E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [MP3 LAME].wav

---------------------------------------
General Comments:
---------------------------------------
1R File: E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [vorbis aoTuV].wav
1R Rating: 4.2
1R Comment: Cymbals are slightly boosted by noise. Not unpleasant, but audible on direct comparison.
---------------------------------------
2R File: E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [AAC faac].wav
2R Rating: 3.5
2R Comment: Cymbals are smeared.
Percussive instruments (11.00 - 16.00) are slightly hollowed.
---------------------------------------
3R File: E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [AAC Nero Digital].wav
3R Rating: 3.0
3R Comment: Cymbals are smeared and also distorted.
Pre-echo on percussive part.
---------------------------------------
4L File: E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [MPC 1.15v].wav
4L Rating: 4.0
4L Comment: Cymbals are distorted (they are frail and sound a bit false, like after a noise reduction process).
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [MP3 LAME].wav
5L Rating: 3.0
5L Comment: Cymbals are distorted. Annoying.
Slight smearing on percussive segment.
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [MPC 1.15v].wav
   8 out of 8, pval = 0.0030
Original vs E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [vorbis aoTuV].wav
   10 out of 12, pval = 0.019


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [MPC 1.15v].wav
Playback Range: 04.620 to 08.096
   1:32:43 AM p 1/1 pval = 0.5
   1:32:50 AM p 2/2 pval = 0.25
   1:32:56 AM p 3/3 pval = 0.125
   1:33:05 AM p 4/4 pval = 0.062
   1:33:10 AM p 5/5 pval = 0.031
   1:33:13 AM p 6/6 pval = 0.015
   1:33:16 AM p 7/7 pval = 0.0070
   1:33:20 AM p 8/8 pval = 0.0030

Original vs E:\SUMMER TESTS 2005\HQ180\13_LadyMacbeth [vorbis aoTuV].wav
Playback Range: 04.620 to 08.096
   1:33:54 AM p 1/1 pval = 0.5
   1:34:05 AM f 1/2 pval = 0.75
   1:34:08 AM p 2/3 pval = 0.5
   1:34:11 AM p 3/4 pval = 0.312
   1:34:14 AM p 4/5 pval = 0.187
   1:34:17 AM p 5/6 pval = 0.109
   1:34:21 AM p 6/7 pval = 0.062
   1:34:28 AM p 7/8 pval = 0.035
   1:34:34 AM p 8/9 pval = 0.019
   1:34:38 AM p 9/10 pval = 0.01
   1:34:43 AM p 10/11 pval = 0.0050
   1:34:46 AM f 10/12 pval = 0.019

LAME and Nero Digital both produce the worst sounding: cymbals are distorted and smearing is also audible. Problems occurring with faac are less disturbing, despite of an unusual form of artefact (hollowed sound on drums). MPC and aoTuV are both very good. Musepack distorts cymbals which don't sound true again (see sample_01 and to a less extend sample_08) whereas Vorbis adds noise (see sample_07 and sample_11) which is less unpleasant to my ears and also harder to detect. ABX: 8/8 for MPC and 10/12 for aoTuV.


[span style=\'font-size:13pt;line-height:100%\']Sample 14: Vivaldi RV93
Short description: small concertino performed on period instruments.
Possible problems: on mandolin, softened or distorted harpsichord in continuo.
replaygain_sample_gain: -1.32 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 18 août 2005
Testname:

Tester: guruboolez

1R = E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [MP3 LAME].wav
2R = E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [AAC Nero Digital].wav
3L = E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [AAC faac].wav
4R = E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [vorbis aoTuV].wav
5R = E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [MPC 1.15v].wav

---------------------------------------
General Comments:
---------------------------------------
1R File: E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [MP3 LAME].wav
1R Rating: 3.0
1R Comment: Harpsichord is distorted, and slightly smeared (rather imprecise).
---------------------------------------
2R File: E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [AAC Nero Digital].wav
2R Rating: 3.5
2R Comment: Harpsichord is not perfect; better than previous file (more precise).
Violin suffers from distortions too [it's the only encoding in this situation].
---------------------------------------
3L File: E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [AAC faac].wav
3L Rating: 3.2
3L Comment: Slightly better than 1R, but worse than 2R (same issues).
---------------------------------------
4R File: E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [vorbis aoTuV].wav
4R Rating: 4.0
4R Comment: Violins are not transparent: they sound a bit false... can't define the problem: it's like a mix between thinness and fatness, noise reduction and additionnal noise. Weird... But it's very limited.
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93  [REFERENCE].wav
5L Rating: 4.5
5L Comment: Very subtle distortion on string [ABXed with pval = 0.054...]
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [vorbis aoTuV].wav
   9 out of 10, pval = 0.01
Original vs E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [MPC 1.15v].wav
   8 out of 10, pval = 0.054


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [vorbis aoTuV].wav
Playback Range: 01.134 to 04.214
   1:58:25 AM p 1/1 pval = 0.5
   1:58:30 AM p 2/2 pval = 0.25
   1:58:34 AM p 3/3 pval = 0.125
   1:58:38 AM p 4/4 pval = 0.062
   1:58:47 AM f 4/5 pval = 0.187
   1:58:52 AM p 5/6 pval = 0.109
   1:58:55 AM p 6/7 pval = 0.062
   1:59:02 AM p 7/8 pval = 0.035
   1:59:07 AM p 8/9 pval = 0.019
   1:59:13 AM p 9/10 pval = 0.01

Original vs E:\SUMMER TESTS 2005\HQ180\14_Vivaldi RV93 [MPC 1.15v].wav
Playback Range: 01.134 to 04.214
   1:59:40 AM p 1/1 pval = 0.5
   2:00:01 AM f 1/2 pval = 0.75
   2:00:09 AM p 2/3 pval = 0.5
   2:00:14 AM p 3/4 pval = 0.312
   2:00:24 AM p 4/5 pval = 0.187
   2:00:27 AM p 5/6 pval = 0.109
   2:00:34 AM p 6/7 pval = 0.062
   2:00:38 AM p 7/8 pval = 0.035
   2:00:43 AM f 7/9 pval = 0.089
   2:00:46 AM p 8/10 pval = 0.054

LAME is worst (problems are some distortions and a lack of precision) followed by faac and then Nero Digital which is –by far- the most efficient encoding¹. aoTuV has something strange: a kind of mix between thinness and fatness. The fatness issue also occurred with sample_04. MPC was transparent: I rated the reference and missed by hair’s breadth the ABX test (pval=0.054).
¹ [span style=\'font-size:8pt;line-height:100%\']168 kbps against 201 kbps (vorbis) and 213 (mpc) which are better, and 193 (faac) and 187 (mp3) which are worse.[/span]


[span style=\'font-size:13pt;line-height:100%\']Sample 15: Troisème Ballet
Short description: Concerto for bagpipe (http://www.hotpipes.com/bpgd026.jpg) played on period instruments and including some others rare and rustic instruments (like hurdy-gurdy (http://www.wga.hu/art/l/la_tour/georges/1/07hurdyg.jpg)).  [/b]-  [span style=\'font-size:8pt;line-height:100%\']See
[/color] [span style=\'font-size:8pt;line-height:100%\']this page for more informations about these old instruments[/font] (http://www.hotpipes.com/hgpipes.html). [/span]
Possible problems: ringing for bagpipe, lack of details for hurdy-gurdy.[/color][/span][/span]
[span style=\'font-size:13pt;line-height:100%\']replaygain_sample_gain: +3.21 dB[/span][/b] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 18 août 2005
Testname:

Tester: guruboolez

1L = E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [MP3 LAME].wav
2L = E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [AAC faac].wav
3L = E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [vorbis aoTuV].wav
4L = E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [AAC Nero Digital].wav
5L = E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [MPC 1.15v].wav

---------------------------------------
General Comments:
---------------------------------------
2L File: E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [AAC faac].wav
2L Rating: 3.0
2L Comment: bagpipe sounds false, with annoying distortions. Slight warbling problems also (faac?).
---------------------------------------
4L File: E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [AAC Nero Digital].wav
4L Rating: 4.5
4L Comment: Distorted.
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [MPC 1.15v].wav
5L Rating: 4.5
5L Comment: Bagpipe is distorted on tonal notes (like an echo within the sound).
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [MPC 1.15v].wav
   10 out of 12, pval = 0.019
Original vs E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [MP3 LAME].wav
   10 out of 16, pval = 0.227
Original vs E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [AAC Nero Digital].wav
   9 out of 12, pval = 0.072
Original vs E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [vorbis aoTuV].wav
   5 out of 12, pval = 0.806


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [MPC 1.15v].wav
Playback Range: 00.000 to 04.404
   2:31:23 AM p 1/1 pval = 0.5
   2:31:29 AM p 2/2 pval = 0.25
   2:31:35 AM p 3/3 pval = 0.125
   2:31:41 AM p 4/4 pval = 0.062
   2:31:53 AM p 5/5 pval = 0.031
   2:32:00 AM p 6/6 pval = 0.015
   2:32:05 AM p 7/7 pval = 0.0070
   2:32:09 AM f 7/8 pval = 0.035
   2:32:14 AM p 8/9 pval = 0.019
   2:32:21 AM p 9/10 pval = 0.01
   2:32:28 AM f 9/11 pval = 0.032
   2:32:36 AM p 10/12 pval = 0.019

Original vs E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [MP3 LAME].wav
Playback Range: 00.000 to 15.253
   2:25:48 AM p 1/1 pval = 0.5
   2:25:54 AM p 2/2 pval = 0.25
   2:26:00 AM p 3/3 pval = 0.125
   2:26:12 AM f 3/4 pval = 0.312
Playback Range: 09.434 to 13.404
   2:26:35 AM f 3/5 pval = 0.5
   2:26:40 AM p 4/6 pval = 0.343
   2:26:44 AM p 5/7 pval = 0.226
   2:26:49 AM f 5/8 pval = 0.363
   2:26:53 AM p 6/9 pval = 0.253
   2:26:58 AM p 7/10 pval = 0.171
   2:27:01 AM p 8/11 pval = 0.113
   2:27:09 AM f 8/12 pval = 0.193
   2:27:12 AM f 8/13 pval = 0.29
   2:27:17 AM p 9/14 pval = 0.211
   2:27:21 AM f 9/15 pval = 0.303
   2:27:25 AM p 10/16 pval = 0.227

Original vs E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [AAC Nero Digital].wav
Playback Range: 00.000 to 04.404
   2:30:21 AM p 1/1 pval = 0.5
   2:30:27 AM f 1/2 pval = 0.75
   2:30:33 AM p 2/3 pval = 0.5
   2:30:36 AM p 3/4 pval = 0.312
   2:30:40 AM p 4/5 pval = 0.187
   2:30:43 AM p 5/6 pval = 0.109
   2:30:49 AM f 5/7 pval = 0.226
   2:30:53 AM p 6/8 pval = 0.144
   2:30:57 AM p 7/9 pval = 0.089
   2:31:00 AM p 8/10 pval = 0.054
   2:31:06 AM f 8/11 pval = 0.113
   2:31:10 AM p 9/12 pval = 0.072

Original vs E:\SUMMER TESTS 2005\HQ180\15_Troisème Ballet [vorbis aoTuV].wav
Playback Range: 00.000 to 04.404
   2:28:51 AM p 1/1 pval = 0.5
   2:28:57 AM p 2/2 pval = 0.25
   2:29:04 AM p 3/3 pval = 0.125
   2:29:09 AM f 3/4 pval = 0.312
   2:29:15 AM p 4/5 pval = 0.187
   2:29:20 AM p 5/6 pval = 0.109
   2:29:25 AM f 5/7 pval = 0.226
   2:29:29 AM f 5/8 pval = 0.363
   2:29:34 AM f 5/9 pval = 0.5
   2:29:37 AM f 5/10 pval = 0.623
   2:29:42 AM f 5/11 pval = 0.725
   2:29:45 AM f 5/12 pval = 0.806

Most encoders are able to encode this sample perfectly or very limited problems. Only exception is faac, which distorts the bagpipe and which also adds a slight warbling, typical of this encoder (see sample_03 for similar artefact)¹. Nero Digital and MPC are both² close to perfection to my ears and they only fail (very slightly) on tonal notes of the bagpipe (10/12 for MPC³ and 9/12 for Nero Digital). LAME and aoTuV are both perfect to my ears (10/16 & 5/12).
¹ [span style=\'font-size:8pt;line-height:100%\']It might be interesting to link this poor performance to the consummed bitrate: faac used 224 kbps for this short sample (the highest one for faac on 18 samples, and also higher than other competitors for this sample).[/span]
² [span style=\'font-size:8pt;line-height:100%\']But Nero Digital is by far the most efficient, using 154 kbps instead of 213 kbps for Musepack[/span]
³ [span style=\'font-size:8pt;line-height:100%\']As for sample_12 I succeed in ABXing MPC 1.15v whereas 1.14 was transparent last year: did my ears become more sensitive or has 1.15 regressed in some areas?[/span]


[span style=\'font-size:13pt;line-height:100%\']Sample 16: basson
Short description: solo bassoon with light instrumental accompaniment.
Possible problems: micro-attacks on one moment may be softened or distorted.
replaygain_sample_gain: +0.68 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 18 août 2005
Testname:

Tester: guruboolez

1L = E:\SUMMER TESTS 2005\HQ180\16_basson [AAC Nero Digital].wav
2R = E:\SUMMER TESTS 2005\HQ180\16_basson [MPC 1.15v].wav
3L = E:\SUMMER TESTS 2005\HQ180\16_basson [MP3 LAME].wav
4L = E:\SUMMER TESTS 2005\HQ180\16_basson [vorbis aoTuV].wav
5L = E:\SUMMER TESTS 2005\HQ180\16_basson [AAC faac].wav

---------------------------------------
General Comments:
---------------------------------------
1L File: E:\SUMMER TESTS 2005\HQ180\16_basson [AAC Nero Digital].wav
1L Rating: 4.0
1L Comment: Excellent, but audible distortions during 13.0 - 17.0 range.
---------------------------------------
2R File: E:\SUMMER TESTS 2005\HQ180\16_basson [MPC 1.15v].wav
2R Rating: 4.5
2R Comment: Tiny difference during 13.0 - 17.0
---------------------------------------
3L File: E:\SUMMER TESTS 2005\HQ180\16_basson [MP3 LAME].wav
3L Rating: 4.2
3L Comment: Excellent, but again, small distortions during 13.0 - 17.0 (less than 1L and slightly more than 2R)
---------------------------------------
4L File: E:\SUMMER TESTS 2005\HQ180\16_basson [vorbis aoTuV].wav
4L Rating: 3.5
4L Comment: Sound, excellent, becomes "fat" during the problematic 13-17 range.
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\16_basson [AAC faac].wav
5L Rating: 2.5
5L Comment: The most distorted. Weak performance during 13.04 - 17.04 segment
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\16_basson [MP3 LAME].wav
   8 out of 10, pval = 0.054
Original vs E:\SUMMER TESTS 2005\HQ180\16_basson [AAC Nero Digital].wav
   9 out of 10, pval = 0.01
Original vs E:\SUMMER TESTS 2005\HQ180\16_basson [MPC 1.15v].wav
   10 out of 12, pval = 0.019
E:\SUMMER TESTS 2005\HQ180\16_basson [MPC 1.15v].wav vs E:\SUMMER TESTS 2005\HQ180\16_basson [MP3 LAME].wav
   11 out of 12, pval = 0.0030


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\16_basson [MP3 LAME].wav
Playback Range: 13.002 to 17.037
   2:50:57 AM p 1/1 pval = 0.5
   2:51:02 AM p 2/2 pval = 0.25
   2:51:08 AM p 3/3 pval = 0.125
   2:51:16 AM f 3/4 pval = 0.312
   2:51:22 AM p 4/5 pval = 0.187
   2:51:32 AM p 5/6 pval = 0.109
   2:51:38 AM p 6/7 pval = 0.062
   2:51:46 AM p 7/8 pval = 0.035
   2:51:55 AM p 8/9 pval = 0.019
   2:52:03 AM f 8/10 pval = 0.054

Original vs E:\SUMMER TESTS 2005\HQ180\16_basson [AAC Nero Digital].wav
Playback Range: 13.002 to 17.037
   2:48:22 AM p 1/1 pval = 0.5
   2:48:26 AM p 2/2 pval = 0.25
   2:48:29 AM p 3/3 pval = 0.125
   2:48:32 AM p 4/4 pval = 0.062
   2:48:36 AM p 5/5 pval = 0.031
   2:48:39 AM p 6/6 pval = 0.015
   2:48:43 AM p 7/7 pval = 0.0070
   2:48:47 AM p 8/8 pval = 0.0030
   2:48:51 AM f 8/9 pval = 0.019
   2:48:55 AM p 9/10 pval = 0.01

Original vs E:\SUMMER TESTS 2005\HQ180\16_basson [MPC 1.15v].wav
Playback Range: 13.002 to 17.037
   2:49:17 AM p 1/1 pval = 0.5
   2:49:30 AM p 2/2 pval = 0.25
   2:49:35 AM f 2/3 pval = 0.5
   2:49:42 AM p 3/4 pval = 0.312
   2:49:50 AM f 3/5 pval = 0.5
   2:49:57 AM p 4/6 pval = 0.343
   2:50:03 AM p 5/7 pval = 0.226
   2:50:08 AM p 6/8 pval = 0.144
   2:50:15 AM p 7/9 pval = 0.089
   2:50:23 AM p 8/10 pval = 0.054
   2:50:27 AM p 9/11 pval = 0.032
   2:50:34 AM p 10/12 pval = 0.019

E:\SUMMER TESTS 2005\HQ180\16_basson [MPC 1.15v].wav vs E:\SUMMER TESTS 2005\HQ180\16_basson [MP3 LAME].wav
Playback Range: 12.921 to 15.443
   2:53:48 AM p 1/1 pval = 0.5
   2:53:55 AM p 2/2 pval = 0.25
   2:53:59 AM p 3/3 pval = 0.125
   2:54:03 AM p 4/4 pval = 0.062
   2:54:07 AM p 5/5 pval = 0.031
   2:54:11 AM p 6/6 pval = 0.015
   2:54:15 AM p 7/7 pval = 0.0070
   2:54:19 AM p 8/8 pval = 0.0030
   2:54:24 AM f 8/9 pval = 0.019
   2:54:28 AM p 9/10 pval = 0.01
   2:54:44 AM p 10/11 pval = 0.0050
   2:54:48 AM p 11/12 pval = 0.0030

There were no micro-attacks issue (less stronger than those produced by trombone on sample_09. But encoding problems occur elsewhere -and for all encoders- : during one precise range when bassoon is playing at higher volume. It may be interesting to note that all encoders also allocate a lower bitrate than average (150…160 kbps and 175 kbps for MPC).
faac presents the most distorted rendering. The encoder is followed by aoTuV which presents this time (I would say another time… ¹) a fat texture on the described range; this is slightly irritating. The three remaining encoders present non-annoying distortions: Nero Digital, LAME (slightly better) and MPC (even better²).
¹ [span style=\'font-size:8pt;line-height:100%\']Same problems on sample_04 and sample_14. Last year, MEGAMIX also obtained an unusually low note on this sample (and exactly the same range).[/span]
² [span style=\'font-size:8pt;line-height:100%\']ABX comparison opposing directly LAME to MPC: 11/12[/span]



[span style=\'font-size:13pt;line-height:100%\']Sample 17: Seminarist
Short description: baritone voice accompanied by piano; the sample contains a lot of sibilant consonants.
Possible problems: there’s a big risk of distortion on sibilant.
replaygain_sample_gain: -0.45 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 18 août 2005
Testname:

Tester: guruboolez

1L = E:\SUMMER TESTS 2005\HQ180\17_seminarist [MPC 1.15v].wav
2L = E:\SUMMER TESTS 2005\HQ180\17_seminarist [AAC faac].wav
3R = E:\SUMMER TESTS 2005\HQ180\17_seminarist [vorbis aoTuV].wav
4L = E:\SUMMER TESTS 2005\HQ180\17_seminarist [MP3 LAME].wav
5L = E:\SUMMER TESTS 2005\HQ180\17_seminarist [AAC Nero Digital].wav

---------------------------------------
General Comments:
---------------------------------------
1L File: E:\SUMMER TESTS 2005\HQ180\17_seminarist [MPC 1.15v].wav
1L Rating: 3.2
1L Comment: Sibilant consonent are unatural, "slow".
Issues on voice: during 12.3 - 16.0 the baritone voice has something hard to define: false, hollowed, fluctuating?
---------------------------------------
2L File: E:\SUMMER TESTS 2005\HQ180\17_seminarist [AAC faac].wav
2L Rating: 3.0
2L Comment: Voice is distorted.
---------------------------------------
3R File: E:\SUMMER TESTS 2005\HQ180\17_seminarist [vorbis aoTuV].wav
3R Rating: 4.0
3R Comment: Similar to 1L for sibilants
---------------------------------------
4L File: E:\SUMMER TESTS 2005\HQ180\17_seminarist [MP3 LAME].wav
4L Rating: 3.7
4L Comment: Sibilants are "slow", a bit stretched like chewing-gum. Not shoking (reference file is agressive in my opinion...).
Distorted vocal part between 23.66 - 25.00
---------------------------------------
5L File: E:\SUMMER TESTS 2005\HQ180\17_seminarist [AAC Nero Digital].wav
5L Rating: 3.5
5L Comment: Similar to 1L and 3R for sibilants, but a bit more distorted (clear on 9.00 - 10.00)
---------------------------------------

ABX Results:

 faac is not really good and is betrayed by distortions on voice. MPC has presents distortions on sibilant but also on tonal part of voice. Musepack is now used to have problems with voice¹. Nero Digital is slightly better and is followed by LAME; both only show problem on sibilants, moderately annoying (reference is, as I already wrote it during the test, very aggressive). aoTuV is with MPC the one which offers the lesser problem on sibilant, but with a perfect quality on tonal part of the voice. It’s a nice progress since last year².
¹ [span style=\'font-size:8pt;line-height:100%\']See comments for sample_03 (chorus), sample_10 (mezzo-soprano) and sample_12 (soprano). Now it’s the turn to male voice to trigger distortion.[/span]
² [span style=\'font-size:8pt;line-height:100%\']Vorbis MEGAMIX obtained its worst note (2.3) and was clearly worse than LAME 3.97a3; Vorbis aoTuV beta 4 now corrects all problems that were previously audible on voice.[/span]



[span style=\'font-size:13pt;line-height:100%\']Sample 18: Butterfly Lovers
Short description:violin concerto. Violin is playing alone, then is intertwined with the orchestra
Possible problems: like Liebestod, orchestral strings may be risky for some encoders. Blending between violin and violins is possible.
replaygain_sample_gain: +3.34 dB[/b]
[/font][/span] [span style=\'font-size:11pt;line-height:100%\'](indicative only: files were tested at their original volume)[/span]


Code: [Select]
ABC/HR for Java, Version 0.5a, 18 août 2005
Testname:

Tester: guruboolez

1L = E:\SUMMER TESTS 2005\HQ180\18_Butterfly [AAC faac].wav
2L = E:\SUMMER TESTS 2005\HQ180\18_Butterfly [AAC Nero Digital].wav
3L = E:\SUMMER TESTS 2005\HQ180\18_Butterfly [MPC 1.15v].wav
4R = E:\SUMMER TESTS 2005\HQ180\18_Butterfly [MP3 LAME].wav
5L = E:\SUMMER TESTS 2005\HQ180\18_Butterfly [vorbis aoTuV].wav

---------------------------------------
General Comments:
---------------------------------------
1L File: E:\SUMMER TESTS 2005\HQ180\18_Butterfly [AAC faac].wav
1L Rating: 4.5
1L Comment: Maybe... I guess... I'm not sure... Go to ABX module.
After ABX: the minor "thing" really exists. Strings are a bit different.
---------------------------------------
2L File: E:\SUMMER TESTS 2005\HQ180\18_Butterfly [AAC Nero Digital].wav
2L Rating: 3.5
2L Comment: Violins are not transparent; there's a form of 'acidity' [I see it like this] on these instruments (I mean orchestra, not solo violin).
---------------------------------------
3L File: E:\SUMMER TESTS 2005\HQ180\18_Butterfly [MPC 1.15v].wav
3L Rating: 3.5
3L Comment: Orchestra sounds fat. It's clear on 8.0 - 12.0 range. I'd bet it's Vorbis and usual coarseness issue.
---------------------------------------
4R File: E:\SUMMER TESTS 2005\HQ180\18_Butterfly [MP3 LAME].wav
4R Rating: 4.8
4R Comment: Minor difference I can 'feel' - I need to ABX them.
After ABX: difference was very hard to catch. Really good encoding, though it's not *fully* transparent.
---------------------------------------

ABX Results:
Original vs E:\SUMMER TESTS 2005\HQ180\18_Butterfly [MP3 LAME].wav
   13 out of 16, pval = 0.01
Original vs E:\SUMMER TESTS 2005\HQ180\18_Butterfly [AAC faac].wav
   12 out of 12, pval < 0.001
Original vs E:\SUMMER TESTS 2005\HQ180\18_Butterfly [vorbis aoTuV].wav
   9 out of 16, pval = 0.401


---- Detailed ABX results ----
Original vs E:\SUMMER TESTS 2005\HQ180\18_Butterfly [MP3 LAME].wav
Playback Range: 09.518 to 16.630
   4:09:27 AM p 1/1 pval = 0.5
   4:09:33 AM p 2/2 pval = 0.25
   4:09:41 AM f 2/3 pval = 0.5
   4:09:46 AM p 3/4 pval = 0.312
   4:09:54 AM f 3/5 pval = 0.5
   4:09:59 AM f 3/6 pval = 0.656
   4:10:05 AM p 4/7 pval = 0.5
   4:10:10 AM p 5/8 pval = 0.363
   4:10:17 AM p 6/9 pval = 0.253
   4:10:20 AM p 7/10 pval = 0.171
   4:10:34 AM p 8/11 pval = 0.113
   4:10:43 AM p 9/12 pval = 0.072
   4:10:56 AM p 10/13 pval = 0.046
   4:11:02 AM p 11/14 pval = 0.028
   4:11:15 AM p 12/15 pval = 0.017
   4:11:21 AM p 13/16 pval = 0.01

Original vs E:\SUMMER TESTS 2005\HQ180\18_Butterfly [AAC faac].wav
Playback Range: 03.850 to 13.903
   4:06:54 AM p 1/1 pval = 0.5
   4:07:06 AM p 2/2 pval = 0.25
   4:07:16 AM p 3/3 pval = 0.125
   4:07:23 AM p 4/4 pval = 0.062
   4:07:30 AM p 5/5 pval = 0.031
   4:07:37 AM p 6/6 pval = 0.015
   4:07:43 AM p 7/7 pval = 0.0070
   4:07:48 AM p 8/8 pval = 0.0030
   4:08:01 AM p 9/9 pval = 0.0010
Playback Range: 17.592 to 27.699
   4:08:27 AM p 10/10 pval < 0.001
   4:08:35 AM p 11/11 pval < 0.001
   4:08:40 AM p 12/12 pval < 0.001

Original vs E:\SUMMER TESTS 2005\HQ180\18_Butterfly [vorbis aoTuV].wav
Playback Range: 09.518 to 16.630
   4:11:43 AM f 0/1 pval = 1.0
   4:11:49 AM f 0/2 pval = 1.0
   4:11:56 AM f 0/3 pval = 1.0
   4:12:02 AM f 0/4 pval = 1.0
   4:12:08 AM p 1/5 pval = 0.968
   4:12:11 AM p 2/6 pval = 0.89
   4:12:18 AM p 3/7 pval = 0.773
   4:12:22 AM p 4/8 pval = 0.636
   4:12:30 AM f 4/9 pval = 0.746
   4:12:34 AM p 5/10 pval = 0.623
   4:12:44 AM p 6/11 pval = 0.5
   4:12:48 AM p 7/12 pval = 0.387
   4:12:51 AM f 7/13 pval = 0.5
   4:13:00 AM p 8/14 pval = 0.395
   4:13:08 AM f 8/15 pval = 0.5
   4:13:17 AM p 9/16 pval = 0.401

Nero Digital produces an irritating sound. It’s a consequence of encoding problems with violins, already heard and commented for sample_12. MPC offers a similar quality. It sounded fat & coarse, and I was ready to bet that this encoding corresponded to Vorbis, former specialist of this typical problem… faac is in comparison much better. I had nothing but “warm fuzzy feeling” during ABCHR phase, and it was clarified (and verified!) by ABX (12/12): minor distortions on strings. Same thing for LAME: just a feeling on ABCHR which was confirmed by a positive ABX comparison. But the issue is so small that I really can’t describe it nor calling it “distortion”. Nice progress since last year¹. aoTuV is fully transparent to my ears again².
¹ [span style=\'font-size:8pt;line-height:100%\']3.97a3 has a clear ringing on strings, totally solved by alpha 11.[/span]
² [span style=\'font-size:8pt;line-height:100%\']Again… because Vorbis was also transparent with MEGAMIX last year. It’s interesting to note that it was the only transparent encoding of Vorbis during last listening test. This year, Vorbis aoTuV beta 4 ends with six transparent encodings.[/span]
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 21 August, 2005, 02:51:26 PM
[span style='font-size:14pt;line-height:100%']VI. Statistical analysis and conclusions[/span]



[/color]


Now, we could compare the evolution during one year (between MP3, MPC and Vorbis only).


Musepack Audio: From an evolution standpoint, MPC is the clear looser: it lost the quality crown, stolen by Vorbis. Last year MPC ends the test by getting an uncontestable first place; now the format is tied with Vorbis (which is better on average) and LAME (worse on average). On 18 samples MPC was ranked first 15 times in 2004; now it’s four times only! Musepack has also lost the efficiency trophy: with classical at least the bitrate is now superior to LAME and Vorbis. I remind that last year, MPC ends at first place with 10 kbps less than MEGAMIX and even 20 kbps less than LAME 3.97 alpha 3.
As a consequence of increased bitrate and stagnant quality, I would say that MPC is loosing its former attractiveness (for classical music). It’s not really surprising considering the low evolution of the format in a world of constant progress. Other format have simply catch up their lost time.

Vorbis: Vorbis is now impressive. Last year constant noise boost or coarseness spoiled the performance of this format. I was surprised to hear those problems on –q6,00 setting supposed to be free of them due to lossless coupling. Aoyumi has apparently identifies precisely the cause of this problem and he worked to solve it. Not entirely though: some remaining trails are still audible with few samples but the intensity is now really frail (at least on the tested samples). Other artifacts are also corrected: micro-attacks on Orion_II (sample 09) are now much less dusty (aoTuV performed even better than any other tested format at this bitrate!). Performance are remarkable and the slightly gain in bitrate consumption is the icing on the cake. Just a question: Does aoTuV mean Tuned for Victory? Or Tuned by Visitors? From what planet is Aoyumi coming from?
By the way, I wouldn’t say anymore than Vorbis is not mature enough. At least not for classical, which appeared to be a weak point (http://www.hydrogenaudio.org/forums/index.php?showtopic=23355&view=findpost&p=225107) for this format

LAME 3.97: LAME vitality defies the common sense. The format is supposed to have reached maturity for years and therefore to stagnate. The tested preset is not only better but is also faster (thanks to –vbr-new) and more efficient (-11 kbps!). The progresses are important. To precisely check them I reencoded all reference files with alpha 3 –V2 and compared them to alpha 11 –V2 –vbr-new. Indeed obvious problems are solved: the audible ringing in orchestra (sample_18) has totally disappeard, the weird distortion on organ (sample_05) is truly lowered… 2005 seems to be an exceptional vintage for LAME, comparable I would say to the release of LAME 3.90 in December 2001.




To finish, I would recall (for the n[span style='font-size:8pt;line-height:100%']th[/span] time) that all results, analysis and also tested settings are valid 1/ for classical music and 2/ for the 18 samples and 3/ for my sensitivity. Results may differ with other musical genre (MP3 would probably have more annoying issues with sharp electronic music I suppose; bitrate of -q175 and faac won't probably lead to 180 kbps with metal...), and it would be nice to see similar checkups to get a modern idea about performances of AAC, MP3, MPC and Vorbis at this transparent (or rather near transparent) setting


P.S. ABX logs are available here (http://audiotests.free.fr/tests/2005.08/HQ180/HQ180_ABXlog.7z).
P.S.#2: I made the 18 samples avaialble last year but I can't currently upload them (dial-up and I don't have 30 MB free yet).


Thanks for reading and comment
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: JeanLuc on 21 August, 2005, 02:53:15 PM
Jesus Christ ... now that's one hell of a listening test.

Great work Guru ... as always.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 21 August, 2005, 02:55:34 PM
Thanks:)

Few words to say that I had to split the messages dedicated to "comments" in order to display the full set.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: SirGrey on 21 August, 2005, 03:13:02 PM
Wow ! Great work.
Very interesting...
Especially lame progress on your beloved classical music
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: skelly831 on 21 August, 2005, 03:25:55 PM
AMAZING Guru!

This is the kind of test that makes people think about their choices of format.
I've always believed in MPC, Lame's position was expected i think, especially with the latest alpha baing so nicely improved, but the results for AAC are something to think about.

Thanks a lot Guru!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: HbG on 21 August, 2005, 03:40:36 PM
You're invaluable to this community!

And, i start liking my signature more and more
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: rjamorim on 21 August, 2005, 04:10:48 PM
Awesome! Thank-you very much, Bin Boolez*!


[span style='font-size:8pt;line-height:100%']* Internal joke, just ignore it[/span]
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: bug80 on 21 August, 2005, 04:19:23 PM
I bow, guruboolez.  Great test!

Now if only my iRiver flash player had better Ogg Vorbis support..
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Yaztromo on 21 August, 2005, 04:31:02 PM
Thanks Guru. It's more than obvious you've dedicated a lot of time to this test.

I don't think anybody had seriously thought Vorbis had got THAT good. The results are quite amazing.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: ff123 on 21 August, 2005, 04:37:05 PM
Guru,

I'm always amazed to read the professional-quality listening tests you perform for free, and then think about all the lousy ones I've had the displeasure to endure in the glossy magazines!

ff123
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Cyaneyes on 21 August, 2005, 04:45:49 PM
Guru.. you've made me very happy I purchased a DAP that supports Vorbis.

Thank you for your dedication!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Lyx on 21 August, 2005, 04:58:43 PM
Amazing. When you think he cannot get any better, then he shows you wrong. New landmark in blind-listening-test performance and presentation - and with some very interesting surprises. Thank you!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: CiTay on 21 August, 2005, 05:39:49 PM
Well done, guruboolez!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Busemann on 21 August, 2005, 05:48:52 PM
Quote
• Apple AAC: There's still no VBR mode with iTunes. Consequently it's currently impossible to use Apple's AAC encoder unless other contenders will output an average bitrate close to either 160 kbps or 192 kbps. It's unlikely...


There's always QuickTime Pro 7 and its VBR encoder:-)
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Lyx on 21 August, 2005, 06:03:58 PM
Quote
Vorbis is now –thanks to Aoyumi– an excellent audio format for 180 kbps encodings (and classical music). It has apparently no flaws (lowest note was 3.5) but only minor problems audible from time to time: additional noise, fatness, or softened details.

...

Vorbis is now impressive. Last year constant noise boost or coarseness spoiled the performance of this format. I was surprised to hear those problems on –q6,00 setting supposed to be free of them due to lossless coupling. Aoyumi has apparently identifies precisely the cause of this problem and he worked to solve it. Not entirely though: some remaining trails are still audible with few samples but the intensity is now really frail (at least on the tested samples).


As one of those who were/are sensitive to the hf-boost issue and one of the first folks to mention it, there were some questions which i asked myself to which i couldn't find an answer. And someway now seems a turning point in vorbis-development where i would like to remove my remaining doubts about vorbis.

What is the cause of vorbis' noise issue? Most here do know that it was introduced between RC3 and 1.0final. Now Aoyumi seems to have succeeded - at least in high-bitrates - to almost nullify this problem, so he obviously has an idea where the problem is/was. So what was it? Also, it does not seem to be an on/off-issue... it was slowly "fixed" over the course of time - from that it would seem to me that the "reason" for this problem is/was a fundamental vorbis problem. If this is the case, then i would wonder "how can such fundamental changes happen between a release candidate and a final release?".

I'm not trying to critizice someone here. Just asking if someone could lift the "magic" from this mysterious vorbis issue.

- Lyx
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: NumLOCK on 21 August, 2005, 06:19:10 PM
Great work Guruboolez !

It is normal though - independently of the psymodel - that transform codecs do have a big encoding advantage on these highly tonal samples (1 sine wave = 1 coefficient). This puts Musepack at a big disadvantage - but still, it is very very interesting to read, especially about Vorbis 

Would be interesting to test hard rock samples some day, to see if Vorbis beats the others there too 
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: vinnie97 on 21 August, 2005, 06:28:46 PM
holy eff, excellent test, Guru!  *not worthy*

oh and hail Vorbis!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: kwanbis on 21 August, 2005, 06:47:25 PM
excellent guruboolez
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: HotshotGG on 21 August, 2005, 07:09:49 PM
Quote
What is the cause of vorbis' noise issue? Most here do know that it was introduced between RC3 and 1.0final. Now Aoyumi seems to have succeeded - at least in high-bitrates - to almost nullify this problem, so he obviously has an idea where the problem is/was. So what was it? Also, it does not seem to be an on/off-issue... it was slowly "fixed" over the course of time - from that it would seem to me that the "reason" for this problem is/was a fundamental vorbis problem. If this is the case, then i would wonder "how can such fundamental changes happen between a release candidate and a final release


http://www2.ocn.ne.jp/~mp3lab/exp_lab/vorb...rbis_sweep.html (http://www2.ocn.ne.jp/~mp3lab/exp_lab/vorbis_sweep/vorbis_sweep.html) here you go.  . That might help you to shed some light on the subject. In the description you can see the amplitude is heavily modulated so as he said before it looks like it was a combination point stereo (which only has an effect on the amplitude) and something else possible maybe even noise normalization. He has done an excellent job with it and the positive results are shown here in this "everything but the kitchen sink listening test". It's good to see improvements on a lot of the other codecs as well though  . At higher -q values myself I really can't tell the difference, however on lower -q values you can see how noise normalization has positive impact subjectively.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: vlada on 21 August, 2005, 07:19:39 PM
Great test, thank you. I'm just wondering, how would QT AAC implementation compare with others and WMA 9 Pro. I'm not sure about it, but I heard that WMA Pro is meant for high bitrates, so this test should be ideal to compare it to others. Would it be too difficult to ad these formats?

Anyway, thank you for your hard work.

Regards,
Vlada
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: bond on 21 August, 2005, 08:34:16 PM
great test!

and great work from aoyumi!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: kl33per on 21 August, 2005, 09:08:36 PM
Awesome work guru. I don't recall the test one year ago, and as such was surprised by the results.  Very well done!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: LoFiYo on 21 August, 2005, 09:10:34 PM
Great work Mr Guru. Your occasional listening tests alone make it worth visiting this website every day. Thank you very much.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 21 August, 2005, 09:45:14 PM
Thanks to you for all support

Quote
There's always QuickTime Pro 7 and its VBR encoder:-)
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=321602")

True
Is it available on Windows? It's also not certain that one of the VBR preset would correspond to the desired bitrate. And correct me if I'm wrong: it's seems that batch encoding with current beta of QuickTime is impossible. I hope that a VBR encoding mode with iTunes will be released soon.

Quote
It is normal though - independently of the psymodel - that transform codecs do have a big encoding advantage on these highly tonal samples (1 sine wave = 1 coefficient). This puts Musepack at a big disadvantage - but still, it is very very interesting to read, especially about Vorbis 
[a href="index.php?act=findpost&pid=321612"][{POST_SNAPBACK}][/a]


Two notes:
1/ on 18 samples I would call "highly tonal" two samples only: sample_05 (organ) and sample_03 (chorus). Most other samples are not especially tonal (except maybe the trumpet and the bassoon ones).
2/ musepack is rather disadvantaged on efficiency than on quality: the encoder could allocate a lot of bits on those moments in order to maintain the quality. Extreme example with this short [a href="http://guruboolez.free.fr/SINGLE/S30_OTHERS_Accordion_A.ofr]accordion passage[/url]: mpc reaches 300 kbps with standard! On the extreme opposite we found WMAPro which encode the same sample at 100 kbps only with VBR90 (~200 kbps in average)...

Quote
Would be interesting to test hard rock samples some day, to see if Vorbis beats the others there too 

I'd like to see such test too

Quote
I'm not sure about it, but I heard that WMA Pro is meant for high bitrates, so this test should be ideal to compare it to others. Would it be too difficult to ad these formats?
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=321632")

I've [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=14091&hl=]already tested WMAPro at 128 kbps[/url] and it performed amazingly well on classical samples (and didn't performed badly on Roberto's collective test based on various musical genre). It implies that WMAPro is not only meant for high bitrate and has a great potential (considering the greenness of this encoder) at lower bitrate.
Unfortunately, WMAPro offers few VBR modes (six only) and none of these would correspond to the desired bitrate. WMAPro also has poor pre-echo performance and suffers from the same bad issues than musepack audio (http://www.hydrogenaudio.org/forums/index.php?showtopic=35030&hl=) on low volume tracks (low bitrate -> severe ringing/noise eradication).
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: rjamorim on 21 August, 2005, 09:59:54 PM
Quote
Is it available on Windows?


Yes.

Quote
And correct me if I'm wrong: it's seems that batch encoding with current beta of QuickTime is impossible.


It has always been impossible.

Yopu will share the pain I endured while preparing listening tests before iTunes was released [/quote]

Me too. About time someone shows up to replace me in testing dictatorship.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 21 August, 2005, 10:09:01 PM
Quote
Quote
And correct me if I'm wrong: it's seems that batch encoding with current beta of QuickTime is impossible.


It has always been impossible.

I'm not that sure. There was once a small app with an unpronoucable name (qtibaocas, qutibolas, qbtbicolous, qtboacoas  ) used to drive QT encoder from an external application. dBPowerAmp (fortunately easier to articulate) also has this feature.

Quote
You will share the pain I endured while preparing listening tests before iTunes was released

Hey! I also tested QT AAC encoder before iTunes first release on Windows (see link on my previous poest). And I also have to encode 150 full tracks in order to know the corresponding bitrate for each preset. Without any batch mode, it's an herculean task.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: rjamorim on 21 August, 2005, 10:42:13 PM
Quote
I'm not that sure. There was once a small app with an unpronoucable name (qtibaocas, qutibolas, qbtbicolous, qtboacoas   ) used to drive QT encoder from an external application. dBPowerAmp (fortunately easier to articulate) also has this feature.


Ah, sure. I meant that QT itself never offered a batch feature.

Quote
Hey! I also tested QT AAC encoder before iTunes first release on Windows (see link on my previous poest). And I also have to encode 150 full tracks in order to know the corresponding bitrate for each preset. Without any batch mode, it's an herculean task.[a href="index.php?act=findpost&pid=321656"][{POST_SNAPBACK}][/a]


It is indeed. If I was home, I would help you out by downloading the samples and testing at least a handful of them.

Oh well... :/
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: PabUK on 22 August, 2005, 06:48:50 AM
Excellent test, as always. Thanks.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Sunhillow on 22 August, 2005, 07:33:36 AM
Wow! A great test, Guru!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: user on 22 August, 2005, 08:02:04 AM
Chapeau Guruboolez !

Congratulations to the progress of Vorbis.

Regarding MPC, the result is split, somehow for classical samples there seems to be a degration in quality and size-effectivity (bitrate boost), comparing MPC v1.14b with MPC v1.15v
I cannot dare to ask Guru, to compare this sample set again between 1.14b and 1.15v to get a ranking of this new multiformat test including not only 1.15v but also 1.14.

Lame - MP3

The overall impression of reading the forum, and now some proof by Gurus tests (low bitrate 80 kbit/s and mid-to-high bitrate 185 kbit/s) I would recommend nowadays new Lame 3.97 over the 3.90.3.
Does the Lame - MP3 recommended settings thread need a (complete, or some major) rewrite ?








My personal point of view nowadays:


low bitrates, ca. 128k abr/vbr, target portable usage in more or less noisy environments

I use still MP3 format, will change from Lame 3.90 to lame 3.97
For portability reasons, MP3_Lame is still uptodate, as this ("free !" format is not only available in every cheap player, from car-CD-player to 1 GB USB sticks and so forth. No need to buy certain expensive portable brand players. Warranted quick 'n' (but! in quality terms) dirty exchangebality of some music with friends.
Due to the memory increasements eg. in USB sticks, the quality/bitrate issue comparing other modern formats around 128k is nothing too important to worry about, like saving 20k.



Music for high-end, Archiving, listening in living-rooms etc.

Lossless, as wavpack (-x -m) (or from previous archivings flac (-8)), due to DVD+-R or HD, no storage-size problems.
Lossy high-quality format MPC 1.14/1.15v at high bitrate (230 - 280 kbit/s, --quality 7 - 8 , --ms 15 , --xlevel) is also suitable for this goal. MPC at this quality level allows easier transport of high quality music eg. on laptop or cheap backup archive solution (on DVD+-R or HD).
Though Vorbis could now be considered ok for same goals, at quality settings 7-8 also.
Both lossy HQ formats have limited prebuilt-purchaseable "hardware" support, for clarification: MP3 has unlimited prebuilt-purchaseable "hardware" support in my definition.





*The series of multi-format tests from 80 - 185 kbit/s, carried out by Guruboolez is so impressive, I have to repeat it here again. The clear descriptions, systematically, the way of elaboration and thoughts before testing, the documentations, I cannot imagine a further improvement, or should i say: Perfect as always ?!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Aoyumi on 22 August, 2005, 08:09:47 AM
Thank you for a test and a detailed report, guruboolez.
This result is glad also for me. It proves what the direction of my tuning was not mistaken in. 

>Lyx
Quote
What is the cause of vorbis' noise issue? Most here do know that it was introduced between RC3 and 1.0final. Now Aoyumi seems to have succeeded - at least in high-bitrates - to almost nullify this problem, so he obviously has an idea where the problem is/was. So what was it? Also, it does not seem to be an on/off-issue... it was slowly "fixed" over the course of time - from that it would seem to me that the "reason" for this problem is/was a fundamental vorbis problem. If this is the case, then i would wonder "how can such fundamental changes happen between a release candidate and a final release?".

About a noise problem, it does not depend on a single cause. The change for an improvement was made through aoTuV beta2, beta3, and beta4.

The code added by beta2 rectifies change of audio energy (this brought about a big change especially). The code added by beta3 reduces the noise of an attack portion more efficiently as compared with megamix etc. More comprehensive tuning is performed in beta4. Probably, change of ATH, noise/tone masking, and noise compander etc. is useful to the improvement of roughness.

Magic does not exist there. 
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: echo on 22 August, 2005, 08:22:20 AM
Once again I am impressed by your professional listening tests guruboolez. Thank you for all your efforts.

And also more congratulations to aoyumi! His tunings have probably made vorbis the best encoder for almost all bitrate ranges!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: arman68 on 22 August, 2005, 08:25:44 AM
Very good and comprehensive test. Thank you.

It confirms the impression I recently had of Vorbis. Until a couple of months ago, I would not touch Vorbis with a bargepole, due to my own testing I did about 2 years ago, which revealed lots of artifacts that hurt my hears.

I was sold on MPC.

However, I recently retested Vorbis aoTuV beta4, and was amazed to find all the artifacts gone. I quickly did some ABX tests to see how far down I could push the bitrate for use on my portable, and I found out that Vorbis outshone every other codec! I was not sure about higher bitrates, as my ears are not good enough, but your tests confirmed what I suspected.

Vorbis is now my codec of choice.

Interesting that lame 3.97 outperforms AAC. There is still life in the old dog    Good to know when planning to use a portable which only supports AAC and MP3.

Oh, and congratulations to Ayoumi for his outstanding work on Vorbis. I bow to you.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Zurman on 22 August, 2005, 10:20:10 AM
Impressive !

And quite surprising (vorbis>mpc and lame close to mpc...)

Only one regret : wma is not in the test... ;( (for bitrates reasons I know)
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Ivan Dimkovic on 22 August, 2005, 11:18:44 AM
Hi,

First of all, thanks Guru for your very hard work.

As far as the Nero AAC is concerned, it has been already known that "High Quality" codec demonstrates some issues with classical music, and that "Fast Mode" has much better performance in that arena (as it has been proved in earlier Guru's tests) - as Guru already noted, fast mode was not regarded stable enough, and in the current version - it is still not.

I have some news for the HA users - at the moment we are testing new SBR optimizations internally - that will improve SBR quality (16-80 kbps) a lot - and as soon as SBR optimizations are done,  work will be continued in making "fast mode" of LC-AAC stable for non-classical music items as well.

Quote
Interesting that lame 3.97 outperforms AAC. There is still life in the old dog wink.gif Good to know when planning to use a portable which only supports AAC and MP3.


Please note that this is true for a particular music genre (classical) - particular codec configuration (non-SFE mode of Nero AAC), particular bit rate range (VBR ~170 kbps) and limited to a single audiophile listener.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Mr_Rabid_Teddybear on 22 August, 2005, 11:33:24 AM
I'm now really happy I recently purchased this little baby:

[a href="http://imageshack.us" target="_blank"]   
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Corsair on 22 August, 2005, 12:16:45 PM
Quote
Quote
Interesting that lame 3.97 outperforms AAC. There is still life in the old dog wink.gif Good to know when planning to use a portable which only supports AAC and MP3.

Please note that this is true for a particular music genre (classical) - particular codec configuration (non-SFE mode of Nero AAC), particular bit rate range (VBR ~170 kbps) and limited to a single audiophile listener.
[a href="index.php?act=findpost&pid=321756"][{POST_SNAPBACK}][/a]

I don't think it's really possible to call all classical samples 'a particular genre' (in terms of samples), since those samples can be quite diverse (like they are in guruboolez's test). You have, for example: solo instruments, chamber recordings (a couple of instruments), voices (with or without instruments playing), orchestras in small or large concert halls... note that there is also a wide range of instruments that can be used.
What I'm trying to say is that, unlike most other genres, classical samples can be very different and so when you have a thorough test like this one, you can get a pretty good overall picture of each codec (encoder).
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: HotshotGG on 22 August, 2005, 12:37:46 PM
Quote
Probably, change of ATH, noise/tone masking, and noise compander etc. is useful to the improvement of roughness.

Magic does not exist there.


I was digging through the code before just to try and get a better understanding of how things work? what exactly is noise companding and noise biasing? I mean what specific roles do they play? I understand the ATH and noise/tone masking routines using the FFT and MDCT, but I don't really understand the other two that well.  It's my understanding that when you are changing the impulse_noisetune if you are using advanced-command line switch you are actually adjusting the noise bias?
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Ivan Dimkovic on 22 August, 2005, 12:49:28 PM
Quote
Quote
Quote
Interesting that lame 3.97 outperforms AAC. There is still life in the old dog wink.gif Good to know when planning to use a portable which only supports AAC and MP3.

Please note that this is true for a particular music genre (classical) - particular codec configuration (non-SFE mode of Nero AAC), particular bit rate range (VBR ~170 kbps) and limited to a single audiophile listener.
[a href="index.php?act=findpost&pid=321756"][{POST_SNAPBACK}][/a]

I don't think it's really possible to call all classical samples 'a particular genre' (in terms of samples), since those samples can be quite diverse (like they are in guruboolez's test). You have, for example: solo instruments, chamber recordings (a couple of instruments), voices (with or without instruments playing), orchestras in small or large concert halls... note that there is also a wide range of instruments that can be used.
What I'm trying to say is that, unlike most other genres, classical samples can be very different and so when you have a thorough test like this one, you can get a pretty good overall picture of each codec (encoder).
[a href="index.php?act=findpost&pid=321772"][{POST_SNAPBACK}][/a]


During the experiments with Nero AAC algorithms it has been found out that most problems with 2-pass loop (high quality) encoder came from ATH and noise fluttering of the S (side) channel in the range between 11-20 kHz.

This particular effect was noticeable only with tracks that have silent "background" in that particular frequency range and very high interchannel correlation,  so M/S mode was triggered on  - and most "classical" (I also hate the term, trust me)  recordings fall in that category.

This effect could even affect L/R mode in some cases, but only when the original input is very silent - replaygaining that AAC output to higher dynamic range actually amplifies the problem.

The reason why SFE (scale factor estimation) aka "Fast" mode was found to be so much better by Guru in the previous test was direct noise injection which did not have any of the M/S related ATH issues.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Digisurfer on 22 August, 2005, 04:26:06 PM
Time to play devils advocate. What is the point of this test? All it shows are results that only apply to guruboolez's hearing, and only with classical music. Everyone is different and needs to test for themselves which is better to them, since everyone hears a little differently. I could do the same test, present the results in the same fancy manner, and come up with completely different results. The other option is an average based on a group of blind testers which makes much more sense, doesn't it?

That said, I'm not surprised by the results since they fall almost along the same lines as my own ABX testing. I use Vorbis (aotuvb4) for my portable an think it's fantastic, but if that didn't exsist I would use AAC over MP3 simply because I find the artifacts less annoying, which emphasises why people should do their own testing rather than choosing based on someone elses tests.

Hopefully I haven't offended anyone. I found this post very interesting and want to thank guruboolez as well for putting in such a huge effort. Great work!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: germanjulian on 22 August, 2005, 04:35:57 PM
thank you for this great test.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 22 August, 2005, 05:02:56 PM
Quote
Regarding MPC, the result is split, somehow for classical samples there seems to be a degration in quality and size-effectivity (bitrate boost), comparing MPC v1.14b with MPC v1.15v
I cannot dare to ask Guru, to compare this sample set again between 1.14b and 1.15v to get a ranking of this new multiformat test including not only 1.15v but also 1.14.
[a href="index.php?act=findpost&pid=321715"][{POST_SNAPBACK}][/a]

There are two different problems in my opinion:

- bitrate consumption: mpc --standard has increased by more than 10 kbps compared to former release of mppenc (I can post a full bitrate table if you want). 1.15 series is by far the version which presents the highest bitrate (not only with classical: people listening different kind of music have confirmed it - but how much is something I can't say). But this inflation is not necessary a problem: some users don't really care about few additionnal kbps, and it could bring higher quality (higher bitrate doesn't necessary mean lower efficiency).

-quality (with classical): I'm not sure that 1.15 presents regression compared to 1.14 beta. The problem maybe occur earlier. I compared 1.15u to a much older release of mppenc (1.01j) which clearly revealed issues with the latest version of mppenc (+ inflated bitrate). Now I can't tell when exactly the problem happened, or if the quality (with classical) has slowly decreased with version > 1.0 stable.


Quote
Thank you for a test and a detailed report, guruboolez.
This result is glad also for me. It proves what the direction of my tuning was not mistaken in.  
(...)
Magic does not exist there. 
[a href="index.php?act=findpost&pid=321717"][{POST_SNAPBACK}][/a]


As usual, congrats to you Aoyumi. I have tested three times your encoder (at 80, 96 and now 180 kbps) and each time I discovered the results with a deep sigh... of astonishment. I probably have to cease my listening tests before people start to suspect me from zealotry  Congrats! Your job is near to reconciliate me with high bitrate lossy encodings.

Quote
Interesting that lame 3.97 outperforms AAC. There is still life in the old dog    Good to know when planning to use a portable which only supports AAC and MP3.
[a href="index.php?act=findpost&pid=321722"][{POST_SNAPBACK}][/a]

I don't want to defend Nero AAC, but keep in mind that LAME -V2 and Nero Digital -streaming won't probably lead to similar music with most musical genre. I don't have the material to build a precise bitrate table with anything else but classical, but from whay I read in the past it's Nero AAC -normal which produce a similar bitrate to LAME --preset standard. And -normal should sound better than -streaming tested in my comparison. It's very important to remind that -besides the samples- the fairness of presets used in my test is not universal.


Quote
(...) as Guru already noted, fast mode was not regarded stable enough, and in the current version - it is still not.
[a href="index.php?act=findpost&pid=321756"][{POST_SNAPBACK}][/a]

I'd like to ask you what does the unstability issue of 'fast' encoder consist. I know two serious issues:
- smearing (it seems that 'high' encoder has better pre-echo handling).
- bloated bitrate on some occasions. I made a graphical comparison which illustrate this point. Bitrate are based on 150 full tracks (> 16 hours of music):

I'm pretty impatient to see you working more intensively on the LC core.

Quote
What I'm trying to say is that, unlike most other genres, classical samples can be very different and so when you have a thorough test like this one, you can get a pretty good overall picture of each codec (encoder).
[a href="index.php?act=findpost&pid=321772"][{POST_SNAPBACK}][/a]

I'm fully agree with you: 'classical' is a generic term which doesn't really mean anything and which wrongly encompasses most written instrumental/lyrical music composed before the 20th century.
About lossy encoding: I would also believe that any encoder able to handle perfectly all situation encountered with 'classical' would be a champion for every musical genre. But such encoder don't exists and my current listening test is also far to represent all possible situations happening with classical. That's why it would be nice to see tests focusing on other problems than those tested here.
My results can't of course be generalized and extrapolated to other musical genre as well as it would be excessive to believe that results could be totally different with a another set of samples.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: sh1leshk4 on 22 August, 2005, 05:05:24 PM
Quote
What is the point of this test? All it shows are results that only apply to guruboolez's hearing, and only with classical music. Everyone is different and needs to test for themselves which is better to them, since everyone hears a little differently. I could do the same test, present the results in the same fancy manner, and come up with completely different results. The other option is an average based on a group of blind testers which makes much more sense, doesn't it?[a href="index.php?act=findpost&pid=321838"][{POST_SNAPBACK}][/a]

If so, then what's the relevance of having most af the personally ABX-ed listening tests to be shown as a proof that a codec is better (at some bitrates and/or genres) from another?
And it's not like we're seeing listening tests done in a group everyday.
Plus, with your reasoning, then results coming from a group of blind testers probably won't make sense as well, since it's probably valid only for that certain group.

I guess personally conducted listening tests by others can't be used as something to back up some claims and using them might be against TOS #08...

To guruboolez, thank you for conducting the test.
To Aoyumi, thank you for helping Vorbis to be a better codec one leap at a time.
To Ivan, I'm eager to see the outcome of the next version of Nero AAC encoder.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Lyx on 22 August, 2005, 05:06:15 PM
Quote
The other option is an average based on a group of blind testers which makes much more sense, doesn't it?
[a href="index.php?act=findpost&pid=321838"][{POST_SNAPBACK}][/a]

And how many users on this board would be able and willing to take such a demanding test? IIRC even on ha.org, the amount of people who are able to do tests at near-200kbit is very low - which is part of the reason why people are grateful for guru's test. And then one would have to get those few together simultaneusly to participate in a test. Sure, of course it would be better if it could be done - but that may be a little difficult to achieve.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Axon on 22 August, 2005, 05:08:18 PM
While I don't doubt that this is a very significant study, I also agree that more work is necessary before we can bless 3.97 stable as the recommended HA version. I think that is eventually going to happen though.

It's very tempting to try to confirm guruboolez's results myself, as a starting point, but I don't think I can scrounge up the 10+ hours of intense ABX time I'd most likely need.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: ching-3 on 22 August, 2005, 05:13:32 PM
Great work guru, a very good test.

I always believed that Aoyumi tuned vorbis very well
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 22 August, 2005, 05:13:58 PM
Quote
Time to play devils advocate. What is the point of this test? All it shows are results that only apply to guruboolez's hearing, and only with classical music.
[a href="index.php?act=findpost&pid=321838"][{POST_SNAPBACK}][/a]

Exactly.
But every listening test presents the same kind of limitation. None listener can't go beyond its own subjectivity.
What I can do is posting objective analysis: graphical illustration showing lowpass, mix-pasted comparisons, or eaqual benchs. But you know what people think about them and how much the penalty is for using such tools for claming anything about quality
We're doomed to our subjectivity. I can also post all samples and expect that other users will try to perform the same test; by the way I always uploaded sample to that purpose. Feedback = 0. It's also understandable: I won't be very interested either to perform a full listening test on music I don't listen to. But you'll also notice that almost nobody has posted in the last years any listening comparison based on their favorite samples.

[span style='font-size:8pt;line-height:100%']EDIT: two mistakes.[/span]
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: rjamorim on 22 August, 2005, 05:14:51 PM
Quote
I could do the same test, present the results in the same fancy manner, and come up with completely different results.[a href="index.php?act=findpost&pid=321838"][{POST_SNAPBACK}][/a]


Could you really? Have you done so? Or is that just speculation?
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Digisurfer on 22 August, 2005, 06:20:03 PM
Quote
Quote
I could do the same test, present the results in the same fancy manner, and come up with completely different results.[a href="index.php?act=findpost&pid=321838"][{POST_SNAPBACK}][/a]


Could you really? Have you done so? Or is that just speculation?
[a href="index.php?act=findpost&pid=321860"][{POST_SNAPBACK}][/a]

Speculation of course, primarily because differeing results are certinaly within the realm of possibility. Every one is different after all. I wasn't just talking about 180k tests by the way, my query applies to any bitrate. I'm lazy so I wouldn't really bother of course. I only bought it up because one of the things I see people posting here at HA.org all the time is that you have to ABX test for yourself in order to get any truly meaningful results, and that makes perfect sense of course. Thus, any testing I might do is only relevent to me, and is why I feel it would be a waste of time to post such results even if some folks may find said results interesting, though I honestly doubt anyone would actually care all that much. After all, I'm just a nobody and guruboolez is the one with the golden ears which, oddly enough, seem to have attained a strange sort of celebrity status around here, hehe. More power to him too. Like I said, despite what seems like (admitedly very minor) hypocrisy, I have to admit I find the tests fun and interesting to read just like everyone else. Thanks again for all your hard work guru! Again, I hope I have not offended anyone, since that was never my intent. Just find the whole thing rather amusing is all, given what is normally posted whenever someone new comes along and asks "what is the best codec/bitrate?".
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Lyx on 22 August, 2005, 06:33:33 PM
I think the flaw in your logic is that you see this as a black/white issue. Tests by other people - especially if you do know about their skill relative to yours - can be a helpful indication. Basically, statistics...... the more of those indications, the more robust the overall picture. However, that picture is only an "average" - and you may be above or below it... or somewhere else ;)

Basically, other peoples tests *are* helpful because they are an indication. Here's an example: I do know that guru's skills are far above mine(both in terms of training as well as ears). Thus, when listening to classical music i could asume that V3 should be transparent to me most of the time. However, if i will ABX the sample-set myself, then i may find out that even v4 is sufficient for me. Thus, other people's tests to give one an indication...... but for fine-tuning, it is necessary to ABX oneself. Some people may decide to skip the additional time for fine-tuning and instead "encode slightly higher than probably needed" - thus, trading space for time & effort.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 22 August, 2005, 06:52:23 PM
Quote
Speculation of course, primarily because differeing results are certinaly within the realm of possibility. Every one is different after all.

Yes but blue is blue for everyone except maybe for daltonic persons. Everyone is different, but the possible profiles are not infinite. And would be the gap between these profiles really important?

Quote
Thus, any testing I might do is only relevent to me, and is why I feel it would be a waste of time to post such results

It's an admirable disinterestedness! I'm sincerly admirative.
But this board is a common place for sharing its own experience. When several individual experience are going along the same lines, then it starts to become trustable. Some formats are considered as superior to others: AAC over VQF, MPC over MP3, etc... These affirmations are based on similar individual experience. If everybody would keep their results secret, no knowledge is possible.

Now I could also keep my listening test for myself. But what will it imply? If I'll answer "Vorbis" to someone interesting to know what should be an ideal format for ~200 kbps encoding with classical, you know what will happen... Most people will ask me to backup my claim because MPC is well-known to have no rivality at this bitrate. One will brandish TOS#8, another one request my message to be deleted, I could get warn, etc...

Quote
After all, I'm just a nobody and guruboolez is the one with the golden ears which, oddly enough, seem to have attained a strange sort of celebrity status around here, hehe. More power to him too.


I don't like this kind of charisma either. I don't want people following blindly my conclusions. I'm not a guru [span style='font-size:8pt;line-height:100%'](yes, I know, my nickname could imply the opposite - but if there's one guru, it's rather Pierre Boulez than me)[/span]. That's why I often request additionnal listening tests and certainly not to set up an altar for my holy conclusions

Quote
Again, I hope I have not offended anyone, since that was never my intent.
For me it's OK. You're just questioning the way 'trustable knowledge' of something based on individual experience is possible. Well, it's just epistemology.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: QuantumKnot on 22 August, 2005, 07:27:51 PM
It is always very enjoyable and informative to read a listening test by guruboolez.  In fact, I haven't visited HA for a while but once I saw a listening test done by guruboolez, I headed straight for this thread

What can I say?  I am extremely happy with Ogg Vorbis!  Many thanks to Aoyumi for fixing the problems and maturing Vorbis to such an advanced state that it can even rival the once-dominant MPC.  Has anyone told Monty about these results?  Monty needs to know!!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: vinnie97 on 22 August, 2005, 08:28:02 PM
I have to agree with Guru.  Either these artifacts exist or they don't.  The level at which various artifacts become bothersome to the individual is the only issue in question (and it's really only significant when dealing with higher bitrates...lower bitrates produce much more noticeable artifacts).  Anyone with a golden pair of ears and artifact awareness (such as guru) can make a valid assessment when they blindly compare to the reference, IMO.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: boiling_ice2k4 on 22 August, 2005, 10:19:51 PM
nice listening test guruboolez  great to see the progress that vorbis has made in the last year, and its neat knowing the format still has a great deal of room for future improvements if necessary.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: HotshotGG on 23 August, 2005, 12:25:19 AM
Quote
What can I say? I am extremely happy with Ogg Vorbis! Many thanks to Aoyumi for fixing the problems and maturing Vorbis to such an advanced state that it can even rival the once-dominant MPC. Has anyone told Monty about these results? Monty needs to know!!


He drops in occasionally, but I presume he is busy doing other more important stuff. 

Quote
I hope I have not offended anyone, since that was never my intent. Just find the whole thing rather amusing is all, given what is normally posted whenever someone new comes along and asks "what is the best codec/bitrate?". wink.gif


all the listening tests show to me is a small or substantial improvement in the actual codec. "Best" is whatever is best for your ears and whatever you prefer.  Don't forget that most things outside this community are irrelivent (unless websites point here) and unless they actually get merged into the mainline, much like in 1.1.1.  It's just too bad their aren't any more developers working on getting multichannel coupling up to snuff, etc or who have the ability to do so. Taming that beast is another esotreric task ;D.  Aoyumi work is always greatly appreciated though.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: user on 23 August, 2005, 07:04:01 AM
Quote
Quote
Regarding MPC, the result is split, somehow for classical samples there seems to be a degration in quality and size-effectivity (bitrate boost), comparing MPC v1.14b with MPC v1.15v
I cannot dare to ask Guru, to compare this sample set again between 1.14b and 1.15v to get a ranking of this new multiformat test including not only 1.15v but also 1.14.
[a href="index.php?act=findpost&pid=321715"][{POST_SNAPBACK}][/a]

There are two different problems in my opinion:

- bitrate consumption: mpc --standard has increased by more than 10 kbps compared to former release of mppenc (I can post a full bitrate table if you want). 1.15 series is by far the version which presents the highest bitrate (not only with classical: people listening different kind of music have confirmed it - but how much is something I can't say). But this inflation is not necessary a problem: some users don't really care about few additionnal kbps, and it could bring higher quality (higher bitrate doesn't necessary mean lower efficiency).

-quality (with classical): I'm not sure that 1.15 presents regression compared to 1.14 beta. The problem maybe occur earlier. I compared 1.15u to a much older release of mppenc (1.01j) which clearly revealed issues with the latest version of mppenc (+ inflated bitrate). Now I can't tell when exactly the problem happened, or if the quality (with classical) has slowly decreased with version > 1.0 stable.



yeah, I know it from posts, I used to use MPC 1.01j once, too. And 1.14 for some time, until I switched to 1.15u,v.
The bitrate increase is not only for classic or your impressive 150 samples, but also clearly known, reported here in forum, too, when comparing 1.14 with 1.15r,s,t,u,v. The bitrate increase happens in all genres, classic, jazz, pop, rock, metal probably, too.
I agree, that we don't need to discuss the bitrate as such.
I am using 1.15 myself, as it improved other problem samples. But I was pointing to the facts, just to make it not forgotten..


One remark to the validity of this test and others:

If a (the!) trained person tests samples, the results will be more consistent than you carry out a test with a bigger group which includes "deaf" people, who just lower the p-values, who don't add any significance.
Of course, primarily you have to say, that this test is primarily valid only for this 1 test-person.
But we have some history here, and the reason why Guruboulez tests are very welcomed here, is following: Other persons repeated gurus tests (with or without posting results) and eitehr confirmed them, or admitted, not to hear the difference, but most important: no contradiction.
By experience some people here know, that guruboulez results are valid not only to him, but also to other persons, normally worded "made same experience".
So, if somebody above tries to play "devils advocate", and questions validity of this test, he should not ridicule himself. The only way to question results is in presenting listening test with opposite/different results.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: rjamorim on 23 August, 2005, 09:43:07 AM
Quote
If a (the!) trained person tests samples, the results will be more consistent than you carry out a test with a bigger group which includes "deaf" people, who just lower the p-values, who don't add any significance.


Actually the deaf group adds significance, as the world isn't populated only by golden ears.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Aoyumi on 23 August, 2005, 09:43:34 AM
Quote
Quote
Probably, change of ATH, noise/tone masking, and noise compander etc. is useful to the improvement of roughness.

Magic does not exist there.


I was digging through the code before just to try and get a better understanding of how things work? what exactly is noise companding and noise biasing? I mean what specific roles do they play? I understand the ATH and noise/tone masking routines using the FFT and MDCT, but I don't really understand the other two that well.  It's my understanding that when you are changing the impulse_noisetune if you are using advanced-command line switch you are actually adjusting the noise bias?
[a href="index.php?act=findpost&pid=321779"][{POST_SNAPBACK}][/a]

Noisebias expresses a frequency noise curve, and noise compand expresses a loudness noise curve. These are bearing the role important for determining a final noise masking value.
Moreover, impulse_noisetune operates noisebias of impulse block (short block). 
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 23 August, 2005, 09:49:14 AM
Quote
The bitrate increase is not only for classic or your impressive 150 samples, but also clearly known, reported here in forum, too, when comparing 1.14 with 1.15r,s,t,u,v. The bitrate increase happens in all genres, classic, jazz, pop, rock, metal probably, too.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=322009\")
I found an old message posted by Frank Klemm about this increase:

Quote
Bitrate is increased by typically 3.1% compared with 1.14. Most of this bitrate increase is related to
some modificitions in the tonality estimation model (which was introduced with 1.15g).
Source [a href=\"http://www.hydrogenaudio.org/forums/index.php?showtopic=6558&view=findpost&p=66098]here[/url].

If you're interested, I also compared the bitrate of six different mppenc version, from Buschmann 1.7.8c to latest 1.15v:

(https://hydrogenaud.io/imgcache.php?id=7d57e4a869d69e80429f528362dfd519" rel="cached" data-warn="External image, click to view at original size" data-url="http://audiotests.free.fr/tests/2005.08/HQ180/mppenc6encoders.png)
(a zoomed version here (http://audiotests.free.fr/tests/2005.08/HQ180/mppenc6encoders_zoom.png).)

If you can't see 1.78c it's simply because 0.90o (my oldest mppenc version) is ultra-close to it.
The 1.04 version is by far the most efficient version of mppenc - but remember how much people were frighten about this bitrate drop! Therefore, bitrate started to increase again with 1.06 (not present here).
1.14 is still close to native --standard bitrate (+3,5 kbps on average for my 150 full tracks).
1.15v is now far from what Buschmann 1.78c / Klemm 1.00 encoders used to output. The difference between the extreme is 15 kbps! Does it really justify some gain with a limited set of samples?

If you want to play with the datas, here's the complete table (sorted by samples):
Code: [Select]
CATALOG	1.7.8c	0.90o	1.01j	1.04	1.14	1.15v
NUMBER
A01 153,2 153,1 157,0 152,7 157,9 164,1
A02 176,8 176,8 182,3 179,2 189,1 192,2
A03 185,6 185,6 189,7 189,4 200,4 205,1
A04 173,5 173,5 177,3 178,4 185,6 188,2
A05 172,8 172,8 176,2 162,2 170,0 177,9
E01 117,1 117,1 118,0 110,0 116,8 121,8
E02 173,8 173,8 175,5 171,6 178,8 186,7
E03 174,6 174,7 178,0 175,7 183,8 188,8
E04 159,7 159,6 160,2 153,1 159,6 167,6
E05 157,5 157,5 161,6 157,0 165,3 167,2
E06 173,8 173,8 175,3 171,7 178,9 187,2
E07 169,1 169,1 169,9 163,1 169,5 178,2
E08 164,4 164,4 170,1 154,4 165,0 168,0
E09 176,0 176,0 177,2 173,0 180,1 188,3
E10 168,1 168,1 170,1 158,5 166,3 173,3
E11 174,1 174,1 177,0 173,6 181,1 188,3
E12 150,3 150,3 152,6 145,2 151,5 157,9
E13 183,3 183,3 183,3 178,3 183,7 193,3
E14 169,9 169,9 172,6 169,2 177,3 185,2
E15 177,7 177,6 177,4 171,9 179,1 190,8
E16 186,0 186,0 187,8 185,0 191,9 199,7
E17 167,6 167,6 170,9 168,0 174,3 178,2
E18 149,8 149,8 151,8 146,3 151,7 157,4
E19 147,3 147,3 148,3 142,1 148,1 155,2
E20 151,0 151,0 152,0 141,0 148,9 157,3
E21 153,8 153,8 156,0 150,1 156,0 162,6
E22 173,5 173,4 175,3 174,1 180,2 187,6
E23 161,9 161,9 163,5 158,4 164,4 172,8
E24 159,1 159,1 160,3 153,1 160,6 169,1
E25 173,9 173,8 176,1 176,0 182,2 188,6
E26 165,1 165,0 167,9 161,7 168,9 174,6
E27 169,2 169,2 171,1 164,9 173,3 180,8
E28 151,7 151,7 156,0 153,9 159,6 159,6
E29 158,1 158,1 160,9 153,6 162,2 166,9
E30 164,2 164,2 168,2 166,0 172,7 175,9
E31 185,1 185,0 186,3 186,6 193,0 201,4
E32 189,7 189,7 189,9 188,0 193,7 204,7
E33 170,9 170,8 172,9 169,1 179,9 185,7
E34 162,3 162,3 164,9 159,0 163,3 168,8
E35 204,5 204,4 203,6 199,6 203,8 220,9
E36 201,5 201,4 201,3 199,9 205,9 216,3
E37 181,4 181,3 182,7 177,7 185,3 193,7
E38 174,9 174,8 176,0 171,8 178,4 187,1
E39 175,6 175,5 176,2 170,4 176,7 186,3
E40 157,7 157,7 160,3 152,2 160,7 168,6
E41 213,5 213,5 212,0 207,2 211,8 225,0
E42 207,7 207,6 207,3 203,1 209,7 220,6
E43 190,0 190,0 190,6 187,6 193,4 205,4
E44 169,8 169,8 171,1 161,9 168,7 178,9
E45 201,5 201,5 201,6 201,2 207,3 219,2
E46 168,6 168,5 171,0 168,0 173,2 182,2
E47 180,2 180,2 181,5 178,1 182,9 189,8
E48 188,3 188,3 189,5 186,8 194,5 203,1
E49 164,6 164,6 166,1 163,2 169,3 177,0
E50 194,4 194,4 196,4 196,5 202,4 210,0
E51 178,3 178,3 179,4 177,5 183,9 193,4
E52 183,5 183,5 183,7 182,3 187,7 196,9
E53 174,7 174,7 176,8 175,5 181,9 189,5
E54 171,0 171,0 172,7 169,4 176,3 181,7
E55 176,0 176,0 176,9 174,7 181,2 188,9
E56 173,8 173,8 177,2 172,6 179,9 185,1
E57 173,5 173,5 176,9 175,5 181,7 185,2
E58 173,1 173,1 177,4 178,1 183,9 185,5
E59 187,7 187,7 189,3 187,7 195,6 207,3
E60 176,3 176,3 179,0 178,7 185,5 191,2
S01 173,2 173,2 175,4 155,5 162,9 171,7
S02 153,6 153,6 155,9 148,9 156,4 163,8
S03 171,6 171,6 172,7 165,6 172,0 181,0
S04 182,8 182,8 184,0 178,7 186,9 193,4
S05 189,5 189,5 190,3 184,5 189,9 201,9
S06 178,1 178,1 178,5 170,0 179,9 187,6
S07 206,9 206,9 206,7 203,7 212,8 221,6
S08 173,9 173,9 176,0 170,1 179,3 186,8
S09 188,7 188,7 189,3 182,1 191,3 200,0
S10 176,7 176,7 178,0 170,4 177,8 190,9
S11 228,5 228,5 227,0 222,0 226,8 246,3
S12 229,3 229,2 228,0 223,9 228,2 248,1
S13 203,5 203,4 203,3 201,6 209,3 220,1
S14 189,4 189,4 191,2 192,0 197,9 207,5
S15 205,4 205,4 204,7 201,3 206,9 221,4
S16 233,9 233,8 232,4 227,7 233,6 251,2
S17 167,0 167,0 169,7 167,4 173,9 180,9
S18 180,8 180,7 180,2 172,3 177,2 184,7
S19 167,9 167,9 168,7 162,0 168,5 177,8
S20 173,2 173,2 172,9 169,3 175,1 185,0
S21 176,8 176,7 178,2 172,2 176,2 184,6
S22 196,3 196,3 195,4 189,7 195,9 207,3
S23 163,5 163,5 165,8 162,2 169,3 177,0
S24 139,6 139,6 141,7 136,2 143,4 149,5
S25 162,8 162,8 165,2 155,9 163,7 172,5
S26 124,4 124,4 125,8 118,9 124,2 132,3
S27 126,4 126,4 127,5 119,2 125,9 134,1
S28 136,8 136,8 137,6 130,6 135,0 145,8
S29 149,0 149,0 151,4 144,3 149,7 158,5
S30 162,8 162,8 164,0 151,8 161,0 168,3
S31 162,8 162,8 164,0 151,8 161,0 168,3
S32 187,4 187,4 187,7 180,7 191,1 201,4
S33 158,6 158,6 161,3 153,4 162,5 170,2
S34 196,2 196,2 197,6 192,5 202,2 212,6
S35 202,4 202,3 203,0 196,9 201,7 213,6
S36 167,8 167,8 173,1 168,3 174,8 180,6
S37 165,0 165,0 169,8 158,2 168,0 172,4
S38 169,7 169,7 172,0 167,1 174,1 186,2
S39 152,7 152,7 155,2 147,2 153,9 167,7
S40 178,2 178,2 181,2 180,8 184,4 202,0
S41 162,0 162,0 165,6 159,4 165,2 171,2
S42 171,2 171,1 172,3 162,4 170,4 178,0
S43 170,7 170,7 173,1 169,2 173,5 185,3
S44 226,6 226,6 225,9 220,0 224,6 236,6
S45 157,2 157,2 159,1 146,9 153,1 164,1
S46 166,6 166,5 169,3 159,9 167,8 178,2
S47 178,6 178,6 181,6 172,7 180,8 189,3
S48 164,0 164,0 166,0 160,8 167,9 175,6
S49 162,3 162,3 163,8 155,7 163,1 169,6
S50 194,5 194,4 196,6 194,2 205,9 219,1
S51 185,4 185,4 187,8 181,3 189,5 193,2
S52 185,3 185,2 186,6 184,1 191,8 199,5
S53 163,3 163,3 164,7 152,4 160,6 169,0
S54 120,9 120,9 122,7 110,5 121,0 131,6
S55 141,2 141,2 142,8 134,6 142,6 151,1
V01 175,1 175,1 176,3 173,8 179,0 188,3
V02 162,7 162,7 165,5 164,5 169,9 174,8
V03 109,5 109,5 110,5 100,4 108,2 113,8
V04 173,4 173,4 176,7 176,7 183,7 186,6
V05 163,3 163,2 165,8 163,4 169,6 175,6
V06 169,2 169,2 172,2 173,0 178,2 182,4
V07 163,4 163,4 166,2 165,0 170,8 176,3
V08 146,4 146,4 148,2 139,9 147,7 153,7
V09 177,2 177,1 179,5 174,1 182,6 188,7
V10 190,8 190,7 191,2 187,8 193,5 203,1
V11 156,1 156,1 159,5 151,9 159,3 164,7
V12 178,2 178,2 181,3 178,7 185,4 194,5
V13 161,9 161,9 165,8 159,4 166,0 174,5
V14 160,6 160,6 163,6 162,3 168,0 176,5
V15 173,7 173,7 175,2 170,3 178,1 190,7
V16 174,2 174,2 176,1 172,5 179,4 187,5
V17 162,1 162,1 163,7 153,9 160,9 171,8
V18 177,1 177,1 178,8 176,5 182,2 190,8
V19 167,6 167,6 169,9 166,8 173,0 180,0
V20 172,8 172,8 175,3 171,8 178,7 186,0
V21 164,6 164,6 167,2 158,9 166,5 171,3
V22 169,3 169,2 173,7 167,5 175,4 179,1
V23 183,3 183,3 184,5 176,8 184,9 192,3
V24 198,7 198,7 199,9 198,1 203,9 214,8
V25 180,4 180,4 183,1 179,1 186,2 194,0
V26 167,5 167,4 168,9 163,5 170,7 178,4
V27 175,9 175,8 178,0 175,3 181,0 189,5
V28 165,4 165,4 167,0 159,4 166,0 176,9
V29 169,1 169,1 173,0 172,4 179,7 185,4
V30 181,0 180,9 182,1 178,9 184,8 194,8

172,71 172,69 174,51 169,37 176,28 184,54
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 23 August, 2005, 09:59:16 AM
Quote
Quote
If a (the!) trained person tests samples, the results will be more consistent than you carry out a test with a bigger group which includes "deaf" people, who just lower the p-values, who don't add any significance.


Actually the deaf group adds significance, as the world isn't populated only by golden ears.
[a href="index.php?act=findpost&pid=322047"][{POST_SNAPBACK}][/a]

True, but what user meant is -I suppose- that results of people unable to tell a difference are not really helpful if the purpose of the test is to know how the difference is between these encoders.
A lot of 5.0 everywhere are telling us that these encoders are transparent to a lot of people (I guess that there's no need to conduce such tests to confirm that ), but not how different are these encoders. That's why critical listeners could be needed here.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: rjamorim on 23 August, 2005, 10:52:28 AM
Quote
True, but what user meant is -I suppose- that results of people unable to tell a difference are not really helpful if the purpose of the test is to know how the difference is between these encoders.
A lot of 5.0 everywhere are telling us that these encoders are transparent to a lot of people (I guess that there's no need to conduce such tests to confirm that ), but not how different are these encoders. That's why critical listeners could be needed here.
[a href="index.php?act=findpost&pid=322053"][{POST_SNAPBACK}][/a]


Indeed. Deaf users are good to know how formats sound to everybody. But not necessarily how formats sound against each other.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: DARcode on 30 August, 2005, 10:27:20 AM
Awesome piece of work and information, simply awesome.

Thanks a lot.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: guruboolez on 31 August, 2005, 06:19:20 AM
Here is the bitrate table (based on 150 full tracks):


Code: [Select]
FILE	faac	Nero	LAME	MPC	Vorbis

A01 175,0 199,2 194,7 164,1 201,7
A02 241,6 156,8 210,8 192,2 206,2
A03 230,6 165,7 230,3 205,1 293,1
A04 203,5 176,1 217,8 188,2 207,3
A05 183,3 161,9 175,8 177,9 182,6
E01 151,3 183,6 135,2 121,8 154,7
E02 179,1 185,6 182,0 186,7 178,8
E03 190,9 179,4 187,0 188,8 184,3
E04 172,9 179,8 178,0 167,6 174,8
E05 215,8 159,5 188,6 167,2 194,4
E06 173,6 186,9 188,9 187,2 179,0
E07 163,6 189,9 180,8 178,2 168,7
E08 182,4 147,8 191,5 168,0 199,4
E09 176,3 187,2 181,4 188,3 178,0
E10 176,4 170,1 167,9 173,3 166,7
E11 211,3 175,8 201,6 188,3 188,3
E12 163,4 184,1 169,5 157,9 167,3
E13 169,6 176,8 172,6 193,3 184,5
E14 163,6 190,7 184,9 185,2 176,5
E15 159,9 198,5 180,5 190,8 176,5
E16 188,0 181,6 194,0 199,7 202,3
E17 188,2 180,7 196,2 178,2 185,7
E18 169,5 197,0 174,3 157,4 177,6
E19 167,9 180,4 167,0 155,2 168,3
E20 166,0 180,2 179,4 157,3 161,8
E21 166,8 185,7 181,3 162,6 170,4
E22 176,5 182,8 187,9 187,6 177,1
E23 158,5 181,8 182,4 172,8 163,0
E24 161,9 189,5 175,9 169,1 163,3
E25 191,3 178,6 194,4 188,6 182,6
E26 176,3 178,0 184,5 174,6 171,3
E27 178,8 175,5 184,9 180,8 168,4
E28 178,8 183,0 191,4 159,6 183,8
E29 189,4 166,1 177,0 166,9 170,9
E30 183,7 184,2 196,9 175,9 174,9
E31 222,0 154,1 192,0 201,4 199,6
E32 190,2 180,9 186,9 204,7 196,8
E33 187,2 162,5 177,1 185,7 183,2
E34 160,6 181,1 174,6 168,8 167,5
E35 186,6 169,0 181,3 220,9 206,2
E36 208,6 173,5 187,3 216,3 216,5
E37 172,0 183,7 175,6 193,7 180,3
E38 174,3 183,6 182,5 187,1 177,2
E39 174,3 184,2 184,0 186,3 188,3
E40 163,7 181,9 170,6 168,6 157,1
E41 175,6 169,9 174,7 225,0 197,2
E42 182,5 177,6 174,3 220,6 190,6
E43 188,7 178,0 188,0 205,4 188,4
E44 170,0 167,8 180,8 178,9 167,9
E45 218,6 163,5 194,6 219,2 212,2
E46 185,3 178,0 187,9 182,2 179,7
E47 177,5 192,4 187,6 189,8 193,3
E48 182,0 182,8 187,7 203,1 187,5
E49 171,2 179,2 180,5 177,0 175,1
E50 226,3 179,6 214,9 210,0 211,6
E51 180,3 180,6 187,9 193,4 185,1
E52 182,3 176,4 186,0 196,9 188,7
E53 189,6 174,4 190,0 189,5 185,2
E54 173,6 177,1 183,4 181,7 179,0
E55 175,1 184,8 188,9 188,9 181,6
E56 213,3 163,2 186,4 185,1 184,4
E57 214,6 165,5 195,5 185,2 181,7
E58 206,1 177,5 202,4 185,5 181,9
E59 189,0 166,4 184,5 207,3 199,1
E60 202,4 173,8 193,0 191,2 203,9
S01 175,7 134,1 138,7 171,7 153,4
S02 173,1 175,1 173,4 163,8 166,1
S03 172,4 185,9 181,0 181,0 165,9
S04 175,0 172,1 180,1 193,4 174,0
S05 169,5 172,1 178,0 201,9 182,9
S06 183,4 166,3 166,8 187,6 173,7
S07 181,6 175,8 179,8 221,6 192,5
S08 188,3 174,8 181,3 186,8 171,0
S09 185,8 174,5 178,0 200,0 180,5
S10 230,6 158,1 185,5 190,9 181,7
S11 204,8 166,8 185,4 246,3 224,4
S12 217,5 160,1 185,0 248,1 232,2
S13 246,6 160,2 206,6 220,1 226,7
S14 229,1 162,5 216,5 207,5 226,6
S15 221,2 166,1 193,1 221,4 214,3
S16 204,5 164,6 174,6 251,2 220,2
S17 172,9 174,9 188,9 180,9 175,9
S18 161,0 175,0 184,8 184,7 169,5
S19 160,6 190,0 180,2 177,8 168,6
S20 164,6 188,5 204,3 185,0 174,5
S21 176,3 172,5 175,6 184,6 169,6
S22 171,9 172,0 181,6 207,3 189,6
S23 163,0 198,4 190,3 177,0 175,3
S24 176,2 182,3 174,3 149,5 176,7
S25 158,6 166,9 182,0 172,5 165,9
S26 151,9 189,4 150,4 132,3 156,8
S27 156,6 176,1 161,4 134,1 163,2
S28 153,4 198,1 151,9 145,8 163,0
S29 157,9 196,6 175,9 158,5 168,0
S30 161,3 171,6 141,7 168,3 156,6
S31 161,3 171,6 141,7 168,3 156,6
S32 187,8 168,6 159,2 201,4 192,2
S33 188,1 169,2 173,8 170,2 169,9
S34 165,3 175,0 185,7 212,6 178,7
S35 197,1 166,5 178,3 213,6 190,5
S36 152,9 197,1 177,2 180,6 180,6
S37 176,6 171,9 175,2 172,4 191,6
S38 163,8 194,2 193,0 186,2 185,7
S39 162,9 194,7 177,3 167,7 171,1
S40 172,1 180,1 184,9 202,0 196,2
S41 183,2 165,0 180,4 171,2 182,9
S42 164,1 163,9 174,8 178,0 163,3
S43 159,7 182,6 185,3 185,3 179,0
S44 193,3 169,0 157,4 236,6 205,3
S45 154,0 170,9 157,0 164,1 150,8
S46 164,0 169,1 172,0 178,2 165,9
S47 175,1 167,8 176,9 189,3 174,3
S48 163,3 190,7 180,2 175,6 167,7
S49 165,1 177,3 174,0 169,6 170,2
S50 186,6 165,1 182,2 219,1 194,1
S51 166,3 164,1 183,0 193,2 175,3
S52 190,5 178,7 183,2 199,5 182,8
S53 160,4 160,8 140,4 169,0 162,5
S54 163,9 158,7 146,5 131,6 162,4
S55 161,5 173,4 152,6 151,1 166,6
V01 172,7 184,1 188,1 188,3 179,8
V02 165,1 189,9 194,8 174,8 169,9
V03 171,1 172,0 141,5 113,8 147,0
V04 214,6 169,8 202,8 186,6 198,0
V05 171,3 186,0 189,1 175,6 174,3
V06 184,3 185,8 192,7 182,4 182,8
V07 175,5 181,8 199,8 176,3 171,5
V08 172,2 173,7 178,6 153,7 165,5
V09 180,2 171,6 173,3 188,7 175,7
V10 183,7 172,2 181,3 203,1 179,1
V11 168,5 178,9 178,8 164,7 154,9
V12 175,5 178,0 192,4 194,5 182,0
V13 183,1 177,1 189,3 174,5 171,4
V14 167,6 186,6 198,5 176,5 173,2
V15 173,4 178,1 190,9 190,7 179,2
V16 164,1 180,3 189,5 187,5 169,6
V17 177,3 185,1 187,3 171,8 174,6
V18 182,6 181,5 188,0 190,8 187,2
V19 179,1 168,9 180,2 180,0 170,2
V20 191,0 175,2 191,8 186,0 204,4
V21 194,9 163,7 178,2 171,3 175,1
V22 185,3 174,2 180,9 179,1 176,9
V23 165,3 178,9 181,5 192,3 168,7
V24 232,5 162,0 195,8 214,8 219,7
V25 192,8 163,2 184,1 194,0 180,9
V26 184,6 173,0 172,6 178,4 173,7
V27 196,9 172,5 186,6 189,5 182,4
V28 167,7 184,6 180,2 176,9 170,5
V29 189,0 177,9 187,6 185,4 178,3
V30 181,9 182,8 184,8 194,8 178,9

180,92 176,14 181,79 184,54 181,48

faac Nero LAME MPC Vorbis

Graphicaly and sorted by size:

 
I tried with older version of aacenc32.dll: identical - identical - identical... 
In short, -streaming 'high' encodings are bit-to-bit identical from aacenc32.dll version 2.9.9.999 (dated from 2004.12.22) to 3.2.0.20! The difference start to appear with 2.9.9.998 dated from 2004.11.27. In other words, no progress in this area since eight months. What happened with Nero Digital team? Have Ivan only worked on SBR and low bitrate stuff during this time?

examples:
files are not available anymore for download
Here is the bitrate table (based on 150 full tracks):


Code: [Select]
FILE	faac	Nero	LAME	MPC	Vorbis

A01 175,0 199,2 194,7 164,1 201,7
A02 241,6 156,8 210,8 192,2 206,2
A03 230,6 165,7 230,3 205,1 293,1
A04 203,5 176,1 217,8 188,2 207,3
A05 183,3 161,9 175,8 177,9 182,6
E01 151,3 183,6 135,2 121,8 154,7
E02 179,1 185,6 182,0 186,7 178,8
E03 190,9 179,4 187,0 188,8 184,3
E04 172,9 179,8 178,0 167,6 174,8
E05 215,8 159,5 188,6 167,2 194,4
E06 173,6 186,9 188,9 187,2 179,0
E07 163,6 189,9 180,8 178,2 168,7
E08 182,4 147,8 191,5 168,0 199,4
E09 176,3 187,2 181,4 188,3 178,0
E10 176,4 170,1 167,9 173,3 166,7
E11 211,3 175,8 201,6 188,3 188,3
E12 163,4 184,1 169,5 157,9 167,3
E13 169,6 176,8 172,6 193,3 184,5
E14 163,6 190,7 184,9 185,2 176,5
E15 159,9 198,5 180,5 190,8 176,5
E16 188,0 181,6 194,0 199,7 202,3
E17 188,2 180,7 196,2 178,2 185,7
E18 169,5 197,0 174,3 157,4 177,6
E19 167,9 180,4 167,0 155,2 168,3
E20 166,0 180,2 179,4 157,3 161,8
E21 166,8 185,7 181,3 162,6 170,4
E22 176,5 182,8 187,9 187,6 177,1
E23 158,5 181,8 182,4 172,8 163,0
E24 161,9 189,5 175,9 169,1 163,3
E25 191,3 178,6 194,4 188,6 182,6
E26 176,3 178,0 184,5 174,6 171,3
E27 178,8 175,5 184,9 180,8 168,4
E28 178,8 183,0 191,4 159,6 183,8
E29 189,4 166,1 177,0 166,9 170,9
E30 183,7 184,2 196,9 175,9 174,9
E31 222,0 154,1 192,0 201,4 199,6
E32 190,2 180,9 186,9 204,7 196,8
E33 187,2 162,5 177,1 185,7 183,2
E34 160,6 181,1 174,6 168,8 167,5
E35 186,6 169,0 181,3 220,9 206,2
E36 208,6 173,5 187,3 216,3 216,5
E37 172,0 183,7 175,6 193,7 180,3
E38 174,3 183,6 182,5 187,1 177,2
E39 174,3 184,2 184,0 186,3 188,3
E40 163,7 181,9 170,6 168,6 157,1
E41 175,6 169,9 174,7 225,0 197,2
E42 182,5 177,6 174,3 220,6 190,6
E43 188,7 178,0 188,0 205,4 188,4
E44 170,0 167,8 180,8 178,9 167,9
E45 218,6 163,5 194,6 219,2 212,2
E46 185,3 178,0 187,9 182,2 179,7
E47 177,5 192,4 187,6 189,8 193,3
E48 182,0 182,8 187,7 203,1 187,5
E49 171,2 179,2 180,5 177,0 175,1
E50 226,3 179,6 214,9 210,0 211,6
E51 180,3 180,6 187,9 193,4 185,1
E52 182,3 176,4 186,0 196,9 188,7
E53 189,6 174,4 190,0 189,5 185,2
E54 173,6 177,1 183,4 181,7 179,0
E55 175,1 184,8 188,9 188,9 181,6
E56 213,3 163,2 186,4 185,1 184,4
E57 214,6 165,5 195,5 185,2 181,7
E58 206,1 177,5 202,4 185,5 181,9
E59 189,0 166,4 184,5 207,3 199,1
E60 202,4 173,8 193,0 191,2 203,9
S01 175,7 134,1 138,7 171,7 153,4
S02 173,1 175,1 173,4 163,8 166,1
S03 172,4 185,9 181,0 181,0 165,9
S04 175,0 172,1 180,1 193,4 174,0
S05 169,5 172,1 178,0 201,9 182,9
S06 183,4 166,3 166,8 187,6 173,7
S07 181,6 175,8 179,8 221,6 192,5
S08 188,3 174,8 181,3 186,8 171,0
S09 185,8 174,5 178,0 200,0 180,5
S10 230,6 158,1 185,5 190,9 181,7
S11 204,8 166,8 185,4 246,3 224,4
S12 217,5 160,1 185,0 248,1 232,2
S13 246,6 160,2 206,6 220,1 226,7
S14 229,1 162,5 216,5 207,5 226,6
S15 221,2 166,1 193,1 221,4 214,3
S16 204,5 164,6 174,6 251,2 220,2
S17 172,9 174,9 188,9 180,9 175,9
S18 161,0 175,0 184,8 184,7 169,5
S19 160,6 190,0 180,2 177,8 168,6
S20 164,6 188,5 204,3 185,0 174,5
S21 176,3 172,5 175,6 184,6 169,6
S22 171,9 172,0 181,6 207,3 189,6
S23 163,0 198,4 190,3 177,0 175,3
S24 176,2 182,3 174,3 149,5 176,7
S25 158,6 166,9 182,0 172,5 165,9
S26 151,9 189,4 150,4 132,3 156,8
S27 156,6 176,1 161,4 134,1 163,2
S28 153,4 198,1 151,9 145,8 163,0
S29 157,9 196,6 175,9 158,5 168,0
S30 161,3 171,6 141,7 168,3 156,6
S31 161,3 171,6 141,7 168,3 156,6
S32 187,8 168,6 159,2 201,4 192,2
S33 188,1 169,2 173,8 170,2 169,9
S34 165,3 175,0 185,7 212,6 178,7
S35 197,1 166,5 178,3 213,6 190,5
S36 152,9 197,1 177,2 180,6 180,6
S37 176,6 171,9 175,2 172,4 191,6
S38 163,8 194,2 193,0 186,2 185,7
S39 162,9 194,7 177,3 167,7 171,1
S40 172,1 180,1 184,9 202,0 196,2
S41 183,2 165,0 180,4 171,2 182,9
S42 164,1 163,9 174,8 178,0 163,3
S43 159,7 182,6 185,3 185,3 179,0
S44 193,3 169,0 157,4 236,6 205,3
S45 154,0 170,9 157,0 164,1 150,8
S46 164,0 169,1 172,0 178,2 165,9
S47 175,1 167,8 176,9 189,3 174,3
S48 163,3 190,7 180,2 175,6 167,7
S49 165,1 177,3 174,0 169,6 170,2
S50 186,6 165,1 182,2 219,1 194,1
S51 166,3 164,1 183,0 193,2 175,3
S52 190,5 178,7 183,2 199,5 182,8
S53 160,4 160,8 140,4 169,0 162,5
S54 163,9 158,7 146,5 131,6 162,4
S55 161,5 173,4 152,6 151,1 166,6
V01 172,7 184,1 188,1 188,3 179,8
V02 165,1 189,9 194,8 174,8 169,9
V03 171,1 172,0 141,5 113,8 147,0
V04 214,6 169,8 202,8 186,6 198,0
V05 171,3 186,0 189,1 175,6 174,3
V06 184,3 185,8 192,7 182,4 182,8
V07 175,5 181,8 199,8 176,3 171,5
V08 172,2 173,7 178,6 153,7 165,5
V09 180,2 171,6 173,3 188,7 175,7
V10 183,7 172,2 181,3 203,1 179,1
V11 168,5 178,9 178,8 164,7 154,9
V12 175,5 178,0 192,4 194,5 182,0
V13 183,1 177,1 189,3 174,5 171,4
V14 167,6 186,6 198,5 176,5 173,2
V15 173,4 178,1 190,9 190,7 179,2
V16 164,1 180,3 189,5 187,5 169,6
V17 177,3 185,1 187,3 171,8 174,6
V18 182,6 181,5 188,0 190,8 187,2
V19 179,1 168,9 180,2 180,0 170,2
V20 191,0 175,2 191,8 186,0 204,4
V21 194,9 163,7 178,2 171,3 175,1
V22 185,3 174,2 180,9 179,1 176,9
V23 165,3 178,9 181,5 192,3 168,7
V24 232,5 162,0 195,8 214,8 219,7
V25 192,8 163,2 184,1 194,0 180,9
V26 184,6 173,0 172,6 178,4 173,7
V27 196,9 172,5 186,6 189,5 182,4
V28 167,7 184,6 180,2 176,9 170,5
V29 189,0 177,9 187,6 185,4 178,3
V30 181,9 182,8 184,8 194,8 178,9

180,92 176,14 181,79 184,54 181,48

faac Nero LAME MPC Vorbis

Graphicaly and sorted by size:

 
I tried with older version of aacenc32.dll: identical - identical - identical... 
In short, -streaming 'high' encodings are bit-to-bit identical from aacenc32.dll version 2.9.9.999 (dated from 2004.12.22) to 3.2.0.20! The difference start to appear with 2.9.9.998 dated from 2004.11.27. In other words, no progress in this area since eight months. What happened with Nero Digital team? Have Ivan only worked on SBR and low bitrate stuff during this time?

examples:
files are not available anymore for download
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: esa372 on 31 August, 2005, 10:26:57 AM
Wow!
(https://hydrogenaud.io/imgcache.php?id=ba248cedeccd7d8202cd877c47c06a18" rel="cached" data-warn="External image, click to view at original size" data-url="http://66.49.140.133/assets/icon/clap.gif)
Thank you, guruboolez!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Pio2001 on 01 September, 2005, 05:34:15 PM
Thank you for the test.
I hope it will encourage other people to perform tests at high bitrates, which have been long considered as out of reach.
I'd like to test some metal samples with the same encoders but I'm so short of time in order to do all what I'd like to do. In fact, I've got a lot of time, but too much interesting things to do !

To remarks about statistics : MPC and Vobris seem tied, but the results don't take into account the ABX results, which adds a lot of significance. Maybe if we could analyse all the data, some encoders that look tied wouldn't be so anymore.
And I think that Friedemann's method can't be used here, because, if I'm not mistaken, it assumes that no encoder can be tied.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: bug80 on 01 September, 2005, 05:50:58 PM
Quote
And I think that Friedemann's method can't be used here, because, if I'm not mistaken, it assumes that no encoder can be tied.
[a href="index.php?act=findpost&pid=324341"][{POST_SNAPBACK}][/a]

I believe that is correct. Friedmann can be used in the case of ties, but some modifications to the calculation of the F-value have to be made, if I remember it correctly.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: moozooh on 01 September, 2005, 07:58:53 PM
I feel great urge to thank Guruboolez and Aoyumi for their very different, yet so coherent work on Vorbis. The next big thing is aoTuV beta 5 (or will it be the official release, like 1.2 or something?), which I hope will reduce some ugly noise issues (I can upload a few samples which are ABX-able @ -q5 and, supposedly, -q6).

Guru, your tests will always be greatly appreciated by me and my friends. Thank you, again.

Quote
What I'm trying to say is that, unlike most other genres, classical samples can be very different and so when you have a thorough test like this one, you can get a pretty good overall picture of each codec (encoder).
[a href="index.php?act=findpost&pid=321772"][{POST_SNAPBACK}][/a]

It depends on what do you call a difference. Music can be very different even while being in the same genre, let alone “guitar” or “electronic” music…

Quote
In fact, I haven't visited HA for a while but once I saw a listening test done by guruboolez, I headed straight for this thread
[a href="index.php?act=findpost&pid=321901"][{POST_SNAPBACK}][/a]

Hey, me too!  I've just returned from my vacation, heh.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: alter4 on 02 September, 2005, 01:14:48 AM
And in my opinion, the handling pre-echo mechanism which entered in beta3  are not excellent (this conclusion I made in last week after some tests with killer samples like castanets2)
Waiting for next great work of Aoyumi 
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Yaztromo on 02 September, 2005, 05:34:42 AM
Quote
And in my opinion, the handling pre-echo mechanism which entered in beta3  are not excellent (this conclusion I made in last week after some tests with killer samples like castanets2)
Waiting for next great work of Aoyumi 
[a href="index.php?act=findpost&pid=324395"][{POST_SNAPBACK}][/a]


I came to the same conclusion with beta3 using castanets and hihat. Beta2 does a better job at handling pre-echo in these extreme samples.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: ff123 on 02 September, 2005, 04:37:59 PM
Quote
Quote
And I think that Friedemann's method can't be used here, because, if I'm not mistaken, it assumes that no encoder can be tied.
[a href="index.php?act=findpost&pid=324341"][{POST_SNAPBACK}][/a]

I believe that is correct. Friedmann can be used in the case of ties, but some modifications to the calculation of the F-value have to be made, if I remember it correctly.
[a href="index.php?act=findpost&pid=324350"][{POST_SNAPBACK}][/a]


I believe I made the required modifications (it's been a long while since I wrote the program) to take into account ties for a Friedman analysis

Consider 3 codecs, 2 of them tied for 1st, and the 3rd last.  The rankings would be valued at 2.5, 2.5, 1

BTW, Guru's graph shows that he used parametric analyses (Fisher's LSD and Tukey's HSD), not a Friedman anlysis.  The executable which runs all the analyses is named friedman.exe, though, because the friedman was the first one I wrote.

ff123
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Aoyumi on 02 September, 2005, 11:22:29 PM
Quote
And in my opinion, the handling pre-echo mechanism which entered in beta3 are not excellent (this conclusion I made in last week after some tests with killer samples like castanets2)

Can you exhibit the log of concrete ABX and HR test?  It is more good if there is a detailed comment. In my test range(example - some castanets samples), a result worse than 1.1.0(1) hardly comes out, and, in many cases, the way of beta4 is accepted.

Quote
I came to the same conclusion with beta3 using castanets and hihat. Beta2 does a better job at handling pre-echo in these extreme samples.

Were you tested by beta4 in them?  hihat of -q4 and castanets2 of -q6 have improved from beta3.

EDIT:An expression is corrected.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: alter4 on 05 September, 2005, 03:17:46 AM
I tested castanets2 with beta4
Now I have no ABX log, but I remember result in castanets2 at q5

wav vs beta4 - 21/25

wav vs 1.1.1 - 15/25

I have the screenshots from my visual analyse encoded samples
I can mail it to you.

edit: my home acoustic is poor and q5 near transparent for me
but beta4 pre-echo is large...
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Yaztromo on 05 September, 2005, 05:02:53 AM
I will test with beta4 in the next couple of days.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: alter4 on 08 September, 2005, 04:52:44 AM
(https://hydrogenaud.io/imgcache.php?id=61f6a677815494a98240d3019faac720" rel="cached" data-warn="External image, click to view at original size" data-url="http://foto.mail.ru/mail/alter4/1/i-4.jpg)
(https://hydrogenaud.io/imgcache.php?id=d913c4e96fb2be25f6cbc19416806f52" rel="cached" data-warn="External image, click to view at original size" data-url="http://foto.mail.ru/mail/alter4/1/i-3.jpg)
(https://hydrogenaud.io/imgcache.php?id=169868d042580f0af1f82304e29b5019" rel="cached" data-warn="External image, click to view at original size" data-url="http://foto.mail.ru/mail/alter4/1/i-2.jpg)
the first click in castanets2, no comment!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Aoyumi on 08 September, 2005, 09:04:47 AM
Quote
the first click in castanets2, no comment!


Generally it is not so meaningful to take up the graph of a specific portion.
It may become the origin which causes misapprehension.
Here is an interesting example (http://www.geocities.jp/aoyoume/lt/preecho-b4.html).

1.1.1 has for me pre-echo irritated rather than beta4.
Graph does not irritate me. 
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: alter4 on 08 September, 2005, 10:23:18 AM
My friends (I ask theirs to help) say that in castanets2 aotuv beta4 is better (aotuv have less pre-echo)
but for me pre-echo is more audible in beta4, my friend say that I have anomalous
ear constitution  . In all cases, Aoyumi thank for great job!!!

P.S. Waiting for Yaztromo tests.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Aoyumi on 11 September, 2005, 02:45:21 AM
>alter4
Can the portion which you ABX(ed) be shown?  Or how is it in another portion?
Such information becomes reference of tuning although I cannot promise an improvement. 
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: liekloo on 15 September, 2005, 05:35:17 PM
I know I'm late. Very late even. But I don't care - Guruboolez man, what a test!  and what a result regarding Vorbis (and LAME)
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: zima on 16 September, 2005, 02:35:50 PM
Very impressive...I wish I had sometimes this kind of determination to finish some of my projects...

BTW, I wonder...
This test was made on classical music. And AFAI remember, classical, at least for mp3, was the easiest scenario. Is it true? If yes than is it possible that in case of other formats thisis also the case?

And a question totally againt HA rules (so answer in spirit of "possible", "not sure", etc. would be OK...): perhaps MPC still is on top when it comes to more demading music, industrial for example and such...
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: beto on 16 September, 2005, 04:02:07 PM
Quote
I know I'm late. Very late even. But I don't care - Guruboolez man, what a test!  and what a result regarding Vorbis (and LAME)
[a href="index.php?act=findpost&pid=327100"][{POST_SNAPBACK}][/a]


Me too. I was trying to digest all the impressive amount of information 
Thanks a million for the professionalism and dense information. Kudos to you!!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Seymour on 17 September, 2005, 09:39:50 AM
Guruboolez
Great test!  Thanks. Surely I'll take its results in mind. But for me Musepack is enough. I don't think that I'll get much changing from MPC --standard to Lancer beta 4 at corresponding bitrate...
Aoyumi
Regardless of my decisions you're doing it all right! Thank you too!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: VEG on 22 September, 2005, 04:09:28 PM
Wow! Vorbis bested that MPC on 180kbps?! It's good news!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: kornchild2002 on 22 September, 2005, 05:40:12 PM
Thanks for the testing.

I can't help but think how the iTunes mpeg-4 AAC 160kbps VBR format tests up against these other encoders.  Maybe someone will do a test using iTunes 5 and QuickTime 7.  Just curious, why couldn't a 160kbps mpeg-4 AAC file be compared to others?  I know the bitrate is lower than the others but does this cause a flaw in the tests?
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: stephanV on 23 September, 2005, 03:42:53 AM
Quote
Thanks for the testing.

I can't help but think how the iTunes mpeg-4 AAC 160kbps VBR format tests up against these other encoders.  Maybe someone will do a test using iTunes 5 and QuickTime 7.  Just curious, why couldn't a 160kbps mpeg-4 AAC file be compared to others?  I know the bitrate is lower than the others but does this cause a flaw in the tests?
[a href="index.php?act=findpost&pid=328914"][{POST_SNAPBACK}][/a]

It makes the test more difficult to interpretate. If the codec tested at 160 kbps does worse than the others at higher bit rates (there is a reasonable chance this would happen), then one could always bring up the argument "but it used a lower bit rate", rendering the results meaningless. On the other hand, if the codec tested at 160 kpbs would do equal or better than the ones tested at higher bit rates, logic suggests that it would also do equal or better when tested at that same higher bit rate. So there is no reason to use 160 kbps.

Taking in account the amount of time such a test takes, and the large risk of getting an unuseable result, it would be better to test all codecs at the same bit rate.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Donunus on 23 September, 2005, 06:37:53 AM
I was just wondering if upping lame 3.97 to preset extreme or preset fast extreme will give enough of an improvement to beat mpc and vorbis at their current settings in this test with all test samples. pls comment on this
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: beto on 23 September, 2005, 07:23:21 AM
probably yes. but that's an assumption.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: SCIF on 28 October, 2005, 12:47:58 AM
Does anybody with good headphones want to abx-test ogg/mpc/mp3 on rock/metal material. I think power-metal(like Rhapsody), sympho-metal(Dimmu Borgir-Perfection or Vanity) materials aren't easycompress.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: de Mon on 28 October, 2005, 03:18:34 AM
Quote
Does anybody with good headphones want to abx-test ogg/mpc/mp3 on rock/metal material. I think power-metal(like Rhapsody), sympho-metal(Dimmu Borgir-Perfection or Vanity) materials aren't easycompress.
[a href="index.php?act=findpost&pid=337750"][{POST_SNAPBACK}][/a]


I bet Guru will not test it. His ears will not bear such genre 
Anyway I would like to see such test. There is more metal than classics in my audio collection.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: kuniklo on 31 October, 2005, 04:39:17 PM
I just came across this thread today.  Thanks to Guru for his dedicated testing and to Aoyumi for his improvments to vorbis!
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: SCIF on 02 November, 2005, 12:31:45 AM
Quote
I bet Guru will not test it. His ears will not bear such genre 
Yes. He told me so.
Quote
Anyway I would like to see such test. There is more metal than classics in my audio collection.

I'm not alone $)
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: lextune on 06 January, 2006, 03:37:38 PM
Thanks for this interesting and informative post. It was very helpful to me
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: Garf on 09 January, 2006, 06:13:38 AM
Re-opening so discussion can continue.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: vinnie97 on 09 January, 2006, 05:38:22 PM
I saw this thread was locked yesterday with no explanation why...was wondering if it was an MPC zealot who managed to gain mod control.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: kode54 on 09 January, 2006, 05:49:15 PM
Quote
I saw this thread was locked yesterday with no explanation why...was wondering if it was an MPC zealot who managed to gain mod control.
[a href="index.php?act=findpost&pid=355872"][{POST_SNAPBACK}][/a]


Don't worry, it was opened again because there weren't enough useless replies already. Thanks for your contribution, by the way.
Title: MPC vs VORBIS vs MP3 vs AAC at 180 kbps
Post by: vinnie97 on 09 January, 2006, 06:30:45 PM
No need to be testy.  I didn't see that *other* thread until after responding to this one.