HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: guruboolez on 2005-11-15 08:14:29

Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-15 08:14:29
[span style=\'font-size:14pt;line-height:100%\']Preliminary notes[/span]

Two years ago I performed and published my two (http://www.hydrogenaudio.org/forums/index.php?showtopic=14091&hl=) first listening tests (http://www.hydrogenaudio.org/forums/index.php?showtopic=16395&hl=). Both included different formats and encoders at ~130 kbps and involved a dozen of samples: classical music only. My purpose was to see which encoder was able to produce the best encoding at a friendly bitrate (friendly for portable players), and for a specific kind of music. iTunes AAC & WMAPro appeared to be the best encoders (for myself), and the absolute quality of both encoders at such bitrate surprised me. Last year (December 2004) I performed two similar tests: the first was dedicated to AAC (http://www.hydrogenaudio.org/forums/index.php?showtopic=29924&hl=) (Nero, Apple, old and new encoders) and the second was a match between the best AAC encoder (Nero Digital “fast” VBR) and the most advanced Vorbis one (aoTuV beta 3) (http://www.hydrogenaudio.org/forums/index.php?showtopic=29925&hl=). Quality and enjoyment were even higher!

This year I performed a fresh multiformat listening test at 130 kbps. This new test is very different from their predecessors from a methodological point of view. I progressively improved my approach of listening tests and tried to answered to all criticism addressed in the past to previous tests (and not necessary mine). Consequently, my “personal evaluations” which were first a friendly exercise feasible in one rainy, autumnal afternoon now looks as a gigantic task which took me approximately 10 days (shared with family, friends, job, and discouragement) to complete. I improved several point of the methodology; to sum them up:

diversity : the following test is not only based on “classical” music, and will also include several (fifty!) samples of “modern” music.
grading : described once as “temperamentic (http://www.hydrogenaudio.org/forums/index.php?showtopic=18378&view=findpost&p=181413)” I decided to stick all marks between two anchors, a low and a high one. It will decrease the contrast between different encoders and increase at the same time the difficulty of the full exercise but it should also ensure a more accurate grading. The low anchor is vital to prevent an excessively harsh grading; the high anchor is essential to temper enthusiasm: a very good encoding at 130 kbps should be marked in regard to an excellent and high bitrate one. The presence of both anchors should guarantee a right grading: not too low, not too high.
complexity : people reproached to some listening tests to focus only “critical” or “complex” samples. It may be a problem with some VBR implementations, which sometimes decrease too much the bitrate on “non-complex” samples. In my opinion, a listening test should include both types of samples, at least to verify that non-complex/low bitrate parts are as well encoded as complex/high bitrate ones. Usually, VBR encoders handle very well non-complex part. Usually… The complexity range of my gallery of samples is wide enough to represent all situations (from ultra-low bitrate to ultra high ones) and to check the strength of VBR implementations.
abundance : a bunch of 12…15 samples is maybe not enough to give an accurate idea of the strength and weakness of different encoders. I experienced it myself in the past: my previous tests didn’t reveal some problems I only noticed after on real usage, and more important, they were unable to expose the recurrence of the detected problems. Detecting one problem (like rumbling or ringing) is one thing, measuring the periodicity of this problem is another thing. My test is based on 200 samples; this number should be enough to expose all common problems plus several uncommon ones and is also sufficient to get an idea of their redundancy. This is in my opinion the biggest advantage of my personal listening test over collective ones (which must stay friendly to avoid discouragement and attract a lot of testers).
statistical analysis : it might appear as trivial to mention this, but statistical analysis of results and confidence bars are presents (they were not used last year and the year before).
“Apples and Oranges” : no need to recall the problem. This test only mobilizes VBR encoders. No debate this time.


[span style=\'font-size:14pt;line-height:100%\']THE TEST: CHOICE OF ENCODERS [/span]


The market of audio encoders is ruled by a Darwinian process: the stronger only survive. Between my first test (October 2003 (http://www.hydrogenaudio.org/forums/index.php?showtopic=14091&hl=)) and this one (november 2005), only few encoders really progressed. Most other (some of them are still in use) are unchanged or only changed once: MPC, WMA (Standard and Pro), faac, all MP3 encoders (excepted LAME). Another one appeared and disappeared in the meantime (Compaact!).
On the hardware side, the situation is now very different from the one I lived two years ago. With the exception of one or two devices, AAC and Vorbis support in hardware players were more a dream than reality. Testing different audio formats was useful for a virtual and opened future, rich in dreams and promises. Now, the concrete situation is more interesting than dreams. MP3 and WMA (Std) are still the two well-established formats, but Vorbis now benefits from a growing interest of several manufacturers and if AAC still looks like an Apple monopoly the iPod market has at least mutated into several form (flash memory players, Microdrive™ based jukebox). One victim of reality is WMAPro, still not supported; and the growing popularity of WMA labeled as PlaysForSure (based on WMA Std) seems to sentence WMAPro to a long exile.
For all these considerations, I restricted the test to the most usable and interesting encoders: AAC-LC (highly developed by Apple and Nero Digital), MP3 (vigorous as ever, thanks to LAME devs), Vorbis (saved from inertia by Aoyumi). Besides these four encoders, I add two anchors. More precisely:

Apple AAC: I used iTunes 6.0.0.18 (based on QuickTime 7.03), at 128 kbps and with the recently added VBR mode . I test Apple AAC in VBR for the first time. I sadly discovered that this encoder use the same trick as the MP3 encoder included in iTunes: the minimal size of the frames are not inferior to the targeted bitrate (apart maybe digital silence). In other words, for 128 VBR encodings the bitrate starts at 128 kbps and is increased with complexity. No need to precise that if average bitrate stays close to the target, the variations are necessary limited. One advantage: this restricted mode prevents the VBR engine to use inadequately low bitrate frames, and should guarantee quality from bad surprises compared to a CBR encoding.

Nero Digital AAC: I used the very new encoder released two weeks ago (aac.dll v.3 and aacenc32.dll v.4.2.1.0 ), in VBR mode too. –internet profile is the closed to 128 kbps (slightly inferior with classical music, but higher with non-classical. I didn’t use the “fast” mode, which is now pretty similar but probably inferior to the “high” one.

LAME MP3: I used latest alpha of 3.98 (alpha 2) in order to add the –athaa-sensitivity 1 command to the –V5 --vbr-new mode. For the second group of samples and to slightly lower the bitrate I simply used –V5 –vbr-new.

Vorbis: I used aoTuV beta 4 (4.5 was released during the testing phase) instead of official 1.1.1 which corresponds to the 18 months old aoTuV beta 2 version. I used –q4,25 for the first group and –q4,00 for the second.

As low anchor, I looked for something really low and also usable in batch mode. I found a very old AAC encoder on ReallyRareWares called mbaacencoder version 0.3: it’s awfully slow, quality is terrific and is as anecdote ideal to get an idea of all progress made around AAC between 1999 (release date of mbaaencoder) and 2005 (Apple and Nero Digital). I tried to get joint stereo and LC profile in batch mode, but the encoder apparently stayed in default mode (Main Profile, 128 kbps and dual stereo).

As high anchor, I didn’t hesitate and used LAME 3.97 beta 1 –V2 --vbr new (or --preset standard) which is a reference for efficient, high quality and universal encodings. Furthermore, it would be interesting to evaluate the remaining gap between modern implementation of AAC and Vorbis at ~128 kbps to HQ MP3 at ~192 kbps.





[span style=\'font-size:14pt;line-height:100%\']SAMPLES [/span]

The test hinges on two big groups of samples: 150 for “classical” music group and 50 for “non-classical” (or “various”, or “modern”, or “popular”… choose your own) group. I already used the first group in three different tests in the past (80 kbps, 96 kbps, and LAME –V5). The complete collection is available for download. The 2nd group consist on all (35) non-classical samples used in previous collective listening tests; they’re all available on rarewares. To decrease the gap between the first and the second group I’ve add 15 other samples, all recently submitted for the postponed 64 kbps listening test of Sebastian Mares. Most of these last files may still be available.


[span style=\'font-size:14pt;line-height:100%\']THE BITRATE [/span]

The bitrate comparison is more accurate for the first group: it’s based on full tracks (6min 30 sec. per file on average) instead of short samples (10 sec. on average), and the complete collection is last but not least very representative of my entire library. For the second group of samples, I proceeded differently and I based the bitrate calculation on the 50 samples (which are longer: 24 sec. on average) and on external data (bitrate table for LAME posted by someone else). This way to evaluate the bitrate is not very precise, but I don’t have enough material to build a more accurate bitrate table. That’s why I tried to lower at maximum the difference in bitrate for all settings, and changed the command line for Vorbis (from –q4,25 to –q4,00) and LAME (--athaa-sensitivity 1 was removed).
To sum up the datas (a complete bitrate table will follow in the next days):
Code: [Select]
CLASSICAL (full tracks)

low anchor 128,00 kbps (estimated)
AAC iTunes 133,33 kbps [+4,16 %]
AAC Nero 125,71 kbps [-1,79 %]
MP3 LAME 130,81 kbps [+2,20 %]
Vorbis aoTuV 131,69 kbps [+2,88 %]
high anchor 181,46 kbps [+41,77 %]


NON-CLASSICAL (short samples)

low anchor 128,00 kbps (estimated)
AAC iTunes 137,31 kbps [+7,27 %]
AAC Nero 134,10 kbps [+4,76 %]
MP3 LAME¹ 137,82 kbps [+7,67 %]
Vorbis aoTuV² 133,42 kbps [+4,23 %]
high anchor 196,28 kbps [+53,34 %]

¹ with --athaa-sensitivity 1 bitrate reaches 139,38 kbps
² with –q4,25 bitrate reaches 140,21 kbps



[span style=\'font-size:14pt;line-height:100%\']TESTING CONDITIONS [/span]

The full test consists on pure ABC notation. The double blind test conditions are ensured by schnofler ABC/HR 0.5 beta (2005.08.31) software. All samples were decoded by CLI decoded within ABC/HR; offset were removed each times and minor differences in gain were systematically corrected (the highest difference reached 1.2 dB). Small mention for Vorbis: all files were decoded with foobar2000 (I still can’t make ABC/HR decode Vorbis files). There are no ABX comparisons: it’s a luxury I can’t afford with 1200 files awaiting for evaluation (200 x 6). If a difference is really unsure, I don’t rank the file. I finally ranked 16 times the reference instead of the encoded one (and 6 mistakes concern the high anchor). The error is inferior to 1.5%. I didn’t discard the errors from the final results (they don’t have a significant impact).
My hardware setting: Beyerdynamic DT-531 headphone; Audigy2 soundcard; Onkyo A-5 amp.



[span style=\'font-size:14pt;line-height:100%\']DREAM AND REALITY…[/span]


Last words before posting the results: I planed to write a complete review, including a complete synthesis on most common problems encountered in this test. Different encoders have different problems, and some of them are recurrent. As example, LAME produce often weird kind of rumbling (noise in low frequencies) and smearing; Vorbis has still issues with what I called “microdetails” (blurred and replaced by noise) and sometimes coarseness; iTunes suffers sometimes from a form of ringing I can’t define; Nero Digital has serious troubles on tonal passage and poor pre-echo performance.
I didn’t compile this memento yet, which should interest developers more than users. But I publish the results yet, because I feel that it’ time for me to close this test (honestly, seeing ABC/HR running somewhere drives me mad or sick).
Results are published as big png files; file size is not an issue (only 111 kb) but the image size may cause issue on small display resolution (800x600). I apologize for inconvenience. Small comments are ending the graphs. Here again, I planned to write more detailed comments, but until I achieve what I planed to do I fear that the week-end and maybe the month will be over. I postponed several activities during the two last weeks to perform and present this test, but I can’t continue anymore. If I remember correctly there’s a life outside ABC/HR  I also suspect that most people are not reading comments or details and are more interested by the final ranking. That’s why my results I’ll post today are a bit in “raw” form. I sincerely apologize, and will try to (slowly) give more textual substance in the next days. Now, results
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-15 08:14:59
[span style='font-size:20pt;line-height:100%']RESULTS[/span]




[span style='font-size:14pt;line-height:100%']I. CLASSICAL: 5 electronic/artificial samples micro-group[/span]



(http://audiotests.free.fr/tests/2005.11/results_gr1.png)





[span style='font-size:14pt;line-height:100%']II. CLASSICAL: 60 orchestral & chamber samples macro-group[/span]



(http://audiotests.free.fr/tests/2005.11/results_gr2b.png)




[span style='font-size:14pt;line-height:100%']III. CLASSICAL: 55 solo instruments samples macro-group[/span]



(http://audiotests.free.fr/tests/2005.11/results_gr3.png)





[span style='font-size:14pt;line-height:100%']IV. CLASSICAL: 30 samples macro-group[/span]



(http://audiotests.free.fr/tests/2005.11/results_gr4.png)





[span style='font-size:14pt;line-height:100%']V. NON-CLASSICAL or MODERN or VARIOUS: 50 samples macro-group[/span]



(http://audiotests.free.fr/tests/2005.11/results_gr5.png)
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-15 08:15:25
Few words to conclude the test…
It’s pretty clear that all encoders tested here correspond to a good or even a very good output quality. There are currently no winner between AAC (iTunes) and Vorbis. It’s funny to see that results are pretty close on the finish line when problems are so different. Encodings are not fully transparent, but quality is in my opinion excellent most often (but not always).
LAME offers to MP3 the chance to stay competitive against AAC and Vorbis. Not fully competitive, but the efficiency of this format forces the respect.
Nero Digital implementation of AAC is slightly disappointing, especially with classical music, which is still a weak point of this encoder. But the quality is far from disaster (it wasn’t the case two years ago), is on average really good, gets even better with “non-classical” music and should satisfy several users.
Last but not least, difference among all these encoders is really small (don't look too much on "zoomed" plots  )

But the average mark is somewhat misleading. LAME quality is ~0.5 point lower to iTunes or Vorbis, but it doesn’t mean for example that quality of encoded albums are 0,5 lower. This lower ranking is rather the expression of higher fragility than lower quality. LAME, and Nero Digital, are more inclined to serious distortions than Vorbis or iTunes AAC at the same bitrate. The concept of quality may be replaced with such encoders by the concept of strength or robustness. To illustrate this I made the following histogram (sorry for poor quality, I’ll change it later):

(http://audiotests.free.fr/tests/2005.11/results_distri.png)

Here, Vorbis and iTunes both get a mark comprise between 4.5 and 5.0 for 50% of the tested samples, whereas Nero only achieve this state (near-transparency or full transparency) for 20% of the same samples. With the classical group of samples, 30% of the them were ranked below 3.0 with Nero when iTunes or Vorbis got the same notation of less than 10% of the sample. The two winners are stronger, and could handle more situations than LAME and Nero Digital AAC.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: PoisonDan on 2005-11-15 08:30:26
Quote
(honestly, seeing ABC/HR running somewhere drives me mad or sick).
[a href="index.php?act=findpost&pid=341926"][{POST_SNAPBACK}][/a]

I can imagine that. Boy, performing this test must have been such a huge task... I'm extremely impressed!

Thanks a lot for sharing this with us, it's very interesting (especially now that you also included non-classical music).

My hat's off to you, Sir!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: vinnie97 on 2005-11-15 08:34:09
bravo!  You are much braver and patient than myself!  It would seem that buying from the Itunes store isn't such a bad quality sacrifice going by your test.  Also, it's too bad aoTuV saw another update in the middle of your test...now you have to start again...only kidding!  I don't think the quality level you tested was tuned any further in 4.5. 

Thanks again, your blind tests are one of the top attractions around here.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-15 08:42:03
Changes in aoTuV beta 4.5 are for inferior settings (up to -q3,00). Fortunately I would (exceptionally) say
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: krazy on 2005-11-15 08:45:24
Once again guruboolez, thankyou for your amazingly informative tests! And thanks for subjecting your ears to rigours the of modern music..

It's also nice to see that Aoyumi's work on vorbis is keeping it at the forefront of modern audio compression.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Synthetic Soul on 2005-11-15 08:53:07
Thank you guruboolez.

These tests are so important to the community.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: arman68 on 2005-11-15 09:13:44


Quote
Consequently, my “personal evaluations” which were first a friendly exercise feasible in one rainy, autumnal afternoon now looks as a gigantic task which took me approximately 10 days (shared with family, friends, job, and discouragement) to complete. I improved several point of the methodology


I am in awe...

I always do my own personal ABX test for my personal usage, but it is nothing compared to the enormous amount of work you do. Your tests and public results are very much appreciated, thank you.

edit: just finished reading the test results twice (to go through all the details), and I find it interesting that Nero still does not match Itunes, even though it uses a true VBR mode, whereas Itunes does not. I have been testing the new nero codec in VBR LC mode at lower bitrates for my W800i, and have been disappointed by it. What I did not do is compare it to Itunes. I will now.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Daijoubu on 2005-11-15 09:32:42
That must have been a heap load of data to compile
Did you nose bleed?
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: tycho on 2005-11-15 09:40:42
Invaluable tests again. Thank you so much. Vorbis aoTuV is the leading codec at medium bitrates (tied with iTunes AAC). And from other tests you did, Vorbis also shines at low and high bitrates. Nice to confirm that LAME -V2 --vbr-new is still superior to iTunes AAC at medium bitrates (and pretty tied with AAC ~180kbps, I guess).
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: LadFromDownUnder on 2005-11-15 09:41:49
As we say "down under", "Good on ya, mate!"
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Atlantis on 2005-11-15 10:18:22
Thanks Guru!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: ffooky on 2005-11-15 10:41:09
Cheers Guru, fascinating stuff.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: ilikedirtthe2nd on 2005-11-15 10:45:44
Thanks a lot, very interesting test again!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: robert on 2005-11-15 10:46:34
Thanks Guruboolez, very informative.

About LAME encoder being not well balanced:
Quote
LAME MP3: I used latest alpha of 3.98 (alpha 2) in order to add the –athaa-sensitivity 1 command to the –V5 --vbr-new mode. For the second group of samples and to slightly lower the bitrate I simply used –V5 –vbr-new.

I'm wondering, would your result be different if the encoder settings would have been the same for classical and none classical groups?
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Enig123 on 2005-11-15 11:07:36
Oh. I don't think this test can be ignored only because it's done by just one person. Nero company really need some work to improve there aac implementation (maby already in Ivan's brain  ).

Thank you guruboolez for your great work.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: dimzon on 2005-11-15 11:09:19
Thanx!
guruboolez, how about low-bitrate comparision (64kbps and below)
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: pest on 2005-11-15 11:15:05
 

you must be crazy
impressive work!

thanks a lot guruboolez
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Zurman on 2005-11-15 11:38:20
Nice listening test, as always
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: rjamorim on 2005-11-15 12:11:04
Awesome, awesome, awesome.


Very big thanks, Francis. You're a legend.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-15 12:23:19
Quote
Quote
LAME MP3: I used latest alpha of 3.98 (alpha 2) in order to add the –athaa-sensitivity 1 command to the –V5 --vbr-new mode. For the second group of samples and to slightly lower the bitrate I simply used –V5 –vbr-new.

I'm wondering, would your result be different if the encoder settings would have been the same for classical and none classical groups?
[{POST_SNAPBACK}][/a]
(http://index.php?act=findpost&pid=341959")

I don't think so. The --athaa-sensitivity command prevents a specific kind of ringing (I'm used to call it "background ringing"), and I don't remember any sample of the second group suffering from this problem (there are maybe one or two of them).

I already noticed this disparity in performance between classical group and "various" samples during my summer listening tests performed at [a href="http://foobar2000.net/divers/tests/2005.07/80/80TEST_PLOTS_06.png]80 kbps [/url]and 96 kbps (http://foobar2000.net/divers/tests/2005.07/96/96_TEST_COMPA_GR1_GR2.png).
The difference is also not very important. And as you can see it on the distributive histograms, the main difference occurs on the last part (ranking > 4.5). ~40% of the tested samples (classical) were ranked below 4.5 with LAME, but the proportion falls to 20% for the second category. It seems that for LAME, there are more "easy" to handle situation in my sample gallery than for the 50 samples I collected from various listening tests. (I don't know if I'm really clear...).
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-15 12:28:54
Quote
Thanx!
guruboolez, how about low-bitrate comparision (64kbps and below)
[a href="index.php?act=findpost&pid=341964"][{POST_SNAPBACK}][/a]

I'm not very happy with the quality of current encoders at this bitrate. Not really suitable for my personal use. Curiosity would therefore be my only motivation for such exercise.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Busemann on 2005-11-15 12:36:31
Surprising to see how close Vorbis and iTunes are to the high anchor. I guess one could safely use 160kbps VBR for transparency with iTunes now (I previously used 192kbps).
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: QuantumKnot on 2005-11-15 13:27:33
To guruboolez, thank you for yet another incredibly fascinating and informative listening test.

I am again very pleased to see Vorbis doing so well.  Full credits to Aoyumi for his wonderful work.  I'm also very pleased to see iTunes AAC doing so well too.  It seems we do get value for money with these two encoders (ie. they're free!!! even better  )
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: henkersmahlzeit on 2005-11-15 13:43:02
Thanks Thanks Thanks ... what a job!     
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Zurman on 2005-11-15 13:46:57
Quote
Surprising to see how close Vorbis and iTunes are to the high anchor. I guess one could safely use 160kbps VBR for transparency with iTunes now (I previously used 192kbps).
[a href="index.php?act=findpost&pid=341977"][{POST_SNAPBACK}][/a]

Well, I think this test shows exactly the contrary 
Good quality, yes, but transparency, no (at least to Guru's ears  )
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: rjamorim on 2005-11-15 13:52:40
Quote
Quote
Surprising to see how close Vorbis and iTunes are to the high anchor. I guess one could safely use 160kbps VBR for transparency with iTunes now (I previously used 192kbps).[a href="index.php?act=findpost&pid=341977"][{POST_SNAPBACK}][/a]

Well, I think this test shows exactly the contrary  [a href="index.php?act=findpost&pid=341993"][{POST_SNAPBACK}][/a]


How come? If VBR 128 comes close to transparency even to Guruboolez, VBR 160 should be transparent to most of us.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Zurman on 2005-11-15 14:11:56
Quote
Quote
Quote
Surprising to see how close Vorbis and iTunes are to the high anchor. I guess one could safely use 160kbps VBR for transparency with iTunes now (I previously used 192kbps).[a href="index.php?act=findpost&pid=341977"][{POST_SNAPBACK}][/a]

Well, I think this test shows exactly the contrary  [a href="index.php?act=findpost&pid=341993"][{POST_SNAPBACK}][/a]


How come? If VBR 128 comes close to transparency even to Guruboolez, VBR 160 should be transparent to most of us.
[a href="index.php?act=findpost&pid=341996"][{POST_SNAPBACK}][/a]

I messed up with the bitrates
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: DARcode on 2005-11-15 14:24:16
Wow  ! How far can a  guruboolez appreciation month be now  ?!

Seriously, your contribution to the community and HA site is incredible and very much appreciated, man!

A whole hearted thank you very much!

Spent the whole lunch break brwosing your results, awesome stuff!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: arman68 on 2005-11-15 14:28:12
Quote
Quote
Thanx!
guruboolez, how about low-bitrate comparision (64kbps and below)
[a href="index.php?act=findpost&pid=341964"][{POST_SNAPBACK}][/a]

I'm not very happy with the quality of current encoders at this bitrate. Not really suitable for my personal use. Curiosity would therefore be my only motivation for such exercise.
[a href="index.php?act=findpost&pid=341976"][{POST_SNAPBACK}][/a]


Although I do agree it would be nice to have someone as professional as guruboolez to do those tests; with such low bitrates, almost anyone can ABX and make their own mind. Why don't you try it? It is very interesting to discover the different audio artifacts introduced by lossy compression. It kinds of ruins your listening experience, as you will then recognize them easily  but I think it is worth it.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Sunhillow on 2005-11-15 15:28:44
DARcode seems to be VERY impressed 

guruboolez should write for a big HiFi magazine... No, wait, he doesn't support voodoo thingies! Noone would like to read him there!

Really impressing and, as always, very conclusive. From me also a big Thank you!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: lordraiden on 2005-11-15 15:31:38
Somebody can do a short conclusion about test I don't understand very well

What is the best? and the 2º.....
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: kornchild2002 on 2005-11-15 15:46:44
Wow, thanks guruboolez as you have done so much work that I myself just could not do.  I am quite surprised at the results.  I is nice to see that Vorbis is still going hard and strong but I was really surprised by the iTunes mpeg-4 AAC results in that it narrowed the gap between itself and Vorbis for all sample stested.  It is also nice to know that consumers can get their hands on both products for free.

Even after all the buzz about the new Nero AAC encoder it is interesting to see that it is beaten out by the iTunes AAC encoder.  Hats off to the Lame community as the Lame mp3 encoder is still in healthy competition with other formats.

Again, thanks buruboolez for the results and all the hard work.  I will make sure to use your results in further discussions.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-15 15:51:14
Quote
Somebody can do a short conclusion about test I don't understand very well

What is the best? and the 2º.....
[a href="index.php?act=findpost&pid=342021"][{POST_SNAPBACK}][/a]


The absolute best encoder is the high anchor (LAME -V2 at ~190 kbps), but this setting is off-competition (bitrate is too high, and comparison with other would be unfair).
That's why the real first place is for iTunes AAC and Vorbis aoTuV, which are statistically equivalent.
With classical music, LAME is third and Nero Digital fourth.
With non-classical, LAME and Nero Digital are both last, despite the better mark obtained by Nero.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: lordraiden on 2005-11-15 15:55:00
is better iTunes AAC than Nero Digital for music at 160 kbps or 190 kbps?
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-15 15:59:26
Quote
is better iTunes AAC than Nero Digital for music at 160 kbps or 190 kbps?
[a href="index.php?act=findpost&pid=342026"][{POST_SNAPBACK}][/a]

You should try yourself. There are not many people that will be ready to start a listening comparison with modern AAC encoders at high bitrate.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Gambit on 2005-11-15 16:02:39
Quote
Quote
is better iTunes AAC than Nero Digital for music at 160 kbps or 190 kbps?
[a href="index.php?act=findpost&pid=342026"][{POST_SNAPBACK}][/a]

You should try yourself. There are not many people that will be ready to start a listening comparison with modern AAC encoders at high bitrate.
[a href="index.php?act=findpost&pid=342027"][{POST_SNAPBACK}][/a]

I think that we will soon see that even the planned 128kbps test will be very hard for some people. The performance of modern codecs at those bitrates is really good, and I dare to say transparent for Joe Average's casual listening.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: arman68 on 2005-11-15 16:06:18
Quote
I think that we will soon see that even the planned 128kbps test will be very hard for some people. The performance of modern codecs at those bitrates is really good, and I dare to say transparent for Joe Average's casual listening.


I could not agree more. I will try to take part in the test if I find the time, but I doubt I will be able to hear any difference in most cases, unless they are problem samples.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: SirGrey on 2005-11-15 16:18:30
Uh. Thanks, guruboolez ! 

Quote
The performance of modern codecs at those bitrates is really good, and I dare to say transparent for Joe Average's casual listening.

Yeah. I completely agree.
I'm still unsure if I should participate in 128 kbit test
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Sebastian Mares on 2005-11-15 16:23:22
Quote
I'm still unsure if I should participate in 128 kbit test
[a href="index.php?act=findpost&pid=342035"][{POST_SNAPBACK}][/a]


It's appreciated if you do.

By the way, thanks Guru for the effort and the excellent test!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: kuniklo on 2005-11-15 17:07:39
One more voice of thanks for your thorough and professional work.  Your tests should be in print somewhere.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: singaiya on 2005-11-15 17:37:20
Like everybody else... wow! merci beaucoup. I think this is the last piece of evidence for my switch to Vorbis.

One interesting thing I noticed is that the high anchor (Lame V2 vbrnew) did much better with this expanded classical set than in your previous test of high bitrates
MPC vs VORBIS vs MP3 vs AAC at 180 kbps, 2nd checkup with classical music (http://www.hydrogenaudio.org/forums/index.php?showtopic=36465&view=findpost&p=321547)
where on those 18 classical samples it received a score of 3.6, here it gets 4.61. Actually, in that test you used 3.97alpha11 but I think there was no change to 3.97beta1.

[span style='font-size:8pt;line-height:100%']edit: linked to post(#2), punctuation[/span]
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: richard123 on 2005-11-15 19:18:45
Allow me to add my thanks.

Great job.

Makes me feel better about using iTunes AAC at 160k
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: yulyo! on 2005-11-15 19:43:32
Great test Guru.
it seems Nero new encoder isn't that good as Gabriel said. 
by the way...where is Gabriel's and the others nero aac encoder devs?
do't get me wrong, i love Nero AAC and i hoped that the new one wil be better, but...
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Sebastian Mares on 2005-11-15 19:46:27
Quote
Great test Guru.
it seems Nero new encoder isn't that good as Gabriel said.  
by the way...where is Gabriel's and the others nero aac encoder devs?
do't get me wrong, i love Nero AAC and i hoped that the new one wil be better, but...
[a href="index.php?act=findpost&pid=342069"][{POST_SNAPBACK}][/a]


Gabriel is a LAME developer. I think you mean Garf.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: kwanbis on 2005-11-15 19:55:06
cool guru.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: yulyo! on 2005-11-15 19:58:04
     right
sorry, my mistake
 
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Garf on 2005-11-15 20:04:06
Quote
Great test Guru.
it seems Nero new encoder isn't that good as Gabriel said.  
by the way...where is Gabriel's and the others nero aac encoder devs?
do't get me wrong, i love Nero AAC and i hoped that the new one wil be better, but...
[a href="index.php?act=findpost&pid=342069"][{POST_SNAPBACK}][/a]


1) I'd wait for results from more than 1 person before making a conclusion.
2) Guruboolez already stated it was a major improvement over his previous test.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: yulyo! on 2005-11-15 20:13:03
i know Garf, it really is.
but i remember someone saiying that the new encoder will be the best encoder 
by the way, great work.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: [JAZ] on 2005-11-15 20:41:44
Quote
i know Garf, it really is.
but i remember someone saiying that the new encoder will be the best encoder 
by the way, great work.
[a href="index.php?act=findpost&pid=342077"][{POST_SNAPBACK}][/a]


The quote was for the SBR+PS profile, so that still has to be proven (to be correct or not).


Thanks to guruboolez from here too. These tests are always appreciated.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: beto on 2005-11-15 21:36:50
Guru, outstanding as usual.

Many thanks for sharing it with us.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: jaybeee on 2005-11-15 21:57:13
@guruboolez: many thanks, and massive respect 
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Garf on 2005-11-15 22:03:48
Quote
Quote
i know Garf, it really is.
but i remember someone saiying that the new encoder will be the best encoder 
by the way, great work.
[a href="index.php?act=findpost&pid=342077"][{POST_SNAPBACK}][/a]


The quote was for the SBR+PS profile,
[a href="index.php?act=findpost&pid=342085"][{POST_SNAPBACK}][/a]


I don't think I even claimed that.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: vinnie97 on 2005-11-15 22:35:27
I have a feeling Nero AAC would be the new contender at 64 kbps, which is perfect for flash players, if only one could purchase such a device.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: [JAZ] on 2005-11-15 22:36:52
Quote
I don't think I even claimed that.
[a href="index.php?act=findpost&pid=342106"][{POST_SNAPBACK}][/a]


Eurrrm.. yep. the "best encoder" part of "yulyo" post should read "comparable to 128kbps MP3".  I was too fast typing that.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: skelly831 on 2005-11-15 23:26:20
All hail the great Guruboolez!

Nicely done! Muchas Gracias!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: de Mon on 2005-11-16 01:09:40
Guru, how could you make such a big testing ALONE in such SHORT time? Aren't you affraid Interpol will suspect you for illegal cloning of yourself? 

Great work Guru! And very useful for both users and developers. Thanks!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: smz on 2005-11-16 01:22:25
Really powerfull work, Guru! All my compliments to you!

Sergio
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: kl33per on 2005-11-16 02:14:17
Again Guru, I am astounded.  How you find the time and motivation to perform such a massive undertaking is of constant amazement to myself.  Thanks so much.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Aoyumi on 2005-11-16 14:13:02
I was surprised by the too much huge test. The contents of a test are very interesting also for me.
Thank you, Guru. 
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Cpt. Spandrel on 2005-11-17 00:44:26
Quote
[span style='font-size:14pt;line-height:100%']Preliminary notes[/span]
As low anchor, I looked for something really low and also usable in batch mode. I found a very old AAC encoder on ReallyRareWares called mbaacencoder version 0.3: it’s awfully slow, quality is terrific


I'm guessing that the last word there should be 'terrible'? Otherwise you'd seem to be unduly generous 
Wonderful test Guru, much appreciated.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Wedge on 2005-11-17 00:57:06
very impressive test. thanks alot!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: krazy on 2005-11-17 03:47:34
Quote
Quote
[span style='font-size:14pt;line-height:100%']Preliminary notes[/span]
As low anchor, I looked for something really low and also usable in batch mode. I found a very old AAC encoder on ReallyRareWares called mbaacencoder version 0.3: it’s awfully slow, quality is terrific


I'm guessing that the last word there should be 'terrible'? Otherwise you'd seem to be unduly generous 
[{POST_SNAPBACK}][/a]
(http://index.php?act=findpost&pid=342426")

Heh, not to hijack this thread with semantics, but terrific is an [a href="http://www.answers.com/terrific]antonym of itself[/url]. 
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-17 09:50:23
Quote
2) Guruboolez already stated it was a major improvement over his previous test.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=342075")

Not exactly. Last year, the experimental "fast" mode was superior to [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=29925&hl=]Vorbis[/url] and iTunes (http://www.hydrogenaudio.org/forums/index.php?showtopic=29924&hl=), on classical music. This year, it ends last. It's not what I call a major improvement. But to be complete, I must admit that what I tested last year was more a ~135...140 kbps preset, and the current one a 125...130 kbps. Results are not totally comparable. Nevertheless, there are some obvious regression, occuring with different kind of samples (like harpsichord, organ and other tonal signal). That's why I'm disappointing to see the new encoder not following some progress revealed last year (with classical at least). But it's much much better than 2003 encoders
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: QuantumKnot on 2005-11-17 10:25:22
Based on these comprehensive listening tests (as well as previous ones by guruboolez), I think its the right time to make aoTuV beta 4 the recommended Vorbis encoder, so I've made the necessary changes.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: =trott= on 2005-11-17 11:29:03
Don't get me wrong, I fully believe guruboulez knows more about audio and has better ears than I will ever have. I also think the results from this test are very interesting, to say the least.
However, I have to side with Garf here: no matter who the person or what the equipment, no matter how well-presented the result, no matter how scientific the method, this is statistical data compiled by a single individual. Thus I am not fully convinced about the relevancy. As far as I understand, if you were to be sitting in guru's living room, wearing his headphones, playing back the music through his pc, the results would be statistically valid. (though by no means this means that I consider them invalid...)

Maybe Roberto or ff123 can clear this up for me?
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: dimzon on 2005-11-17 12:11:49
Quote
Quote
Thanx!
guruboolez, how about low-bitrate comparision (64kbps and below)
[a href="index.php?act=findpost&pid=341964"][{POST_SNAPBACK}][/a]

I'm not very happy with the quality of current encoders at this bitrate. Not really suitable for my personal use. Curiosity would therefore be my only motivation for such exercise.
[a href="index.php?act=findpost&pid=341976"][{POST_SNAPBACK}][/a]

Whe need to know which HE-AAC implementation (Nero6, Nero7, WinAmp, CT Reference) is better for DVD-Rip!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-17 12:26:56
Quote
Whe need to know which HE-AAC implementation (Nero6, Nero7, WinAmp, CT Reference) is better for DVD-Rip!
[a href="index.php?act=findpost&pid=342534"][{POST_SNAPBACK}][/a]

It shouldn't be too hard to test it for most people. Nero Digital is AFAIK the only choice for multichannel encoding with HE Profile - no tests are needed here.
For stereo, it seems from the very little comparisons I done that Ivan worked conscientiously on High Efficiency profile. The biggest annoyance audible with previous encoder is now gone. Now, I can't tell you which one (between Nero and Coding Tech. encoders) is better. I'm not very interested to test them - HE encodings are still irritating to my ears.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-17 12:37:44
Quote
One interesting thing I noticed is that the high anchor (Lame V2 vbrnew) did much better with this expanded classical set than in your previous test of high bitrates
MPC vs VORBIS vs MP3 vs AAC at 180 kbps, 2nd checkup with classical music (http://www.hydrogenaudio.org/forums/index.php?showtopic=36465&view=findpost&p=321547)
where on those 18 classical samples it received a score of 3.6, here it gets 4.61. Actually, in that test you used 3.97alpha11 but I think there was no change to 3.97beta1.
[a href="index.php?act=findpost&pid=342050"][{POST_SNAPBACK}][/a]

That's indeed a good point. I was much more attentive and severe during my high bitrate listening test (otherwise most encoders would be ranked eaqually transparent). As I said it in the first post, I didn't insisit in this test to find subtle differences. That's why there are several 5.0 notes.

There wasn't also any anchor in the HQ test to temper the notation (but I tried to not be too harsh). It's maybe a good occasion to recall that anchors shouldn't be optional and replaced by regular contender

No need to say that 4.61 is much more representative of the real quality of LAME -V2 --vbr new in the ITU notation scale than 3.6.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: dimzon on 2005-11-17 13:25:14
Quote
Now, I can't tell you which one (between Nero and Coding Tech. encoders) is better. I'm not very interested to test them - HE encodings are still irritating to my ears.
[a href="index.php?act=findpost&pid=342537"][{POST_SNAPBACK}][/a]

Thanx a lot! So it's possible to use free CT Encoder from WinAmp for stereo without quality tradeoff!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-17 13:47:16
Quote
Thanx a lot! So it's possible to use free CT Encoder from WinAmp for stereo without quality tradeoff!
[a href="index.php?act=findpost&pid=342552"][{POST_SNAPBACK}][/a]

I don't know. CT encoder could be better or could be worse: what I tested is by far too limited to draw any conclusion. If you're interested by quality with HE-AAC, I strongly suggest you to perform your own tests (and sharing results if possible ).
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: dimzon on 2005-11-17 13:56:12
Quote
Nero Digital is AFAIK the only choice for multichannel encoding with HE Profile - no tests are needed here.
[a href="index.php?act=findpost&pid=342537"][{POST_SNAPBACK}][/a]

enc_aacplus.dll from WinAmp5.1 can encode 5.1 HE-AAC (up to 128kbps  )
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Halcyon on 2005-11-17 17:42:13
Thank you Guru, you never seize to amaze.

Can I pick you brain or technique here? Feel free to skip these, if they are too cumbersome or too personal

Do you play an acoustical unamplified instrument? If so, which?

Do you sing or have you been taugh singing as a trained skill?

Have you trained your pitch listening skills consciously (relative or absolute)?

Which one is your leading ear (more accurate in tonality estimation)? Do you use both ears throughout all the tests or sometimes just one ear?

Have you consciously trained by listening to test sample sets (known errors in perceptual coding)? Or just listened to a lot of lossy encoder output?

At which volume levels you usually listen to (soft, medium, loud)? Or do you vary volume throughout the test?

Have you ever listened to any of the encoders via loudspeakers (not for tonality issues or small artifacts, but for soundstaging issues)?

At what time of the day do you you usually test?

How long tests can you do before you have to stop or find that you can't make out the differences anymore (number of repeats or duration roughly in minutes)?

Do you use a lot of quick back-and-forth A<->B switching or do you just listen to sample A in full, then sample B in full and then decide?

What are the top 3 most important things for yourself when you do a listening test?

I'd really appreciate if you have time to answer any of these questions.

Thanks again for the taking the time to not only to do the test, but go through the ordeal of preparing and publishing the results!

best regards,
halcyon

PS I just smile at the thought of what kind of accuracy you'd reach if you had a proper pair of electrostatic headphones at your use (very low distortion, very good tonality), a powerful noiseless/distortion-free amp and a good sound card.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: yandexx on 2005-11-18 18:15:17
Guru, thanks a lot for your test! We really appreciate your work! (and you know  )

Halcyon, these questions are like for real interview with guruboolez
I'd like to hear answers too
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Garf on 2005-11-18 19:31:41
Quote
Nevertheless, there are some obvious regression, occuring with different kind of samples (like harpsichord, organ and other tonal signal). That's why I'm disappointing to see the new encoder not following some progress revealed last year (with classical at least). But it's much much better than 2003 encoders
[a href="index.php?act=findpost&pid=342508"][{POST_SNAPBACK}][/a]


I believe this is an issue of specific listener preference/sensitivity + genre, and that your conclusion is not generally applicable (and in fact, most people will experience the opposite). That's why, for example, we never really recommended fast mode, despite you saying it was much better.

It may be possible to "fix" it for both groups and we have made some progress on this in the meantime.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 10:21:14
Quote
I'd really appreciate if you have time to answer any of these questions.
[a href="index.php?act=findpost&pid=342599"][{POST_SNAPBACK}][/a]

I have the bad feeling that someone from CIA or KGB is questionning me...

Do you play an acoustical unamplified instrument? If so, which?
Unfortunately, now. If I had, it would be harpsichord, gamba or cello.

Do you sing or have you been taugh singing as a trained skill?
No. I'm fond of baroque music (especially opera), and it's too late for me to be a castrato (it's also forbidden).

Have you trained your pitch listening skills consciously (relative or absolute)?
My what?

Which one is your leading ear (more accurate in tonality estimation)? Do you use both ears throughout all the tests or sometimes just one ear?
A leading ear? I'm right-handed, but my hearing seems ambidextrous.

Have you consciously trained by listening to test sample sets (known errors in perceptual coding)? Or just listened to a lot of lossy encoder output?
I don't think that listening to lossy encodings would help to pass a listening test. Otherwise there would be millions golden-ears on this planet. In my case, my training ground is ABC/HR and not foobar2000 or winamp.

At which volume levels you usually listen to (soft, medium, loud)? Or do you vary volume throughout the test?
It's hard to tell. Some people are convinced to listen at moderate volume when they're sharing their music with the full train with a simple pair of earbuds. I'd say "medium". Sometimes very loud (enthusiastic mood), sometimes very quiet. During listening test, it's "medium" (sometimes louder, sometimes quieter).

Have you ever listened to any of the encoders via loudspeakers (not for tonality issues or small artifacts, but for soundstaging issues)?
No.

At what time of the day do you you usually test?
As soon as I can (200 samples: no time to loose).

How long tests can you do before you have to stop or find that you can't make out the differences anymore (number of repeats or duration roughly in minutes)?
It depends. Some samples are less exhausting than others. The enthusisasm for the music plays a big role also. No need to precise that I've more energy for my own music than for something I don't like.
I'm often taking a break; some of them a very short (they may happen during ABC or ABX phases, and not necessary between two samples). I'm also used to stop few minutes between two samples. After 5...15 samples (without ABXing), I must stop longer.

Do you use a lot of quick back-and-forth A<->B switching or do you just listen to sample A in full, then sample B in full and then decide?
It depends on the tested bitrate. Here, degradations are often obvious. That's why I could often pull down the slider within two or three seconds for each sample. I rarely hit the X button. I'm working in few steps:
- first listening pass: I'm trying to quickly unmask each encoding (listening to all A-B A-B A-B A-B, first seconds only). I'm giving an approximate notation according to the level of annoyance.
- second listening pass to pull down the remaining sliders and to adjust (still vaguely) the sliders.
- if encoders are still unmasked, I'm changing the playback range and listen carefully. (again, I'm trying to increase the accuracy of the ranking at this stage).

Then begins the second big step, and it's the most longer: time for me to give an accurate mark and most dramatic point, to decide for a hierarchy. People which are not used to perform listening tests might not understand why it's so hard, but I can swear you that grading different form of distortions is anything but obvious. Sometimes the hierarchy is not debatable (it happens with discriminative samples), but most often it's problematic. Another point: each contenders makes this step longer and tedious. That's why I'm  when I see people asking to test 5, 6 or sometimes 7 files in collective tests. I suppose that some of them not realizes the amount of work that each additional competitor implies.


What are the top 3 most important things for yourself when you do a listening test?
- Coffee
- Tea
- listening to a full composition instead of 10 seconds samples from time to time (this, and all other kind of breaks)

Less materially: it's very important that the test makes sense. If I have the feeling that one of the choosen encoder or setting is not the good one, it ruins the motivation. That's why it's very important to know the average bitrate of the VBR preset, or to know if you're testing the good implementation or setting. Otherwise, the test becomes useless and is just a waste of time.


PS I just smile at the thought of what kind of accuracy you'd reach if you had a proper pair of electrostatic headphones at your use (very low distortion, very good tonality), a powerful noiseless/distortion-free amp and a good sound card.

Results won't be different I suppose. They won't change the accuracy at least, simply because all grading can't be accurate by themselves (it's often 2.0, 2.5, 3.0, 3.5... and this won't change with better or poorer hardware )
A better headphone might help me to find additionnal subtle distortions, but to make progress training is always better than equipment.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 12:39:45
Quote
I believe this is an issue of specific listener preference/sensitivity + genre, and that your conclusion is not generally applicable (and in fact, most people will experience the opposite).
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=342945")


Belief and facts in the same time 
How do you know that "most people will experience the opposite"?
I don't think that what I heard is specific to my hearing. I can't prrove it, but I can try to experiment this, and post samples encoded with 3.2.0.15 and 4.2.1.0 and illustrating what I consider as major regression.
• [a href="http://foobar2000.net/divers/temp/S22.zip]S22: organ[/url] (622 Kb)
• S30: accordion (http://foobar2000.net/divers/temp/S30.zip) (428 Kb)
• S44: bagpipe (http://foobar2000.net/divers/temp/S44.zip) (900 Kb) - all archives contains the encoded files, and all are named (I'm not trying to fool nor to play with people).

The quality is very different; the bitrate too. The previous "fast" encoder which I considered as amazingly good was used to spend a lot of bits to maintain quality with very tonal signal (bitrate is very high, but on very short moments only). The current new encoder reacts much more like the previous "high" encoder, and with poorer quality sometimes!
Extreme case is S44:
aacenc3 'high' = 134 kbps
aacenc3 'fast' = 302 kbps
aacenc4 'high' = 113 kbps

I also put iTunes encoding for comparison: there's no bloated bitrate, and quality is in my opinion near transparency.

[span style='font-size:12pt;line-height:100%']It would be nice to see other people downloaded these three examples, and to give their opinion. I'd like to know if my hearing is very far from normality, or if other people agree with me. It would be hard to believe that what I hear are issues that "are not generally applicable". Thanks[/span]

Original files are also here: S22 (http://guruboolez.free.fr/SINGLE/S22_KEYBOARD_Organ_F.ofr), S30 (http://guruboolez.free.fr/SINGLE/S30_OTHERS_Accordion_A.ofr) & S44 (http://guruboolez.free.fr/SINGLE/S44_WIND_Bagpipe_A.ofr)


Quote
That's why, for example, we never really recommended fast mode, despite you saying it was much better.

I said it, and Ivan and JohnV confirmed it. The "fast" encoder was supposed to correct real issues occuring with the "old/high" encoder and mostly audible with classical. And this new encoder was announced to be the successor of the old one when some problems (there were never detailed, even when I asked for them) will be fixed. Now it seems that the successor don't even include the enhancement of the previous experimental encoder 
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Ivan Dimkovic on 2005-11-19 12:50:49
@guruboolez,

There will be an update for the next multiformat listening test that I hope correct the issues you found.

I hope you will find some time to test the upcoming version.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Garf on 2005-11-19 13:11:43
Quote
It would be hard to believe that what I hear are issues that "are not generally applicable".
[a href="index.php?act=findpost&pid=343105"][{POST_SNAPBACK}][/a]


What part of "genre" in my previous post didn't sink in?

You're now intending to 'prove' that point by posting 3 classical samples where one encoder spends two or three times as much bits as the other, and test if people can hear the difference? Eh?

I do not believe Nero 4.2.1.0 "high" is a regression over 3.2.0.15 "fast" mode.

I can believe that for classical it's especially bad for you.

I do not believe that will be generally true.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 13:30:40
Quote
You're now intending to 'prove' that point by posting 3 classical samples where one encoder spends twice as much bits as the other, and test if people can hear the difference? Eh?
[a href="index.php?act=findpost&pid=343112"][{POST_SNAPBACK}][/a]


First: I'm not trying to prove anything. Read what I said: "I can't prrove it, but I can try to experiment this...".
And I believe that posting three samples as "illustration" (I'm quoting myself) is surely better than constantly making claims about "most people hear" without any elements of proof.

Second: try to answer correctly please. You perfectly know that the bitrate you get on short samples is not a problem. The previous -internet mode and the current one both lead to similar bitrate (~130 kbps) and if the previous encoder was clever enough to allocate 300 kbps to maintain the quality, then the current one has issue.

Quote
I do not believe Nero 4.2.1.0 "high" is a regression over 3.2.0.15 "fast" mode.

That's fine. I personaly don't believe anything on this subject: I haven't compared directly the previous encoder with the new one. But I know that the previous "fast" encoder handled very well most situations I've tested, and that I can't say the same thing for the new one. With some samples, quality really seems to me unworthy of a modern AAC implementation.
What do you think about the samples I've posted as illustration?



P.S. enable on your side the option "this post as been edited..."
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: QuantumKnot on 2005-11-19 13:33:22
Just posting my initial impressions, in response to guruboolez' personal invitation to me.  I've had a listen to S30 and will post further results on the other sample when my ears aren't as tired

NB:  I'm far from being golden-eared.

Sample S30

Code: [Select]
-------------------------------------
WinABX v0.23 test report
11/19/2005 22:57:23

A file: C:\Documents and Settings\stephen\Desktop\S30\001 S30 [4.2.1.0 VBR internet high].wav
B file: C:\Documents and Settings\stephen\Desktop\S30\001 S30 [iTunes v6.0.0.18, QuickTime 7.0.3].wav

22:58:16    1/1  p=50.0%
22:59:56    2/2  p=25.0%
23:00:10    3/3  p=12.5%
23:00:32    4/4  p= 6.2%
23:01:08    5/5  p= 3.1%
23:02:07    6/6  p= 1.6%
23:03:31    7/7  p= 0.8%
23:04:03    8/8  p= 0.4%
23:04:23    9/9  p= 0.2%
23:04:50  10/10  p< 0.1%
23:05:08  11/11  p< 0.1%
23:05:41  12/12  p< 0.1%
23:05:42  test finished

-------------------------------------
WinABX v0.23 test report
11/19/2005 23:08:11

A file: C:\Documents and Settings\stephen\Desktop\S30\001 S30_OTHERS_Accordion_A.wav
B file: C:\Documents and Settings\stephen\Desktop\S30\001 S30 [4.2.1.0 VBR internet high].wav

23:09:13    1/1  p=50.0%
23:09:31    2/2  p=25.0%
23:10:32    3/3  p=12.5%
23:11:45    4/4  p= 6.2%
23:11:57    5/5  p= 3.1%
23:13:20    6/6  p= 1.6%
23:13:49    7/7  p= 0.8%
23:14:06    8/8  p= 0.4%
23:14:18    9/9  p= 0.2%
23:14:37  10/10  p< 0.1%
23:14:54  11/11  p< 0.1%
23:15:05  12/12  p< 0.1%
23:15:07  test finished

-------------------------------------
WinABX v0.23 test report
11/19/2005 23:21:05

A file: C:\Documents and Settings\stephen\Desktop\S30\001 S30 [3.2.0.15 VBR internet fast].wav
B file: C:\Documents and Settings\stephen\Desktop\S30\001 S30 [4.2.1.0 VBR internet high].wav

23:21:36    1/1  p=50.0%
23:21:57    2/2  p=25.0%
23:22:37    3/3  p=12.5%
23:23:02    4/4  p= 6.2%
23:23:18    5/5  p= 3.1%
23:23:35    6/6  p= 1.6%
23:24:00    7/7  p= 0.8%
23:24:35    8/8  p= 0.4%
23:25:07    9/9  p= 0.2%
23:25:26  10/10  p< 0.1%
23:25:42  11/11  p< 0.1%
23:26:01  12/12  p< 0.1%
23:26:04  test finished


One ABX test I didn't include above is between iTunes AAC and the original. Basically I failed it and couldn't tell the difference without guessing, so I gave up.

Basically, there seems to be a sharp amplification of the change in notes between 1 and 2 seconds in the "4.2.1.0 VBR internet high" encoded file that was very apparent, which I didn't hear in either the iTunes, "3.2.0.15 VBR internet fast" sample, or the original sample, hence the perfect ABXing.  The change is much more subtle in the other 3 files.


Sample S44

Code: [Select]
-------------------------------------
WinABX v0.23 test report
11/19/2005 23:41:52

A file: C:\Documents and Settings\stephen\Desktop\S44\001 S44 [iTunes v6.0.0.18, QuickTime 7.0.3].wav
B file: C:\Documents and Settings\stephen\Desktop\S44\001 S44 [4.2.1.0 VBR internet high].wav

23:42:21    1/1  p=50.0%
23:42:35    2/2  p=25.0%
23:42:46    3/3  p=12.5%
23:43:30    4/4  p= 6.2%
23:43:40    5/5  p= 3.1%
23:43:51    6/6  p= 1.6%
23:44:06    7/7  p= 0.8%
23:44:14    8/8  p= 0.4%
23:44:25    9/9  p= 0.2%
23:44:36  10/10  p< 0.1%
23:45:02  11/11  p< 0.1%
23:45:12  12/12  p< 0.1%
23:45:14  test finished

-------------------------------------
WinABX v0.23 test report
11/19/2005 23:45:15

A file: C:\Documents and Settings\stephen\Desktop\S44\001 S44 [4.2.1.0 VBR internet high].wav
B file: C:\Documents and Settings\stephen\Desktop\S44\001 S44 [3.2.0.15 VBR internet fast].wav

23:45:43    1/1  p=50.0%
23:45:50    2/2  p=25.0%
23:45:57    3/3  p=12.5%
23:46:01    4/4  p= 6.2%
23:46:10    5/5  p= 3.1%
23:46:19    6/6  p= 1.6%
23:46:30    7/7  p= 0.8%
23:46:46    8/8  p= 0.4%
23:46:55    9/9  p= 0.2%
23:47:30  10/10  p< 0.1%
23:47:38  11/11  p< 0.1%
23:47:45  12/12  p< 0.1%
23:47:46  test finished


Some more ABX results for sample S44.  In the "4.2.1.0 VBR internet high", I heard some ugly distortion within the 1st second, which wasn't there in the iTunes AAC or "3.2.0.15 VBR internet fast" samples, hence the perfect ABXing.

Like the first sample I tested, there are some very noticeable issues with the "4.2.1.0 VBR internet high" encoded files that I didn't hear in the other two.  A definite regression on these two samples (S30 and S44).


EDIT:  Added ABX results for sample S44.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 13:39:04
QuantumKnot> thanks for testing it (ABX scores is a luxury I didn't request)

If other people could tell what they think, it would be nice. Even if you can't tell any difference, comments would be appreciated. Thank you.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Garf on 2005-11-19 13:50:04
You can choose for yourself what you believe or don't believe on this subject.

Arguing about it seems about as pointless since we can't even agree on what would prove or disprove each others claims.

I will note though, that your current "belief" requires us to have done things such as:

1) Consistently putting the old encoder in "high" mode by default, despite knowing about "fast" mode being overall better without any strings attached.
2) Throw away the most significant improvements from the old encoder and don't use them in the new encoder.
3) Having worked for a long time on the new encoder only to make something that was significantly worse than we had before, even worse than the old "high" mode in quite a few situations.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 14:01:32
Quote
You can choose for yourself what you believe or don't believe on this subject.
[a href="index.php?act=findpost&pid=343123"][{POST_SNAPBACK}][/a]

If I'm making listening test it's precisely to not choose what I should believe  I always prefer experimentation to beliefs.

Quote
Arguing about it seems about as pointless since we can't even agree on what would prove or disprove each others claims.

I thought it was precisely the purpose of listening tests... If several people could confirm that the current Nero encoder suffers from specific issue, it would be likely that the issue exists.

Quote
I will note though, that your current "belief" requires us to have done things such as:

1) Consistently putting the old encoder in "high" mode by default, despite knowing about "fast" mode being overall better without any strings attached.
2) Throw away the most significant improvements from the old encoder and don't use them in the new encoder.
3) Having worked for a long time on the new encoder only to make something that was significantly worse than we had before, even worse than the old "high" mode in quite a few situations.


Very sarcastic situation indeed... But is it impossible? If I remember correctly, it was very clear that "in the mind of HA.org administration" (to quote an old recommendation fortunately changed very recently) developers such as Robert, Gabriel, Alexander and Takehiro spent three year of their time to make LAME perform worse than the beloved 3.90.3. Doesn't it mean that developers are not necessary infallible?
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Garf on 2005-11-19 14:12:48
Quote
If I'm making listening test it's precisely to not choose what I should believe smile.gif I always prefer experimentation to beliefs.

I thought it was precisely the purpose of listening tests... If several people could confirm that the current Nero encoder suffers from specific issue, it would be likely that the issue exists.


It would be, if the test was over multiple genres with multiple listeners. I am interested in such a test. I am not interest in the sillyness that this one is.

Realistically, you're testing 3 classical samples where the difference in bitrate allocation (despite the known fact that the averages should be the same!) should already be a big warning sign that something is not quite right.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Garf on 2005-11-19 14:19:47
Quote
Second: try to answer correctly please. You perfectly know that the bitrate you get on short samples is not a problem. The previous -internet mode and the current one both lead to similar bitrate (~130 kbps) and if the previous encoder was clever enough to allocate 300 kbps to maintain the quality, then the current one has issue.


It might be suboptimal on these samples. So what? How does this allow you to make any kind of conclusion by letting someone else confirm that a sample encoded with 3x the bits is better?
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: ToS_Maverick on 2005-11-19 14:24:25
i just tested s22 and was only able to abx it with 8/10 against nero 4.2.1.0. the other samples were transparent to me.

my ears are not well trained, i just try to hear things. i don't want to concentrate and relisten to a sample 100-times, just to pass a test. i think it get's too boring. i won't hear specific issues while i just listen to the music (mostly in my car) anyway
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 14:24:38
Quote
It would be, if the test was over multiple genres with multiple listeners. I am interested in such a test. I am not interest in the sillyness that this one is.[a href="index.php?act=findpost&pid=343129"][{POST_SNAPBACK}][/a]

I know. You repeated several times in the last days that such tests are not "proper". Your opinion, not mine.

Quote
Realistically, you're testing 3 classical samples where the difference in bitrate allocation (despite the known fact that the averages should be the same!) should already be a big warning sign that something is not quite right.

You probably missed the 197 other samples. I can't upload a full set of 200 samples encoded with two or three encoders and asking to other members to test them. That would be silly. And as I said, these three samples are just here to illustrate one problem I discovered. I never said that Nero Digital has regressed everywhere (it would be silly), but just that some enhancement that were appreciable with the previous 'fast' encoder are not reported to the new LC encoder which was yet supposed to be this old fast encoder.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 14:35:32
Quote
It might be suboptimal on these samples. So what?
[a href="index.php?act=findpost&pid=343131"][{POST_SNAPBACK}][/a]

  Is this question serious?
BTW, it's not "it might" but "it is".

Quote
How does this allow you to make any kind of conclusion by letting someone else confirm that a sample encoded with 3x the bits is better?

You made a claim (that most people won't perceive a regression), and is it to me to proove that you're wrong? As I said, I only post three samples as example, just to see if the regression I heard are something fictive and/or personal, or if these regressions are also audible to other people.
What I'm trying to prove is that Nero Digital v.4.2.1.0 has problem that Nero Digital v.3.2.0.15 didn't have with 'fast mode'. It's perfectly true that the previous encoders bloats the bitrate, but at least this VBR mode iwas doing what VBR are for: keeping a constant quality. I can't say the same for current VBR -internet mode. And two persons confirm it.
BTW, the S44 sample correspond to the "biniou sample" I submitted long time ago. The problem was corrected once, but is now resurrected with the new encoder.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Garf on 2005-11-19 14:54:29
Quote
You made a claim (that most people won't perceive a regression), and is it to me to proove that you're wrong?
[a href="index.php?act=findpost&pid=343140"][{POST_SNAPBACK}][/a]


I remember quite well what happened the last time I did want to prove my claim (32kbps HE-AAC): I wasn't allowed to.

Quote
As I said, I only post three samples as example, just to see if the regression I heard are something fictive and/or personal, or if these regressions are also audible to other people.


You are still missing the point entirely: I don't disagree specific samples can be worse. I disagree the encoder is overall performing less good, that is, on MULTIPLE GENRES and MULTPLE LISTENERS.

Since you now even seem to disagree that is an important distinction ("You repeated several times in the last days that such tests are not "proper". Your opinion, not mine."), I have nothing more to say.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 15:00:46
Quote
Quote
You made a claim (that most people won't perceive a regression), and is it to me to proove that you're wrong?
[a href="index.php?act=findpost&pid=343140"][{POST_SNAPBACK}][/a]


I remember quite well what happened the last time I did want to prove my claim (32kbps HE-AAC): I wasn't allowed to.
[a href="index.php?act=findpost&pid=343145"][{POST_SNAPBACK}][/a]

Come on. You're allowed to do it (didn't you performed it in fact?  ), and you know it. You just can't set a test with wrong contenders and choose you own samples.

Quote
You are still missing the point entirely: I don't disagree specific samples can be worse. I disagree the encoder is overall performing less good, that is, on MULTIPLE GENRES and MULTPLE LISTENERS.

Is it with me that you disagree on this point? Did I said that the new Nero is generally worse than the previous one?  If the answer is positive, could you give me the corresponding link? Thanks.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Lyx on 2005-11-19 15:04:48
Garf, Nero AAC may to some extend be your baby.... however, a certain kind of reaction from devs towards listening tests (i'm primarily not talking about the "what" but the "how") torpedoed musepacks public image on ha.org not so long ago.

I asume i'm not alone with this impression when i say that AAC has been in the making for a LONG time, it has been sold to us as the "next big thing" and "the future of lossy encoding".... and when during the previous years the tests didn't look good, we were told over and over "well, its still new and requires some more tuning and more research, then it will rock"..... with nero AAC, this is what we've been told for almost EVERY upcoming version: "yes, there are some errors, but the next version will be much better and probably fix this and that"....

Now, when after years in the medium bitrate arena quicktime AAC can barely compete with vorbis, which does NOT have many of the patented toys available to AAC...... and nero AAC can barely beat LAME-MP3.... the format which supposedly is "obsolete"...... then maybe it's no surprise that people aren't very euphorous about the performance of AAC, especially nero AAC? Or is this encoder only gonna be useful for narrowband-scenarios? (question is intentionally worded in a provocative way)

- Lyx
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: lexor on 2005-11-19 16:03:25
guruboolez great test, you trully have the patience of Job.

Also, Garf claimed several times that "most people" would perceive new encoder as better, is he referring to the old (Roberto and Co.) listening tests or is there another study I've missed? Anyone has a link to the study in which those "most" people participated?
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 16:15:17
Quote
guruboolez great test, you trully have the patience of Job.

Also, Garf claimed several times that "most people" would perceive new encoder as better, is he referring to the old (Roberto and Co.) listening tests or is there another study I've missed? Anyone has a link to the study in which those "most" people participated?
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=343161")

Nobody has so far participated in a listening test proving that "most people" are considering the new encoder as better than the previous one. Not even me. Not on HA.org, Doom9 or any other known board or website.
He's only referring to his own beliefs, disguised as valid and general claims.

BTW, latest listening collective test involving Nero Digital LC-AAC was performed with [a href="http://www.rjamorim.com/test/aac128v2/presentation.html]Nero AAC 2.6.2.0[/url]. It was 21 months ago. I don't think that Garf's reference was this very old encoder when he talked about progress of the latest one. He probably refered to the last one of the 3.2.XXX series.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Ivan Dimkovic on 2005-11-19 17:22:49
Guys... I really think this discussion is not getting anywhere

I suggest to wait 10-or-so days for the multiformat ~128 kbps listening test, where we all should take part - and also Nero AAC will be upgraded with some necessary quality fixes collected during the testing.

With sufficient number of trained listeners, and multiple genres - we'll be able to tell the actual state of audio compression and how much codecs did improve during recent time.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 17:34:50
If you have some time, check the current highest VBR mode "transcoding", which often provide a lower bitrate than 2nd best preset ("audiophile"). Low volume parts are apparently affected; and it may have an audible impact when (a very strong) replaygain track mode is applied. See for example:

http://guruboolez.free.fr/ENSEMBLE/E20_MOD...rings_quiet.ofr (http://guruboolez.free.fr/ENSEMBLE/E20_MODERN_ORCHESTRAL_E_strings_quiet.ofr)
http://guruboolez.free.fr/VOICES/V03_CHORUS_Female_A.ofr (http://guruboolez.free.fr/VOICES/V03_CHORUS_Female_A.ofr)

With V03, even --extreme preset (two steps below transcoding!) has higher bitrate (and quality also). I suppose that's a bug or a tuning issue, because currently Nero Digital "transcoding" sound worse than LAME 3.97b1 -b128 with this sample²

² after replaygaining.


EDIT: changed --normal to --extreme
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Ivan Dimkovic on 2005-11-19 17:42:40
Well, next week is gonna be busy
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Halcyon on 2005-11-19 18:16:32
Thank you very much for going through the FSB 3rd degree questioning ordeal (FSB is the successor to KGB, btw)

Your comments about methodology and motivation were very helpful, thank you again.

BTW, by detection accuracy I was merely referring to absolute tresholds of detection.

I've noticed myself, that just upgrading headphones made previously completely unaudible distortions audible to me.

Hence, at least on certain type of distortions, the treshold was clearly lowered, enabling me to hear and grade artifacts that would have gone unnoticed previously.

regards,
halcyon
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 18:43:25
Quote
I've noticed myself, that just upgrading headphones made previously completely unaudible distortions audible to me.
[a href="index.php?act=findpost&pid=343192"][{POST_SNAPBACK}][/a]

If I replace my Beyerdynamic with a set of cheap erbuds (Sennheiser MX550), several distortion cease to be audible (but not pre-echo, still perceptible). But I'm not fully convinced that a very expensive of headphone (like Stax ones) would really really help me to catch additionnal encoding issues.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: yulyo! on 2005-11-19 20:17:59
Hy guys,
I feel terrible.
I feel that i am the one who started those ...crazy discussions. With only one or two replies (and one mistake  )
I think we should wait for the next 128kbs test and let Ivan and Garf improve their work.
Also, we don't have to forget ONE thing: this test was performed on clasical music ONLY and with guruboolez's ears and equipament ONLY.
Yes, is surprising that Nero AAC is fighting to beat Lame and not Vorbis, but this is it. For now.
So, in conclusion, i think we should wait a few days for Sebastian's test and then start those contradictions again
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-11-19 20:27:25
Quote
Also, we don't have to forget ONE thing: this test was performed on clasical music ONLY[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=343224")

It's precisely ONE thing you can immediately forget...
[a href="http://audiotests.free.fr/tests/2005.11/results_gr5.png]http://audiotests.free.fr/tests/200...results_gr5.png[/url]

Take a look on results before commenting them, please
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: lexor on 2005-11-19 20:55:43
Quote
Also, we don't have to forget ONE thing: this test was performed on clasical music ONLY and with guruboolez's ears and equipament ONLY.
[a href="index.php?act=findpost&pid=343224"][{POST_SNAPBACK}][/a]

did you read the write up?

/EDIT: oops, I guess I should read replies before posting mine, guru got there first.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: clintb on 2005-11-19 22:11:06
Ok, maybe I'm dense, but barring that, my ability to interpret the sea of numbers presented in the graphs is telling me that iTunes is coming super close to LAME.  Is that correct?
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: smz on 2005-11-19 22:33:25
Quote
Ok, maybe I'm dense, but barring that, my ability to interpret the sea of numbers presented in the graphs is telling me that iTunes is coming super close to LAME.  Is that correct?
[a href="index.php?act=findpost&pid=343246"][{POST_SNAPBACK}][/a]


My interpretation is that iTunes @ ~130 Kb/s comes super close to LAME @ ~196 Kb/s (the "high anchor", LAME 3.97 beta 1 –V2 --vbr new) and is definitely better than LAME @ ~130 Kb/s.

Quite embarassing for a strong LAME supporter (like myself) and an an allergic to apples (like the very same myself).

Sergio
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: clintb on 2005-11-19 22:38:35
Quote
Quote
Ok, maybe I'm dense, but barring that, my ability to interpret the sea of numbers presented in the graphs is telling me that iTunes is coming super close to LAME.  Is that correct?
[a href="index.php?act=findpost&pid=343246"][{POST_SNAPBACK}][/a]


My interpretation is that iTunes @ ~130 Kb/s comes super close to LAME @ ~196 Kb/s (the "high anchor", LAME 3.97 beta 1 –V2 --vbr new) and is definitely better than LAME @ ~130 Kb/s.

Quite embarassing for a strong LAME supporter (like myself) and an an allergic to apples (like the very same myself).

Sergio
[a href="index.php?act=findpost&pid=343254"][{POST_SNAPBACK}][/a]

And I just did a FLAC>3.97b1 -V 2 --vbr new conversion on all my music for the new 5th Gen 60GB iPod.  Might have to try out 160K VBR iTunes and get back space for photos and videos.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: smz on 2005-11-19 22:41:10
Quote
And I just did a FLAC>3.97b1 -V 2 --vbr new conversion on all my music for the new 5th Gen 60GB iPod.  Might have to try out 160K VBR iTunes and get back space for photos and videos.
[a href="index.php?act=findpost&pid=343257"][{POST_SNAPBACK}][/a]


I hope you kept your FLACs, like I keep my WavPaks...
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: QuantumKnot on 2005-11-20 00:38:52
Quote
Garf, Nero AAC may to some extend be your baby.... however, a certain kind of reaction from devs towards listening tests (i'm primarily not talking about the "what" but the "how") torpedoed musepacks public image on ha.org not so long ago.


I agree.  I think it is esp. good for devs to welcome listening tests by other people who volunteer to do quality-testing for you.  I know Aoyumi follows these tests very closely and I believe it benefits his Vorbis development.

I just wanted to make a comment about iTunes/QT AAC.  As guru pointed out in a previous reply, I am somewhat amazed at how its ABR-like nature can give such great quality.  In fact, I've made the comment long ago about iTunes AAC on castanets compared with Vorbis at 128 kbps.  iTunes/AAC barely moved in bitrate, yet produced much lower pre-echo than vanilla Vorbis with inflated bitrates.    When I did those ABX tests last night, I noticed that the iTunes files had very minimal bitrate fluctuation, yet I was unable to ABX it against the original.    Is this magic or something?    The devs at Apple certainly are doing a great job, esp when they are giving their AAC encoder away for free.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: ErikS on 2005-11-20 00:51:18
Quote
Quote
guruboolez great test, you trully have the patience of Job.

Also, Garf claimed several times that "most people" would perceive new encoder as better, is he referring to the old (Roberto and Co.) listening tests or is there another study I've missed? Anyone has a link to the study in which those "most" people participated?
[a href="index.php?act=findpost&pid=343161"][{POST_SNAPBACK}][/a]

Nobody has so far participated in a listening test proving that "most people" are considering the new encoder as better than the previous one. Not even me. Not on HA.org, Doom9 or any other known board or website.
He's only referring to his own beliefs, disguised as valid and general claims.
[a href="index.php?act=findpost&pid=343163"][{POST_SNAPBACK}][/a]

I bet he listened to it several times himself and also let other people at Ahead listen. Just that those listening tests are not public...
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: clintb on 2005-11-20 01:54:30
Quote
Quote
And I just did a FLAC>3.97b1 -V 2 --vbr new conversion on all my music for the new 5th Gen 60GB iPod.  Might have to try out 160K VBR iTunes and get back space for photos and videos.
[a href="index.php?act=findpost&pid=343257"][{POST_SNAPBACK}][/a]


I hope you kept your FLACs, like I keep my WavPaks...
[a href="index.php?act=findpost&pid=343258"][{POST_SNAPBACK}][/a]

Oh man, after all that EAC ripping to cue/single files, proper case for the titles, checking date and genre...yeah, the FLAC files are kept and doubly so.  Once, on the primary/usage drive, secondly on an external drive and soon on DVD.

BTW, anybody looking for good DVD media to backup, check Micro Center (www.microcenter.com) for their in-house brand "WinData"....it's TY.  Right now, they're $8.99 for a 50pk spindle.  That's right, $8.99 for 50 Tayo Yuden 8x DVD+R.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: clintb on 2005-11-20 01:56:26
Quote
Quote
Garf, Nero AAC may to some extend be your baby.... however, a certain kind of reaction from devs towards listening tests (i'm primarily not talking about the "what" but the "how") torpedoed musepacks public image on ha.org not so long ago.


I agree.  I think it is esp. good for devs to welcome listening tests by other people who volunteer to do quality-testing for you.  I know Aoyumi follows these tests very closely and I believe it benefits his Vorbis development.

I just wanted to make a comment about iTunes/QT AAC.  As guru pointed out in a previous reply, I am somewhat amazed at how its ABR-like nature can give such great quality.  In fact, I've made the comment long ago about iTunes AAC on castanets compared with Vorbis at 128 kbps.  iTunes/AAC barely moved in bitrate, yet produced much lower pre-echo than vanilla Vorbis with inflated bitrates.    When I did those ABX tests last night, I noticed that the iTunes files had very minimal bitrate fluctuation, yet I was unable to ABX it against the original.    Is this magic or something?    The devs at Apple certainly are doing a great job, esp when they are giving their AAC encoder away for free.
[a href="index.php?act=findpost&pid=343289"][{POST_SNAPBACK}][/a]


This brings up a good question: Who's doing the development on the iTunes AAC encoder?  Anybody that frequents this board, I wonder?
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Nayru on 2005-11-20 02:06:00
Quote
I asume i'm not alone with this impression when i say that AAC has been in the making for a LONG time, it has been sold to us as the "next big thing" and "the future of lossy encoding".... and when during the previous years the tests didn't look good, we were told over and over "well, its still new and requires some more tuning and more research, then it will rock"..... with nero AAC, this is what we've been told for almost EVERY upcoming version: "yes, there are some errors, but the next version will be much better and probably fix this and that"....
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=343148")
Oh, Nero claims much more than that.  For a good laugh read: [a href="http://ww2.nero.com/nerodigital/enu/Nero_Digital_Audio_highlights.html]http://ww2.nero.com/nerodigital/enu/Nero_D...highlights.html[/url]

"CD quality stereo at 48 kb/s"  "Transparent quality at 128 kb/s"  "MP3 quality with 50 % of the  space."

It's not surprising that customers are upset when they find out it's not true.

Quote
Now, when after years in the medium bitrate arena quicktime AAC can barely compete with vorbis, which does NOT have many of the patented toys available to AAC...... and nero AAC can barely beat LAME-MP3.... the format which supposedly is "obsolete"...... then maybe it's no surprise that people aren't very euphorous about the performance of AAC, especially nero AAC? Or is this encoder only gonna be useful for narrowband-scenarios? (question is intentionally worded in a provocative way)

[a href="index.php?act=findpost&pid=343148"][{POST_SNAPBACK}][/a]
Both AAC and Vorbis use quantized MDCT with two different block sizes, followed by Huffman coding.  I would be surprised if there was a substantial difference in coding efficiency between the two formats.  What are the patented toys available to AAC?  SBR?  LTP?  SSR?  It'd be interesting to know how much improvement those actually make...
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: rjamorim on 2005-11-20 03:19:47
Quote
This brings up a good question: Who's doing the development on the iTunes AAC encoder?  Anybody that frequents this board, I wonder?[a href="index.php?act=findpost&pid=343306"][{POST_SNAPBACK}][/a]


Yes. But he prefers to remain anonymous because of some scary Apple policies.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: clintb on 2005-11-20 04:41:46
Quote
Quote
This brings up a good question: Who's doing the development on the iTunes AAC encoder?  Anybody that frequents this board, I wonder?[a href="index.php?act=findpost&pid=343306"][{POST_SNAPBACK}][/a]


Yes. But he prefers to remain anonymous because of some scary Apple policies.
[a href="index.php?act=findpost&pid=343322"][{POST_SNAPBACK}][/a]

He?  It's not one person, is it?  If so, hat's off to him for a fine job.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Maurits on 2005-11-20 12:12:24
Quote
Quote
This brings up a good question: Who's doing the development on the iTunes AAC encoder?  Anybody that frequents this board, I wonder?[a href="index.php?act=findpost&pid=343306"][{POST_SNAPBACK}][/a]


Yes. But he prefers to remain anonymous because of some scary Apple policies.
[a href="index.php?act=findpost&pid=343322"][{POST_SNAPBACK}][/a]

And probably because he'd be bullied by hundreds on this board about when gapless playback is going to be supported by Apple. 
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: de Mon on 2005-11-20 12:57:06
Quote
Both AAC and Vorbis use quantized MDCT with two different block sizes, followed by Huffman coding.  I would be surprised if there was a substantial difference in coding efficiency between the two formats.  What are the patented toys available to AAC?  SBR?  LTP?  SSR?  It'd be interesting to know how much improvement those actually make...
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=343307")


[a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=34075]http://www.hydrogenaudio.org/forums/index....showtopic=34075[/url]
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: rjamorim on 2005-11-20 13:41:58
Quote
He?  It's not one person, is it?  If so, hat's off to him for a fine job.[a href="index.php?act=findpost&pid=343338"][{POST_SNAPBACK}][/a]


It's one person (that I know of) that is member of this forum. But I suspect he's not alone working on the AAC encoder...

Quote
And probably because he'd be bullied by hundreds on this board about when gapless playback is going to be supported by Apple.  [a href="index.php?act=findpost&pid=343403"][{POST_SNAPBACK}][/a]


I bullied him already :B

He said he wouldn't mind including it in the encoder, but it doesn't depend only on him. It also depends on the QuickTime division, the iTunes division, the iPod division... For that reason, it'll probably only happen when a decision comes from above.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Maurits on 2005-11-20 13:58:55
Quote
Quote
And probably because he'd be bullied by hundreds on this board about when gapless playback is going to be supported by Apple.  [a href="index.php?act=findpost&pid=343403"][{POST_SNAPBACK}][/a]


I bullied him already :B

He said he wouldn't mind including it in the encoder, but it doesn't depend only on him. It also depends on the QuickTime division, the iTunes division, the iPod division... For that reason, it'll probably only happen when a decision comes from above.
[a href="index.php?act=findpost&pid=343427"][{POST_SNAPBACK}][/a]

I can imagine that being the problem. I think it would be relatively easy to implement it in the encoder but harder (more complex) to implement it in the decoders in iTunes and iPod. It would be odd when the files made by iTunes support gapless but iTunes wouldn't be able to playback gapless. Or what if the iPod could play gapless but iTunes couldn't, or the other way around, or...or...

I believe Apple said they found gapless playback irrelevant. That's usually Applespeak for "We're going to introduce it very soon..." 
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: sTisTi on 2005-11-20 14:36:57
Quote
BTW, anybody looking for good DVD media to backup, check Micro Center (www.microcenter.com) for their in-house brand "WinData"....it's TY.  Right now, they're $8.99 for a 50pk spindle.  That's right, $8.99 for 50 Tayo Yuden 8x DVD+R.
[a href="index.php?act=findpost&pid=343305"][{POST_SNAPBACK}][/a]

Completely OT here, but are you sure these are not so-called "fake" TY? There has been a lengthy discussion about this at cdfreaks, you might want to check it out before trusting your data to these media.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2005-12-12 07:48:14
Quote
Would it be possible for you to create some small webpage pointing to your tests, like the one of rjamorim (http://www.rjamorim.com/test/), so its easy to see whats going on and compare the results aso... atm it seems more to me your results are vanishing in the depths of the forum
if you cant make a own page, maybe you can create some thread carrying that info?
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=349404")

Yes, I could.

vorbis GT2 vs. vorbis PCVS - 12 samples
extension of the AAC 128 kbs LT - part 2
[a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=10555&hl=]http://www.hydrogenaudio.org/forums/index....topic=10555&hl=[/url]

WMA9 vs WMA9PRO 12 samples test
extension of the AAC 128 kbps LT.
http://www.hydrogenaudio.org/forums/index....topic=10551&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=10551&hl=)

Personal multiformat listening test at ~130 kbps
based on classical (baroque) music only
http://www.hydrogenaudio.org/forums/index....topic=14091&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=14091&hl=)

MP3 decoders test : MAD isn't so good! (for me...)
MAD vs LAME vs Winamp 5 vs foobar2000
http://www.hydrogenaudio.org/forums/index....topic=17728&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=17728&hl=)

128 kb Multiformat listening test...
...based on classical music samples ONLY
http://www.hydrogenaudio.org/forums/index....topic=16395&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=16395&hl=)

listening test at 160 kbps
pre-echo with aoTuV, GT3...
http://www.hydrogenaudio.org/forums/index....topic=22495&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=22495&hl=)

another lossless performance comparison
...but on classical music only
http://www.hydrogenaudio.org/forums/index....topic=28441&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=28441&hl=)

lame 3.90.3 vs lame 3.96.1 at ~130 kbps
ABR and VBR
http://www.hydrogenaudio.org/forums/index....topic=29422&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=29422&hl=)

   
MPC vs OGG VORBIS vs MP3 at 175 kbps
listening test on non-killer samples
http://www.hydrogenaudio.org/forums/index....topic=23355&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=23355&hl=)

AAC: Ahead vs Apple (end 2004)
one year of progress
http://www.hydrogenaudio.org/forums/index....topic=29924&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=29924&hl=)

lame 3.97 alpha 5 testing thread
tests & results
http://www.hydrogenaudio.org/forums/index....topic=30547&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=30547&hl=)

Vorbis quality – wrong direction?
RC3 against post-final encoder
http://www.hydrogenaudio.org/forums/index....topic=18359&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=18359&hl=)

Ahead AAC VBR vs Vorbis aoTuV beta 3
at ~130 kbps with classical music
http://www.hydrogenaudio.org/forums/index....topic=29925&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=29925&hl=)

Short re-encoding blind listening test
wavpack - mp3 - mpc - aac - vorbis
http://www.hydrogenaudio.org/forums/index....topic=32440&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=32440&hl=)

Ogg Vorbis and Nero AAC
microattacks & microdetails VBR 140 kbps
http://www.hydrogenaudio.org/forums/index....topic=32080&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=32080&hl=)

1.01j encoder vs 1.15u
listening test inside
http://www.hydrogenaudio.org/forums/index....topic=34911&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=34911&hl=)

80 kbps personal listening test (summer 2005)
AAC MP3 Ogg Vorbis WMA
http://www.hydrogenaudio.org/forums/index....topic=35438&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=35438&hl=)

lame 3.98 alpha 2 testing thread
vbr new & athaa-sensitivity test at V5
http://www.hydrogenaudio.org/forums/index....topic=37973&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=37973&hl=)

HE-AAC v.1 & v.2 comparison
Winamp vs Helix vs Nero Digital
http://www.hydrogenaudio.org/forums/index....topic=36868&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=36868&hl=)

MPC vs VORBIS vs MP3 vs AAC at 180 kbps
2nd checkup with classical music
http://www.hydrogenaudio.org/forums/index....topic=36465&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=36465&hl=)








____
LAME ALPHA
http://www.hydrogenaudio.org/forums/index....81&#entry300681 (http://www.hydrogenaudio.org/forums/index.php?showtopic=34270&st=0&p=300681&#entry300681)


[span style='font-size:14pt;line-height:100%']DIRECT LINKS[/span]:


alpha 5 -V4 - 20 samples (http://www.hydrogenaudio.org/forums/index.php?showtopic=30547&view=findpost&p=264991) and alpha 5 -V5 - 6 samples (http://www.hydrogenaudio.org/forums/index.php?showtopic=30547&view=findpost&p=264992)
alpha 6 -V2 - 20 samples (http://www.hydrogenaudio.org/forums/index.php?showtopic=30631&view=findpost&p=266785)
alpha 7 -V4 - 54 samples (http://www.hydrogenaudio.org/forums/index.php?showtopic=31255&view=findpost&p=275554)
alpha 8 -V3 - 20 samples (http://www.hydrogenaudio.org/forums/index.php?showtopic=32132&view=findpost&p=281442) and alpha 8 -V2 - 20 samples (http://www.hydrogenaudio.org/forums/index.php?showtopic=32132&view=findpost&p=281443)


Lame test version - may 2005
http://www.hydrogenaudio.org/forums/index....ndpost&p=300065 (http://www.hydrogenaudio.org/forums/index.php?showtopic=34001&view=findpost&p=300065)

Lame test version (June 2005)
http://www.hydrogenaudio.org/forums/index....ndpost&p=304757 (http://www.hydrogenaudio.org/forums/index.php?showtopic=34605&view=findpost&p=304757)


Quote
wouldnt it be more interesting to compare helix/ct to apple in the future and not always nero over and over again? (i know nero is well established (again is used in sebastians public test), but this doesnt mean that there arent better codecs)

Winamp (Coding Tech & Dolby) AAC implementations are less interesting than Nero Digital. It has nothing to do with quality, but with progress. Nero Digital is updated more often. To give you a simple fact: Winamp still embbed Dolby AAC 1.0 for two years now! No update since...
And Coding Tech AAC encoder is still handicaped by a 15 KHz lowpass, making this encoder too easy to detect on ABCHR evaluation. You can test it if you want. But in my opinion, CT AAC encoder (Winamp, Helix) is more interesting for HE profile than LC one.

I'm testing Nero Digital because it's often updated. It's also interesting because Ivan is present on the forum; he could comment the test and propose some improved encoders. Exactly like LAME developers and Aoyumi. There were also a lot of changes during three years. The first encoders I've tested were really poor with classical (see my first tests); the lastest ones are much better. Too bad that the Nero Digital team has released an old generation encoder with a version number corresponding to a major release (aacenc32 v.4 and aac.dll v.3). The current new generation encoders shows interesting improvements compared to the one I've tested here:
- better handling of very tonal signal
- no ringing anymore on low volume part
- much less distortions with harpsichord.

And compared to my beloved old aacenc32 v3.xxx "fast", the new encoder has no bloated bitrate anymore with some kind of sample/music and is able to produce the same kind of high quality.


I'm currently sick with listening test. I must force myself to finish Sebastian's 18 samples. I can't make extensive listening test of any encoder now, including latests lame alpha or newest Nero encoder. I can only perform quick and small comparison. It's enough to appreciate some improvements of latest Nero Digital, but not enough to perform a complete and rigorous evaluation.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Gambit on 2005-12-12 22:50:47
The highly interesting off-topic discussion was split here (http://www.hydrogenaudio.org/forums/index.php?showtopic=39683&st=0).
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: bond on 2005-12-13 00:05:51
well my proposal for guru to create a small webpage with his tests and to maybe use ct in future comparisons was split too, i hope guru might still consider it
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: eltoder on 2006-01-21 10:40:56
Fantastic job, guru. Don't you have some kind of HA award yet?

Quote
The good surprise comes from LAME MP3, which get the best mark (3,95)

Am I missing something, or plot says that it's Vorbis who get 3,95 and LAME get 3,94?
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: pepoluan on 2006-01-22 17:26:21
Quote
Yes, I could.

vorbis GT2 vs. vorbis PCVS - 12 samples
extension of the AAC 128 kbs LT - part 2
http://www.hydrogenaudio.org/forums/index....topic=10555&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=10555&hl=)

... and so on...
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=349584")
Whoa... a treasure trove of listening tests... 

Guruboolez, if you'll be kind enough to indicate the date of each listening test you did, I'll gladly add them to the ever-expanding list of listening tests at [a href="http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests]this HA Wiki page[/url].

Oh and PM me when you do that. So I know when to recheck this thread :-)
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: evereux on 2006-01-22 17:37:10
Check the date of the original posts?
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: pepoluan on 2006-01-22 17:48:14
Quote
Check the date of the original posts?
[a href="index.php?act=findpost&pid=358990"][{POST_SNAPBACK}][/a]
That might do it... however I think it is better for guru to edit his posting above for posterity...
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2006-01-23 07:49:39
Quote
Fantastic job, guru. Don't you have some kind of HA award yet?

Quote
The good surprise comes from LAME MP3, which get the best mark (3,95)

Am I missing something, or plot says that it's Vorbis who get 3,95 and LAME get 3,94?
[a href="index.php?act=findpost&pid=358736"][{POST_SNAPBACK}][/a]

Youp...
Gecko already noticed it... I finally changed the plot, but I can't currently upload it (my new ftp needs a provider access IP to access to the ftp). This week end

pepoluan> I publish tests few hours or days after I finish them. Evereux's advice is fine
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: pepoluan on 2006-01-24 12:00:48
About half have been integrated here (http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests).

Please check it up I'm sure there are mistakes. I have a terrible headache and can't really concentrate.

Will try to finish it up tomorrow.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: bond on 2006-01-24 12:06:36
Quote
About half have been integrated here (http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests).

Please check it up I'm sure there are mistakes. I have a terrible headache and can't really concentrate.

Will try to finish it up tomorrow.
[a href="index.php?act=findpost&pid=359380"][{POST_SNAPBACK}][/a]

great thingie this page! thx a lot!
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: pepoluan on 2006-01-26 12:47:17
Dang. Those tables make me dizzy.

All guru's tests, I think, have been listed in this HA Wiki page (http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests).

I am darned sure there are mistakes. For instance, I'm not sure of the links. But I have no time today, so please check the page out and tell me what's the mistakes. Or fix it yourself if possible 

Oh and please forgive the coloring. It's not yet finished. Still an "alpha version" page  will "go beta" if you guys tell me where the bugs are...

Hmmm.... just one drawback here... nearly all tests are guru's... where are the others...
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: pepoluan on 2006-01-26 17:40:35
Uhh, guru, I noticed your lossless test (http://www.hydrogenaudio.org/forums/index.php?showtopic=28441&hl=) is no longer accessible...

So I haven't put that in the listening tests page (http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests).
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2006-01-28 06:35:52
Quote
Uhh, guru, I noticed your lossless test (http://www.hydrogenaudio.org/forums/index.php?showtopic=28441&hl=) is no longer accessible...

So I haven't put that in the listening tests page (http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests).
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=359947")

[a href="http://guruboolez.free.fr/lossless/]http://guruboolez.free.fr/lossless/[/url]

Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: =trott= on 2006-01-28 08:25:45
Quote
My interpretation is that iTunes @ ~130 Kb/s comes super close to LAME @ ~196 Kb/s (the "high anchor", LAME 3.97 beta 1 –V2 --vbr new) and is definitely better than LAME @ ~130 Kb/s.
[a href="index.php?act=findpost&pid=343254"][{POST_SNAPBACK}][/a]


...which just goes to show that statistics can prove anything. In sebastian's multiformat test mp3 and in fact all other contenders were statistically tied. (possibly making an exception to vorbis aotuv). This also holds true for this test, I believe. (excepting the one classical test...)

I'd rather say that itunes 130 comes super close to lame 196 while still not being all that much better than lame 140.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2006-01-28 08:33:32
Quote
Quote
My interpretation is that iTunes @ ~130 Kb/s comes super close to LAME @ ~196 Kb/s (the "high anchor", LAME 3.97 beta 1 –V2 --vbr new) and is definitely better than LAME @ ~130 Kb/s.
[a href="index.php?act=findpost&pid=343254"][{POST_SNAPBACK}][/a]


In sebastian's multiformat test mp3 and in fact all other contenders were statistically tied. (possibly making an exception to vorbis aotuv). This also holds true for this test, I believe. (excepting the one classical test...)
[a href="index.php?act=findpost&pid=360295"][{POST_SNAPBACK}][/a]

Not exactly. There's a clear hierarchy for both tests:

NON-CLASSICAL (50 samples)
1. high anchor
2. iTunes AAC and Vorbis aoTuV
4. LAME MP3 and Nero Digital AAC

CLASSICAL (150 samples)
1. high anchor
2. iTunes AAC and Vorbis aoTuV
4. LAME MP3
5. Nero Digital AAC

Most tested contenders are not statistically tied
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: =trott= on 2006-01-28 08:50:43
Quote
Not exactly. There's a clear hierarchy for both tests:

NON-CLASSICAL (50 samples)
1. high anchor
2. iTunes AAC and Vorbis aoTuV
4. LAME MP3 and Nero Digital AAC

CLASSICAL (150 samples)
1. high anchor
2. iTunes AAC and Vorbis aoTuV
4. LAME MP3
5. Nero Digital AAC

Most tested contenders are not statistically tied
[a href="index.php?act=findpost&pid=360296"][{POST_SNAPBACK}][/a]


I must have misunderstood something about sebastian's test then...excepting the low anchor there of course, can they not be considered as tied? I see the (relatively small) difference in this test, but in sebastian's I...cannot
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: pepoluan on 2006-01-28 09:13:03
Quote
Quote
Uhh, guru, I noticed your lossless test (http://www.hydrogenaudio.org/forums/index.php?showtopic=28441&hl=) is no longer accessible...

So I haven't put that in the listening tests page (http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests).
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=359947")
[a href="http://guruboolez.free.fr/lossless/]http://guruboolez.free.fr/lossless/[/url]


[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=360282")
Ahh, merci beaucoup!

I've put that in the [a href="http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests]listening tests page[/url], External Tests section.

BTW, the listening tests page (http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests) has "gone beta"! Yay! Feel free to check it out.

Side note:  FLAC lost by a wide margin?? Gee I really must check out LA...
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2006-01-28 09:14:14
Quote
I must have misunderstood something about sebastian's test then.
[a href="index.php?act=findpost&pid=360301"][{POST_SNAPBACK}][/a]


Quote
This also holds true for this [i.e. guruboolez's one] test, I believe. (excepting the one classical test...)
[a href="index.php?act=findpost&pid=360295"][{POST_SNAPBACK}][/a]


My comment was about the last sentence. All contenders are indeed tied for Sebastian's tests, but not for mine
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: guruboolez on 2006-01-28 09:16:10
Quote
Side note:  FLAC lost by a wide margin?? Gee I really must check out LA...
[a href="index.php?act=findpost&pid=360306"][{POST_SNAPBACK}][/a]

Off-topic: there are no looser or winner with a lossless comparison. FLAC is also close to top for decoding speed.

EDIT: you should rather link the WIKI lossless page which offers several links for different lossless comparison. Mine is included
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: pepoluan on 2006-01-28 09:50:10
Quote
Quote
Side note:  FLAC lost by a wide margin?? Gee I really must check out LA...
[a href="index.php?act=findpost&pid=360306"][{POST_SNAPBACK}][/a]
Off-topic: there are no looser or winner with a lossless comparison. FLAC is also close to top for decoding speed.

EDIT: you should rather link the WIKI lossless page which offers several links for different lossless comparison. Mine is included
[a href="index.php?act=findpost&pid=360308"][{POST_SNAPBACK}][/a]
Uhh... stupid me... I just realized... it is a Listening Test... lossless tests surely do not belong here... 

I've removed the link to guru's lossless test site.

By "lost", I mean that FLAC's compression is ... how I put it? not that good ... 
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: vinnie97 on 2006-01-28 22:37:02
Quote
Quote
Quote
Side note:  FLAC lost by a wide margin?? Gee I really must check out LA...
[a href="index.php?act=findpost&pid=360306"][{POST_SNAPBACK}][/a]
Off-topic: there are no looser or winner with a lossless comparison. FLAC is also close to top for decoding speed.

EDIT: you should rather link the WIKI lossless page which offers several links for different lossless comparison. Mine is included
[a href="index.php?act=findpost&pid=360308"][{POST_SNAPBACK}][/a]
Uhh... stupid me... I just realized... it is a Listening Test... lossless tests surely do not belong here... 

I've removed the link to guru's lossless test site.

By "lost", I mean that FLAC's compression is ... how I put it? not that good ... 
[a href="index.php?act=findpost&pid=360313"][{POST_SNAPBACK}][/a]

Yes, but it has wider support than LA, encodes/decodes faster and has error correction.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: rjamorim on 2006-01-29 01:24:28
Quote
I've put that in the listening tests page (http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests), External Tests section.

BTW, the listening tests page (http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests) has "gone beta"! Yay! Feel free to check it out.[a href="index.php?act=findpost&pid=360306"][{POST_SNAPBACK}][/a]


Jesus, dude, you didn't link to ff123's test. It's like, all heresies rolled into one - he is pretty much the man behind this whole mess, for starters
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: pepoluan on 2006-01-29 06:14:57
Quote
Quote
I've put that in the listening tests page (http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests), External Tests section.

BTW, the listening tests page (http://wiki.hydrogenaudio.org/index.php?title=Listening_Tests) has "gone beta"! Yay! Feel free to check it out.[a href="index.php?act=findpost&pid=360306"][{POST_SNAPBACK}][/a]

Jesus, dude, you didn't link to ff123's test. It's like, all heresies rolled into one - he is pretty much the man behind this whole mess, for starters
[a href="index.php?act=findpost&pid=360440"][{POST_SNAPBACK}][/a]
  Uh... URL please? I promise it will be put in the next revision...

*hit head with a sandbag*  stupid me...

EDIT:

Uhh... /me = stupid^2 ... what's Google for    but I see rjamorim has beaten me to it  ...

Well I did add a link to ff123's artifact training page near the top... hope I can atone for my sin this way 
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: Bodhi on 2007-05-25 16:14:18
Few words to conclude the test…
It’s pretty clear that all encoders tested here correspond to a good or even a very good output quality. There are currently no winner between AAC (iTunes) and Vorbis. It’s funny to see that results are pretty close on the finish line when problems are so different. Encodings are not fully transparent, but quality is in my opinion excellent most often (but not always).
LAME offers to MP3 the chance to stay competitive against AAC and Vorbis. Not fully competitive, but the efficiency of this format forces the respect.
Nero Digital implementation of AAC is slightly disappointing, especially with classical music, which is still a weak point of this encoder. But the quality is far from disaster (it wasn’t the case two years ago), is on average really good, gets even better with “non-classical” music and should satisfy several users.
Last but not least, difference among all these encoders is really small (don't look too much on "zoomed" plots  )

But the average mark is somewhat misleading. LAME quality is ~0.5 point lower to iTunes or Vorbis, but it doesn’t mean for example that quality of encoded albums are 0,5 lower. This lower ranking is rather the expression of higher fragility than lower quality. LAME, and Nero Digital, are more inclined to serious distortions than Vorbis or iTunes AAC at the same bitrate. The concept of quality may be replaced with such encoders by the concept of strength or robustness. To illustrate this I made the following histogram (sorry for poor quality, I’ll change it later):

(http://audiotests.free.fr/tests/2005.11/results_distri.png)

Here, Vorbis and iTunes both get a mark comprise between 4.5 and 5.0 for 50% of the tested samples, whereas Nero only achieve this state (near-transparency or full transparency) for 20% of the same samples. With the classical group of samples, 30% of the them were ranked below 3.0 with Nero when iTunes or Vorbis got the same notation of less than 10% of the sample. The two winners are stronger, and could handle more situations than LAME and Nero Digital AAC.


Hi,

would you still conclude this test the same way with "today's codecs"?

I'm not asking for a new test but just your opinion>

Thank you.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: BaByB0y on 2007-05-29 02:29:00
Very nice 
Through the Test, iTunes AAC and Vorbis do good job :x
nXqd
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: TechVsLife on 2007-05-29 21:27:48
I'd like to test the harpsichord, sax solo, and any other classical music samples that are extremely distorted at v5 lame, at least to guruboolez (--I hope at least barely noticeable to me), to see if I need to go up to v3 or v2.  Is there a link to download those samples?
Thanks.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: halb27 on 2007-05-29 22:40:53
I'd like to test the harpsichord, sax solo, and any other classical music samples that are extremely distorted at v5 lame, at least to guruboolez (--I hope at least barely noticeable to me), to see if I need to go up to v3 or v2.  Is there a link to download those samples?
Thanks.

In tis thread: LAME problem samples (http://www.hydrogenaudio.org/forums/index.php?showtopic=39314#) you'll find harp40_1 which is the worst harpsichord sample to me.
It's not just a problem for Lame but a problem for many encoders (not just mp3), and usually it requires a higher quality setting than is usually necessary.

I don't know a sax problem but may be trumpet problems are similar.
There can be a tremolo issue with trumpets. There's a sample 'Trumpet: My Prince' (guess you'll find it when doing a HA Google search above) which has this tremolo issue with Lame 3.97 and 3.98 when using VBR, but also when using FhG CBR (I tried FhG which ships with current dbpowerAmp).
You can also find a trumpet problem in the above link (this was the very problem I started worrying about problem samples), but this problem is rather Lame specific at least when looking at mp3 (it's a problem to some other formats as well). 3.97final has improved on it, and with Lame 3.98b3 the problem is overcome (at least at a higher bitrate which I always use for having a safety margin).

I encourage you to use such a safety margin as well if you can afford the larger filesize. Looks like you're out for that.
Use for instance 3.98b3 -V1 or an ABR or CBR setting in the 200+ kbps range, for instance -b224 -h.
With such a setting music usually is transparent, and in those rare cases when it's not it's at least acceptable.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: TechVsLife on 2007-05-30 02:41:19
Thanks, I found it after googling "trumpet: My prince." 

The sax solo problem is seen in the scores given by guruboolez at the beginning of this thread.

Nothing leapt out at me on an attentive but not painfully concentrated listening, so I'd have to cheat by training myself in order to catch artifacts.  I'll keep a lossless archive in case I gradually acquire the Power (Curse?), or in case someone I know has It, but probably -V 5 is fine for me (I've only tested -V 2 on those samples).  The typical scratches on LP records seem to have been a much greater "distortion" than the anomalies people are talking about here?

I'd hate to think there's a perfect correlation between technical ability to hear the slightest changes, and ability to understand the music, but there's obviously some correlation.  On the other hand, there's prob. some inverse correlation between obsession with technical issues and musical understanding.  (unless someone comes up with an encoder that recreates or betters the original music (-V negative 2?), or that has the ability to rank it, e.g. Bach over Britten.)





I'd like to test the harpsichord, sax solo, and any other classical music samples that are extremely distorted at v5 lame, at least to guruboolez (--I hope at least barely noticeable to me), to see if I need to go up to v3 or v2.  Is there a link to download those samples?
Thanks.

In tis thread: LAME problem samples (http://www.hydrogenaudio.org/forums/index.php?showtopic=39314#) you'll find harp40_1 which is the worst harpsichord sample to me.
It's not just a problem for Lame but a problem for many encoders (not just mp3), and usually it requires a higher quality setting than is usually necessary.

I don't know a sax problem but may be trumpet problems are similar.
There can be a tremolo issue with trumpets. There's a sample 'Trumpet: My Prince' (guess you'll find it when doing a HA Google search above) which has this tremolo issue with Lame 3.97 and 3.98 when using VBR, but also when using FhG CBR (I tried FhG which ships with current dbpowerAmp).
You can also find a trumpet problem in the above link (this was the very problem I started worrying about problem samples), but this problem is rather Lame specific at least when looking at mp3 (it's a problem to some other formats as well). 3.97final has improved on it, and with Lame 3.98b3 the problem is overcome (at least at a higher bitrate which I always use for having a safety margin).

I encourage you to use such a safety margin as well if you can afford the larger filesize. Looks like you're out for that.
Use for instance 3.98b3 -V1 or an ABR or CBR setting in the 200+ kbps range, for instance -b224 -h.
With such a setting music usually is transparent, and in those rare cases when it's not it's at least acceptable.
Title: Personal evaluation at ~130..135 kbps, 200 samples
Post by: shadowking on 2007-05-30 09:26:20
mp3 problems samples aren't slight changes at all. The worst cases make me physically ill the more I listen to them and there is no need to abx even on -v2.