Skip to main content

Topic: 128kbps Extension Test - FINISHED (Read 60852 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • rjamorim
  • [*][*][*][*][*]
128kbps Extension Test - FINISHED
Hello

I would like to announce that the 128kbps Extension listening test is now closed, and the results are available at my Listening Tests page:

http://audio.ciara.us/test/

Here are the overall ratings:


Please post comments at this thread.

Best regards;

Roberto.
  • Last Edit: 04 August, 2003, 01:14:50 AM by rjamorim
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
128kbps Extension Test - FINISHED
Reply #1
Nice test.  The results clearly show how good the newer generation of codecs are.  Looks like Blade served its purpose well.  I wonder if there are still some Blade 128's out there on p2p?  I wonder how Fraunhofer's FastEnc or the older "Radium" hack would have compared to Lame?

Death2.wav is really interesting.  The bitrates for Ogg and MPC don't suggest anything out of the ordinary (about 115 kbit/s), but WMA9Pro really tanks.  A lot of people comment on noisy transients, with something also wrong with the stereo during those parts.

Several people have very good high frequency hearing, and can hear the lowpassing of MPC and AAC (both around 16 kHz).  One person (gecko) seems to find this to be a significant defect.

If new samples are chosen for the next test, I think Waiting.wav should be retained.  It seems to be one of those killer samples which have a big effect on every codec.

ff123

  • treech
  • [*][*][*]
128kbps Extension Test - FINISHED
Reply #2
SWEET!

this will help me decide on what portable (player) to get.

looks like aac and ogg are very good indeed at this bitrate, really good news for portables, since those 2 are most likely to be the ones to get implemented first, (i'de love to get the iriver perl, ogg support)

and the ipod already had aac support iirc ...

128kbps Extension Test - FINISHED
Reply #3
VERY interesting that Musepack came out on top, albeit marginally. Thanks to Roberto, all the test takers and everyone who helped. These comparisons are very useful.  Keep up the good work!
superdumprob
____________________________________________

"If we knew what it was we were doing, it would not be called research, would it?" - Albert Einstein

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - FINISHED
Reply #4
I did two different tests : one on a portable PC, with a (I suppose) crap audio chip (Conexant AC) but good headphone ; another with my main soundcard, Terratec DMX6Fire.

Results are here :

http://membres.lycos.fr/guruboolez/AUDIO/t...t_MF/COMPAQ.htm
http://membres.lycos.fr/guruboolez/AUDIO/t...MF/TERRATEC.htm

Comments (log file), (in anglische) are available too :
http://membres.lycos.fr/guruboolez/AUDIO/test_MF/


I've two different winners :
WMA9PRO on portable, and Quicktime AAC on Terratec (never mpc).
The three best are close each others ; Vorbis was fourth, on two test, not as good as the three others, but more constant than WMA9PRo, and maybe MPC.
LAME is now old, and Blade looks more as a joke than as an encoder.

I'm a bit surprised to see mpc "wining" here. I hope that Frank Klemm is just in holydays : I didn't see him for some month on HA. The development of mpc can't stop !

EDIT : Roberto, thank you another time for publishing these results immediately after then end of the test 
  • Last Edit: 04 August, 2003, 05:33:36 AM by guruboolez

  • spoon
  • [*][*][*][*][*]
  • Administrator
128kbps Extension Test - FINISHED
Reply #5
I would be interested in doing a little statistics based on bitrate, if someone who has access to the samples please post either the total file sizes for each encoder, or the average bitrate (ie 134kbps + 150Kbps...) for each of the codecs.

  • Volcano
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - FINISHED
Reply #6
Well done, Roberto. B)

I too was surprised at how well the codecs performed generally, some tests were *extremely* hard. (However I have to say that in some cases where added background noise was the only problem, I had absolutely no chance of detecting it because my sh*tty equipment hisses like mad. Gotta get a better soundcard soon. )

I'm beginning to lose hope for Vorbis. It gets beaten by MP4 in most cases, and generally performs worse than one would expect, given the fact that this type of bitrate range is considered Vorbis' speciality. Add to that the sluggish development (also with regards to hardware support), and its future doesn't look too bright...


On a side note, @ff123: Have you considered perhaps adding XML support for the results file to ABC/HR? That would make parsing the results into a database a breeze, and they could be published on one HTML page much more easily.

  • spoon
  • [*][*][*][*][*]
  • Administrator
128kbps Extension Test - FINISHED
Reply #7
I wouldn't loose hope  it just shows how competitive things are in that bitrate area, really there is not much between AAC, Ogg, WMA and MPC. Big bucks have gone into developing some of those codecs and it is good that ogg achieves the same without patent infringing (if WMA wanted to use some technology as AAC, they would just cross license patents, that was not an option for Ogg).

I think the 'winners' in years to come will be AAC and WMA, both of these will have large online commercial music shops and wide support on portable players.

  • Lev
  • [*][*][*][*][*]
128kbps Extension Test - FINISHED
Reply #8
Quote
The website you have attempted to access is on a filter list maintained
for Unilever. In an effort to avoid potential embarrassment to you or
subsequent users of this PC, this material will not be displayed.

However the list of filter sites can sometimes be inaccurate or out of
date. If you believe this is the case please call the GIO-ES service desk
on +44 (0) 1244 50 4180 and access to this site will be restored for you
and all other users in Unilever.

The URL which you have attempted to access is listed below. You will be
asked for this information when you call the service desk.

GIO apologises for any inconvenience this may have caused.



--------------------------------------------------------------------------------

URL = http://membres.lycos.fr/guruboolez/AUDIO/t...t_MF/COMPAQ.htm 

heh 

It seems that you are sensitive to MPC artifacts, Guru.  rjamorim wants your ears, I am very happy not to have them 

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - FINISHED
Reply #9
Lev > Is this what appeared when you tried to reach the two results matrix I uploaded ? It works with me... Can someone confirm ?

P.S. I created the webpage with MS Word XP (I haven't anything else installed, and I'm bored by notepad ) : It's big, and maybe problematic.
  • Last Edit: 04 August, 2003, 06:35:24 AM by guruboolez

  • spoon
  • [*][*][*][*][*]
  • Administrator
128kbps Extension Test - FINISHED
Reply #10
Works for me, the filtering is through his company proxy server (unilever).

  • XXX
  • [*]
  • Banned
128kbps Extension Test - FINISHED
Reply #11
For the record, if you have six players and five tie for first place, the sixth player is in sixth (or last) place, not second place.  If you have six players, and two tie for second place, there is no third place; the next place is fourth place.  Win.  Place.  Or Show.  The rest don't really matter, you know.

  • Mac
  • [*][*][*][*][*]
128kbps Extension Test - FINISHED
Reply #12
Taking into account the deviations possible in the results, I see it fair to tie AAC, MPC, OGG & WMA in joint 1st..  [span style='font-size:8pt;line-height:100%'](MPC could be the lowest of the four and OGG the highest if the errors fell in a parculiar way)[/span]

At first I thought it is surprising that 4 very different codecs can achieve 4 levels of quality that converge so closely..  but then I thought, does this just mean they are approaching the soft-limit of achievable compression?  Until radical re-thinks in compression schemes (SBR @ 128k, wavelets) I wouldn't imagine a codec *could* progress much further than this ~4.5 limit.

Thankyou for organizing this test Roberto, these results provide the interesting food for thought that can only come about from a professionally conducted group test
< w o g o n e . c o m / l o l >

  • Lev
  • [*][*][*][*][*]
128kbps Extension Test - FINISHED
Reply #13
Yep - its a Unilever filter... you didnt include 'bondage' or 'granny fisting' in the title  bar, did you?   

I agree with Mac - its almost like a plateau has been reached, and asking people to judge encoders is almost like asking people if they like celery.  Guruboolez seems sensitive to MPC artifacts, whereas many of us aren't, hence it slightly won in the test.  Its almost like a genetic thing, rather than a matter of training or practice.. I have a self proclaimed talent for picking out Oggm for example.

But yep, I love test results like this.  I love figures as opposed to words.  Thanks 

  • phong
  • [*][*][*][*]
128kbps Extension Test - FINISHED
Reply #14
I think one certain conclusion we can make is that anyone selling 128k anything and calling it "CD quality" needs to be thrown in jail. 

A couple notes (based on my listening):

For me, each codec had at least one sample with a pretty catastrophic falure that I would probably notice in casual listening.  MPC came the closest to being "acceptable" to me for all samples, but had a quite noticeable stereo separation issue with TheSource.  Nobody else seemed to mention that problem though...

Blade did serve its purpose, however, I had one sample where lame lost to blade.  I one other person ranked them in that order for that sample (Waiting), and another had them tied, so I might not be completely crazy.  Also on death2 if it weren't for blade for comparision, WMApro would have gotten < 2 and AAC would have been scored quite a bit lower too.

I was able to ABX a lot more of these than I initially expected to.  My ears are not all that trained (well I suppose they're getting there).

I would have ranked them (based on my results before I saw everyone elses):
1. MPC
2. WMApro
3. AAC and vorbis tied (very close, throw out one test and they'd flip-flop)
5. lame (fairly far back)
6 . blade (way way back).

For my results, MPC was the most consistant, almost always either getting first, or very close (with the one exception.)  WMApro was ususally near the top, with one pretty big failure, and somes falling back to the middle.  AAC consistantly came in around third, close to MPC and WMApro, while vorbis tended to jump around more.  Lame usually trailed but occasionally came out near the top, and blade was always way back.  For me MPC was definately "least likely to have serious problems" which is pretty consistant with the consensus of these forums.

My ears got trained as I went along...  Problems that were hardly noticeable initially became really easy (specifically the "brightness" of vorbis on some samples, and the absolute crappiness of blade on all samples).  Other things got harder from having listened to them too much.    I think I would have fairly different results if I were to do another test (WMApro, AAC and esp. vorbis would probably get lower scores now that I am more familiar with their problems).

I thought it was interesting how much people varied in their sensitivity to different problems.  Even though I knew harpsichords were supposed to cause trouble, I really couldn't identify too many with Bachpsichord (and I did try very hard).  OTOH, I'm the only one (I think) that mentioned the stereo problems with TheSource and MPC.

I'd have to say I was "heartened" by the the final results.  While I had MPC as a winner, I don't ever see it having success in the portable market or widespread support.  After looking at my results, my sentimental favorite (vorbis) didn't look like it could stand up against against WMApro, nor could AAC (which I reguard as "less evil" than WMApro).  To see that there's pretty much a four way tie for first is a bit surprising.  If vorbis gets further tuning, I think it could stay competitive.

Who gets the golden ear award?  Guruboolez?  Gecko?  I suppose handing out awards isn't a good idea.  It might encourage people to fake results.  Plus from my comments, I think I'd get the "excessive verbosity" award.

I want to say thanks one more time to rajamorim (espescially for the speedy tabulation) and to all the other participants.
  • Last Edit: 04 August, 2003, 09:01:01 AM by phong
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - FINISHED
Reply #15
Here is the ranking table for this test (based on overall results) :

http://membres.lycos.fr/guruboolez/AUDIO/t.../test_MF/HA.htm

Mpc never goes below the third place (only codec to perform that). Must be the most constant encoder (thanks to VBR I suppose) of the five challengers (with lame, always at the two last places).

Vorbis can't seriously claim one of the first three place.

Lame is outdated (Lev, you could call that 'granny fisting' )
  • Last Edit: 04 August, 2003, 09:44:06 AM by guruboolez

  • S_O
  • [*][*][*][*]
128kbps Extension Test - FINISHED
Reply #16
Wasn´t the test open to 5th august? I´ve done test 1 - 10 already, with comments etc., just hadn´t enough time to do the last both test.
Now I´ve done the ranking of myself. also the last both samples are not included my result is a bit different:

1. Vorbis: 4,7
2. QT AAC: 4,66
3. Musepack: 4,38
4. Lame: 4,01
5. WMA Pro: 3,84
6. Blade: 1,0

I always rated the blade sample to 1, because it´s clearly the worst, also on some samples Blade performs better. The results are not manipulated and didn´t thought that would come out (I thought AAC clearly beats Vorbis and Musepack as higher-bitrate codec is the worst after blade). here are results for sample 1-10:
Code: [Select]
Vorbis    5   - 5   - 4,5 - 4,5 - 5   - 5   - 4   - 4,5 - 4,5 - 5 -- 4,7
Lame      4,5 - 4,8 - 4   - 5   - 2,5 - 3,8 - 4,2 - 3,8 - 2,5 - 5 -- 4,01
MusePack  4,6 - 4,9 - 4,2 - 3,8 - 5   - 5   - 3,8 - 4,5 - 4   - 4 -- 4,38
WMA Pro   5   - 3,2 - 5   - 4,2 - 1,5 - 3,5 - 4   - 4,5 - 4,5 - 3 -- 3,84
QT AAC    4,4 - 4,5 - 5   - 5   - 4,8 - 4,2 - 4,4 - 4,8 - 4,5 - 5 -- 4,66

If you like I can also post all my comments to the samples.

  • smok3
  • [*][*][*][*][*]
  • Moderator
128kbps Extension Test - FINISHED
Reply #17
Quote
I would be interested in doing a little statistics based on bitrate, if someone who has access to the samples please post either the total file sizes for each encoder, or the average bitrate (ie 134kbps + 150Kbps...) for each of the codecs.

that would be interesting to see.
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

  • fragtal
  • [*][*][*][*]
128kbps Extension Test - FINISHED
Reply #18
Quote
1. Vorbis: 4,7
2. QT AAC: 4,66
3. Musepack: 4,38
4. Lame: 4,01
5. WMA Pro: 3,84
6. Blade: 1,0

That's the ranking I've expected.

I hope this test will encourage Xiph.org to do some further Vorbis tuning.
I love the moderators.

  • Jojo
  • [*][*][*][*][*]
128kbps Extension Test - FINISHED
Reply #19
I would like to know which options have been used for the wma 9 pro samples. I mean whether vbr or cbr.
Also, where can I download wma 9 Pro? Or do I have to pay for it?
  • Last Edit: 04 August, 2003, 10:54:57 AM by Jojo
--alt-presets are there for a reason! These other switches DO NOT work better than it, trust me on this.
LAME + Joint Stereo doesn't destroy 'Stereo'

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - FINISHED
Reply #20
Quote
I would like to know which options have been used for the wma 9 pro samples. I mean whether vbr or cbr.
Also, where can I download wma 9 Pro? Or do I have to pay for it?

Just read Roberto's presentation  :

wma9pro VBR 128 two pass (it's more ABR, and very accurate : 128 kbps)

WMA9PRO is available on Microsoft website. Just DL for free Windows Media Encoder 9. If you're running on 98SE OS, take a look at Spoon's dBPowerAMP. It's an easy way to encode with WMA9  (pro, lossless, voice...)

  • rjamorim
  • [*][*][*][*][*]
128kbps Extension Test - FINISHED
Reply #21
Quote
I would be interested in doing a little statistics based on bitrate, if someone who has access to the samples please post either the total file sizes for each encoder, or the average bitrate (ie 134kbps + 150Kbps...) for each of the codecs.

I will add a table with all the bitrates to the results page later today.

Quote
Wasn´t the test open to 5th august?


Oh, sorry, S_O, but quoting the readme:
"7. After you finish the test, save the results and mail them to
  rjamorim@<blanked>. The test ends on August 3rd, 2003. No
  results will be accepted after that date."





Some info for you guys:

Next test will be at 64kbps. I'll compare Vorbis, HE AAC, WMAv9, MP3Pro and Real Audio Cook. But that's discusseable, and I don't want to discuss it right now. I gotta take a break...

Hopefully, people won't have to struggle that much to find artifacts in this test.

I plan to start this test by the beginning of September, probably the 3rd.

People are too fed up with the current samples, so I think I'll only keep Waiting (per ff123's suggestion) and do a call for samples next week.

No decision has been taken yet though, and I sense there'll be a very heaten discussion about whether choosing VBR whenever possible or only CBR. I'll start a pre-test thread as the test gets closer.

Thank-you very much.

Best regards;

Roberto.
  • Last Edit: 04 August, 2003, 01:10:42 PM by rjamorim
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • music_man_mpc
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - FINISHED
Reply #22
These results are very supprizing, even on MusePack.net it says that Ogg is probably better for bitrates <140Kbit.

Quote
Sound tests with other Audioformats show that MPC is the MPC for high quality with bitrates above 140kbits. Below OGG Vorbis is a bit better.
  • Last Edit: 04 August, 2003, 01:19:35 PM by music_man_mpc
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame

  • Gecko
  • [*][*][*][*][*]
128kbps Extension Test - FINISHED
Reply #23
I find it very interesting that aac being cbr and wma pro being two passed reach such a high quality. I've been especially surprised by WMA Pro - it's predecessors being so crappy (allthough apparently it could use some more tuning if we look at the death2 slipup).

Looking at the results, I am also glad mpc was included in the test since it is a serious contender.

I noticed stereo collapse on FloorEssence with mpc (but not on TheSource).

Thanks Roberto, for performing the test and such a quick evaluation!

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
128kbps Extension Test - FINISHED
Reply #24
Quote
I've been especially surprised by WMA Pro - it's predecessors being so crappy (allthough apparently it could use some more tuning if we look at the death2 slipup).

I'm surprised too. The WM9pro encoder used for this test is the first never released. I never heard a baby singing so good for its birth.

Quality is amazing for half samples of this test, which is simply remarkable (first place, with sometimes great notation !). More tuning is needed - death5 is totally crap - the biggest flaw in my opinion on 12 samples, even when including lame.

Last but not least, wma9pro is gapless... AAC/MP4 isn't for the moment.