Hello
I would like to announce that the 128kbps Extension listening test is now closed, and the results are available at my Listening Tests page:
http://audio.ciara.us/test/ (http://audio.ciara.us/test/)
Here are the overall ratings:
(http://audio.ciara.us/test/128extension/plot12.png)
Please post comments at this thread.
Best regards;
Roberto.
Nice test. The results clearly show how good the newer generation of codecs are. Looks like Blade served its purpose well. I wonder if there are still some Blade 128's out there on p2p? I wonder how Fraunhofer's FastEnc or the older "Radium" hack would have compared to Lame?
Death2.wav is really interesting. The bitrates for Ogg and MPC don't suggest anything out of the ordinary (about 115 kbit/s), but WMA9Pro really tanks. A lot of people comment on noisy transients, with something also wrong with the stereo during those parts.
Several people have very good high frequency hearing, and can hear the lowpassing of MPC and AAC (both around 16 kHz). One person (gecko) seems to find this to be a significant defect.
If new samples are chosen for the next test, I think Waiting.wav should be retained. It seems to be one of those killer samples which have a big effect on every codec.
ff123
SWEET!
this will help me decide on what portable (player) to get.
looks like aac and ogg are very good indeed at this bitrate, really good news for portables, since those 2 are most likely to be the ones to get implemented first, (i'de love to get the iriver perl, ogg support)
and the ipod already had aac support iirc ...
VERY interesting that Musepack came out on top, albeit marginally. Thanks to Roberto, all the test takers and everyone who helped. These comparisons are very useful. Keep up the good work!
I did two different tests : one on a portable PC, with a (I suppose) crap audio chip (Conexant AC) but good headphone ; another with my main soundcard, Terratec DMX6Fire.
Results are here :
http://membres.lycos.fr/guruboolez/AUDIO/t...t_MF/COMPAQ.htm (http://membres.lycos.fr/guruboolez/AUDIO/test_MF/COMPAQ.htm)
http://membres.lycos.fr/guruboolez/AUDIO/t...MF/TERRATEC.htm (http://membres.lycos.fr/guruboolez/AUDIO/test_MF/TERRATEC.htm)
Comments (log file), (in anglische) are available too :
http://membres.lycos.fr/guruboolez/AUDIO/test_MF/ (http://membres.lycos.fr/guruboolez/AUDIO/test_MF/)
I've two different winners :
WMA9PRO on portable, and Quicktime AAC on Terratec (never mpc).
The three best are close each others ; Vorbis was fourth, on two test, not as good as the three others, but more constant than WMA9PRo, and maybe MPC.
LAME is now old, and Blade looks more as a joke than as an encoder.
I'm a bit surprised to see mpc "wining" here. I hope that Frank Klemm is just in holydays : I didn't see him for some month on HA. The development of mpc can't stop !
EDIT : Roberto, thank you another time for publishing these results immediately after then end of the test
I would be interested in doing a little statistics based on bitrate, if someone who has access to the samples please post either the total file sizes for each encoder, or the average bitrate (ie 134kbps + 150Kbps...) for each of the codecs.
Well done, Roberto. B)
I too was surprised at how well the codecs performed generally, some tests were *extremely* hard. (However I have to say that in some cases where added background noise was the only problem, I had absolutely no chance of detecting it because my sh*tty equipment hisses like mad. Gotta get a better soundcard soon. )
I'm beginning to lose hope for Vorbis. It gets beaten by MP4 in most cases, and generally performs worse than one would expect, given the fact that this type of bitrate range is considered Vorbis' speciality. Add to that the sluggish development (also with regards to hardware support), and its future doesn't look too bright...
On a side note, @ff123: Have you considered perhaps adding XML support for the results file to ABC/HR? That would make parsing the results into a database a breeze, and they could be published on one HTML page much more easily.
I wouldn't loose hope it just shows how competitive things are in that bitrate area, really there is not much between AAC, Ogg, WMA and MPC. Big bucks have gone into developing some of those codecs and it is good that ogg achieves the same without patent infringing (if WMA wanted to use some technology as AAC, they would just cross license patents, that was not an option for Ogg).
I think the 'winners' in years to come will be AAC and WMA, both of these will have large online commercial music shops and wide support on portable players.
The website you have attempted to access is on a filter list maintained
for Unilever. In an effort to avoid potential embarrassment to you or
subsequent users of this PC, this material will not be displayed.
However the list of filter sites can sometimes be inaccurate or out of
date. If you believe this is the case please call the GIO-ES service desk
on +44 (0) 1244 50 4180 and access to this site will be restored for you
and all other users in Unilever.
The URL which you have attempted to access is listed below. You will be
asked for this information when you call the service desk.
GIO apologises for any inconvenience this may have caused.
--------------------------------------------------------------------------------
URL = http://membres.lycos.fr/guruboolez/AUDIO/t...t_MF/COMPAQ.htm (http://membres.lycos.fr/guruboolez/AUDIO/test_MF/COMPAQ.htm)
heh
It seems that you are sensitive to MPC artifacts, Guru. rjamorim wants your ears, I am very happy not to have them
Lev > Is this what appeared when you tried to reach the two results matrix I uploaded ? It works with me... Can someone confirm ?
P.S. I created the webpage with MS Word XP (I haven't anything else installed, and I'm bored by notepad ) : It's big, and maybe problematic.
Works for me, the filtering is through his company proxy server (unilever).
For the record, if you have six players and five tie for first place, the sixth player is in sixth (or last) place, not second place. If you have six players, and two tie for second place, there is no third place; the next place is fourth place. Win. Place. Or Show. The rest don't really matter, you know.
Taking into account the deviations possible in the results, I see it fair to tie AAC, MPC, OGG & WMA in joint 1st.. [span style='font-size:8pt;line-height:100%'](MPC could be the lowest of the four and OGG the highest if the errors fell in a parculiar way)[/span]
At first I thought it is surprising that 4 very different codecs can achieve 4 levels of quality that converge so closely.. but then I thought, does this just mean they are approaching the soft-limit of achievable compression? Until radical re-thinks in compression schemes (SBR @ 128k, wavelets) I wouldn't imagine a codec *could* progress much further than this ~4.5 limit.
Thankyou for organizing this test Roberto, these results provide the interesting food for thought that can only come about from a professionally conducted group test
Yep - its a Unilever filter... you didnt include 'bondage' or 'granny fisting' in the title bar, did you?
I agree with Mac - its almost like a plateau has been reached, and asking people to judge encoders is almost like asking people if they like celery. Guruboolez seems sensitive to MPC artifacts, whereas many of us aren't, hence it slightly won in the test. Its almost like a genetic thing, rather than a matter of training or practice.. I have a self proclaimed talent for picking out Oggm for example.
But yep, I love test results like this. I love figures as opposed to words. Thanks
I think one certain conclusion we can make is that anyone selling 128k anything and calling it "CD quality" needs to be thrown in jail.
A couple notes (based on my listening):
For me, each codec had at least one sample with a pretty catastrophic falure that I would probably notice in casual listening. MPC came the closest to being "acceptable" to me for all samples, but had a quite noticeable stereo separation issue with TheSource. Nobody else seemed to mention that problem though...
Blade did serve its purpose, however, I had one sample where lame lost to blade. I one other person ranked them in that order for that sample (Waiting), and another had them tied, so I might not be completely crazy. Also on death2 if it weren't for blade for comparision, WMApro would have gotten < 2 and AAC would have been scored quite a bit lower too.
I was able to ABX a lot more of these than I initially expected to. My ears are not all that trained (well I suppose they're getting there).
I would have ranked them (based on my results before I saw everyone elses):
1. MPC
2. WMApro
3. AAC and vorbis tied (very close, throw out one test and they'd flip-flop)
5. lame (fairly far back)
6 . blade (way way back).
For my results, MPC was the most consistant, almost always either getting first, or very close (with the one exception.) WMApro was ususally near the top, with one pretty big failure, and somes falling back to the middle. AAC consistantly came in around third, close to MPC and WMApro, while vorbis tended to jump around more. Lame usually trailed but occasionally came out near the top, and blade was always way back. For me MPC was definately "least likely to have serious problems" which is pretty consistant with the consensus of these forums.
My ears got trained as I went along... Problems that were hardly noticeable initially became really easy (specifically the "brightness" of vorbis on some samples, and the absolute crappiness of blade on all samples). Other things got harder from having listened to them too much. I think I would have fairly different results if I were to do another test (WMApro, AAC and esp. vorbis would probably get lower scores now that I am more familiar with their problems).
I thought it was interesting how much people varied in their sensitivity to different problems. Even though I knew harpsichords were supposed to cause trouble, I really couldn't identify too many with Bachpsichord (and I did try very hard). OTOH, I'm the only one (I think) that mentioned the stereo problems with TheSource and MPC.
I'd have to say I was "heartened" by the the final results. While I had MPC as a winner, I don't ever see it having success in the portable market or widespread support. After looking at my results, my sentimental favorite (vorbis) didn't look like it could stand up against against WMApro, nor could AAC (which I reguard as "less evil" than WMApro). To see that there's pretty much a four way tie for first is a bit surprising. If vorbis gets further tuning, I think it could stay competitive.
Who gets the golden ear award? Guruboolez? Gecko? I suppose handing out awards isn't a good idea. It might encourage people to fake results. Plus from my comments, I think I'd get the "excessive verbosity" award.
I want to say thanks one more time to rajamorim (espescially for the speedy tabulation) and to all the other participants.
Here is the ranking table for this test (based on overall results) :
http://membres.lycos.fr/guruboolez/AUDIO/t.../test_MF/HA.htm (http://membres.lycos.fr/guruboolez/AUDIO/test_MF/HA.htm)
Mpc never goes below the third place (only codec to perform that). Must be the most constant encoder (thanks to VBR I suppose) of the five challengers (with lame, always at the two last places).
Vorbis can't seriously claim one of the first three place.
Lame is outdated (Lev, you could call that 'granny fisting' )
Wasn´t the test open to 5th august? I´ve done test 1 - 10 already, with comments etc., just hadn´t enough time to do the last both test.
Now I´ve done the ranking of myself. also the last both samples are not included my result is a bit different:
1. Vorbis: 4,7
2. QT AAC: 4,66
3. Musepack: 4,38
4. Lame: 4,01
5. WMA Pro: 3,84
6. Blade: 1,0
I always rated the blade sample to 1, because it´s clearly the worst, also on some samples Blade performs better. The results are not manipulated and didn´t thought that would come out (I thought AAC clearly beats Vorbis and Musepack as higher-bitrate codec is the worst after blade). here are results for sample 1-10:
Vorbis 5 - 5 - 4,5 - 4,5 - 5 - 5 - 4 - 4,5 - 4,5 - 5 -- 4,7
Lame 4,5 - 4,8 - 4 - 5 - 2,5 - 3,8 - 4,2 - 3,8 - 2,5 - 5 -- 4,01
MusePack 4,6 - 4,9 - 4,2 - 3,8 - 5 - 5 - 3,8 - 4,5 - 4 - 4 -- 4,38
WMA Pro 5 - 3,2 - 5 - 4,2 - 1,5 - 3,5 - 4 - 4,5 - 4,5 - 3 -- 3,84
QT AAC 4,4 - 4,5 - 5 - 5 - 4,8 - 4,2 - 4,4 - 4,8 - 4,5 - 5 -- 4,66
If you like I can also post all my comments to the samples.
I would be interested in doing a little statistics based on bitrate, if someone who has access to the samples please post either the total file sizes for each encoder, or the average bitrate (ie 134kbps + 150Kbps...) for each of the codecs.
that would be interesting to see.
1. Vorbis: 4,7
2. QT AAC: 4,66
3. Musepack: 4,38
4. Lame: 4,01
5. WMA Pro: 3,84
6. Blade: 1,0
That's the ranking I've expected.
I hope this test will encourage Xiph.org to do some further Vorbis tuning.
I would like to know which options have been used for the wma 9 pro samples. I mean whether vbr or cbr.
Also, where can I download wma 9 Pro? Or do I have to pay for it?
I would like to know which options have been used for the wma 9 pro samples. I mean whether vbr or cbr.
Also, where can I download wma 9 Pro? Or do I have to pay for it?
Just read Roberto's presentation :
wma9pro VBR 128 two pass (it's more ABR, and very accurate : 128 kbps)
WMA9PRO is available on Microsoft website. Just DL for free Windows Media Encoder 9. If you're running on 98SE OS, take a look at Spoon's dBPowerAMP. It's an easy way to encode with WMA9 (pro, lossless, voice...)
I would be interested in doing a little statistics based on bitrate, if someone who has access to the samples please post either the total file sizes for each encoder, or the average bitrate (ie 134kbps + 150Kbps...) for each of the codecs.
I will add a table with all the bitrates to the results page later today.
Wasn´t the test open to 5th august?
Oh, sorry, S_O, but quoting the readme:
"7. After you finish the test, save the results and mail them to
rjamorim@<blanked>. The test ends on August 3rd, 2003. No
results will be accepted after that date."
Some info for you guys:
Next test will be at 64kbps. I'll compare Vorbis, HE AAC, WMAv9, MP3Pro and Real Audio Cook. But that's discusseable, and I don't want to discuss it right now. I gotta take a break...
Hopefully, people won't have to struggle that much to find artifacts in this test.
I plan to start this test by the beginning of September, probably the 3rd.
People are too fed up with the current samples, so I think I'll only keep Waiting (per ff123's suggestion) and do a call for samples next week.
No decision has been taken yet though, and I sense there'll be a very heaten discussion about whether choosing VBR whenever possible or only CBR. I'll start a pre-test thread as the test gets closer.
Thank-you very much.
Best regards;
Roberto.
These results are very supprizing, even on MusePack.net it says that Ogg is probably better for bitrates <140Kbit.
Sound tests with other Audioformats show that MPC is the MPC for high quality with bitrates above 140kbits. Below OGG Vorbis is a bit better.
I find it very interesting that aac being cbr and wma pro being two passed reach such a high quality. I've been especially surprised by WMA Pro - it's predecessors being so crappy (allthough apparently it could use some more tuning if we look at the death2 slipup).
Looking at the results, I am also glad mpc was included in the test since it is a serious contender.
I noticed stereo collapse on FloorEssence with mpc (but not on TheSource).
Thanks Roberto, for performing the test and such a quick evaluation! (http://community.the-underdogs.org/smiley/misc/bow.gif)
I've been especially surprised by WMA Pro - it's predecessors being so crappy (allthough apparently it could use some more tuning if we look at the death2 slipup).
I'm surprised too. The WM9pro encoder used for this test is the first never released. I never heard a baby singing so good for its birth.
Quality is amazing for half samples of this test, which is simply remarkable (first place, with sometimes great notation !). More tuning is needed - death5 is totally crap - the biggest flaw in my opinion on 12 samples, even when including lame.
Last but not least, wma9pro is gapless... AAC/MP4 isn't for the moment.
Iam Happy that my Favorite Codec WMA show the rest that he is better and not so Crappy like Everybody said.
But i have One little Problem a wma9pro Encoded File dont be played in Winamp 2.95.
Is there a plugin that wma9pro played without the Windows Media Player.
ThX
Iam Happy that my Favorite Codec WMA show the rest that he is better and not so Crappy like Everybody said.
But i have One little Problem a wma9pro Encoded File dont be played in Winamp 2.95.
Is there a plugin that wma9pro played without the Windows Media Player.
ThX
Unfortunately, that's the "one little problem" with using wma9pro right now. Although it sounds better than the older wma format, it breaks compatibility with current decoders. I'm sure that will change in due time.
ff123
Iam Happy that my Favorite Codec WMA show the rest that he is better and not so Crappy like Everybody said.
But i have One little Problem a wma9pro Encoded File dont be played in Winamp 2.95.
Is there a plugin that wma9pro played without the Windows Media Player.
ThX
WMA9pro is a good surprise. But standard WMA9 is really worse.
There's no plug for winamp. Foobar2000 can play WMA9 PRO-LOSSLESS... without problem. Just install Special Release offered by Case, on :
www.ca5e.tk
www.musepack.fr.st
Thx i test foobar.
But i Hope a in_put Plugin for winamp will follow.
Call me what you want but foobar is not my Favorite Player,Sorry but i respect the work on it.
Ciao
I noticed stereo collapse on FloorEssence with mpc (but not on TheSource).
I've got to go back and listen to that again when I get home. I thought it was really obvious (present to a lesser degree with lame too), and my ears were barely capable of noticing stereo collapse in FloorEssence or any other samples. I ABXed 10/11 (and the miss was while fishing for other defects) so I don't think I imagined it.
Can one of you sharp-eared folks check it also to help me verify that I'm not insane?
I noticed stereo collapse on FloorEssence with mpc (but not on TheSource).
I've got to go back and listen to that again when I get home. I thought it was really obvious (present to a lesser degree with lame too), and my ears were barely capable of noticing stereo collapse in FloorEssence or any other samples. I ABXed 10/11 (and the miss was while fishing for other defects) so I don't think I imagined it.
Can one of you sharp-eared folks check it also to help me verify that I'm not insane?
I only noticed stereo collapse in one small section of one sample, and it was with Ogg.
BTW, I like your quotes, phong. Here's my favorite:
"It sounds like an leprechaun is making love to the whammy bar."
ff123
BTW, I like your quotes, phong. Here's my favorite:
"It sounds like an leprechaun is making love to the whammy bar."
Well, after listening to a few of the samples, I noticed I started getting much better results in hearing differences and ABXing by using vivid imagery. I'd listen to the original, and try to picture it in my head, then listen to the encoded version, and create an image for that. Then the "X" would automatically bring up the appropriate image. Some of the images made their way into the comments; on others, I thought better of it.
But i have One little Problem a wma9pro Encoded File dont be played in Winamp 2.95.
Is there a plugin that wma9pro played without the Windows Media Player.
delete in_wm.dll and add wma to the extension list in in_dshow's configuration
imho vorbis definitely lost the test
and i will use mp4/aac in the future
(btw. cant wait anyomore for the 64kbps test )
Man, LAME really is dated at this bitrate. Hopefully no one misinterprets this test and decides from it that all these codecs perform equally the same at higher bitrates. Please remember that this is only at 128kbps. alt-preset standard, vorbis gt3, musepack standard, etc will be tested later, if that would even be a valid or necessary test. Taking from the difficulty that this test has shown, it would be no wonder even harder to test the higher bitrates.
Even though I know that it is incorrect to make blanket statements, I'm sure that because of this test we will see them. If it wasn't for compatibility, I think you would see people leaving LAME in droves now. While I guess it might be plausible to make judgements from this data that LAME is dated, that other codecs have seen greater development as of late, and then as such decide to use a different codec for all your encoding needs, I don't think it would be a good idea to directly link this test to overall sound quality.
Oh bollocks, /. sucks:
2003-08-04 19:06:04 128kbps audio codec listening test (articles,music) (rejected)
alt-preset standard, vorbis gt3, musepack standard, etc will be tested later
Will they? When? and by who?
Surely not by me. First, because these bitrates are transparent, we would get most of the codecs reaching score 5. Second, because I am only willing to take one other VBR test. The critics on a VBR test are such a PITA that I definitely don't even consider conducing several ones.
Second, because I am only willing to take one other VBR test. The critics on a VBR test are such a PITA that I definitely don't even consider conducing several ones.
Too bad to read this negative impact, consequence of the recent and uniform critiscism posted the last week. Just ignore them : you did a very nice job ! Some myths are now dead, and without your test, they'll probably stay alive.
EDIT, after reading the answer below : my apologies for people concerned by my unfounded message.
Oh bollocks, /. sucks:
2003-08-04 19:06:04 128kbps audio codec listening test (articles,music) (rejected)
Dude, to tell you the truth, I keep wondering if it's not better to keep these results out of /. :-/
I can only imagine the flood of unfounded criticism about bitrates, samples, methodology, and that I should have included Flac in the test (yeah, I read a comment like that in the AAC test article!)
@Guruboolez: No, the problem wasn't that thread. At least, people were (trying to) make constructive criticism. (although I'm kinda sad because they didn't bother to do this welcome criticism when the time was right, I.E, the pre-test thread)
What really made me sick was people saying my test resoults would be worthless in my e-mail. I got to a point that I didn't even read them properly, just hit the delete button.
Several people have very good high frequency hearing, and can hear the lowpassing of MPC and AAC (both around 16 kHz). One person (gecko) seems to find this to be a significant defect.
Hm, at least how I remember the test, the lowpass was usually one of the most significant things to notice on the aac/mpc samples. I consider this a good thing actually, since lowpassing can be easily fixed without further issues (save higher bitrate) in mpc (and aac?) and you have almost no other artifacts to begin with.
Looking back at my results, I didn't notice the lowpassing very often (only on 4 samples) and I just assume right now that the low rating wasn't a results of the lowpassing but other defects. I was in a hurry and didn't bother to do extensive comments so I just noted down some things that were particularly interesting imo. You could look at it this way that I didn't notice significant lowpassing on all the other samples (other than I would have expected).
imho vorbis definitely lost the test
Do you think? I don't see a clear winner/loser between MPC/WMA/AAC/Vorbis. The results are very closed.
Roberto, you've done a good job and that's what really matters. Congratulations.
imho vorbis definitely lost the test
Do you think? I don't see a clear winner/loser between MPC/WMA/AAC/Vorbis. The results are very closed.
Roberto, you've done a good job and that's what really matters. Congratulations.
I don't want to be considered as an anti-vorbis zealot, and I'm not specialist in statistic, but :
http://membres.lycos.fr/guruboolez/AUDIO/t.../test_MF/HA.htm (http://membres.lycos.fr/guruboolez/AUDIO/test_MF/HA.htm)
Vorbis took the two first places
one time (on 12), and the two lasts
7 times. It's far the worst performance, compared to mpc, aac and wmapro (can't include seriously any mp3 encoder as pretender of the throne).
According to my own results, vorbis isn't exactly at the same level as mpc, mp4 and wmapro. Not too far, of course, but not as transparent as its challenger.
For me personally Ogg Vorbis came out on top. I voted it best on 7 samples (once as worst on "waiting"). Second best on the remaining samples.
Here are my personal codec averages:
vorbis: 3.85
wma pro: 3.32
mpc: 2.87
aac: 2.77
lame: 2.23
blade: 1.23
With the amount of overlap between confidence intervals for most of the samples (considering our top 4 contenders of course), many (or most) are a statistical tie. Yes, vorbis might not ACTUALLY have first or second place in many samples, but statistically speaking, lots of those third and fourth place finishes are a statistical tie with first. The confidence intervals could be narrowed by having more listeners, but the average scores would change too. If the additional listeners preferred vorbis, it could move to the front. Alternatively, if you were willing to accept, say 80% confidence instead of 95%, you might be able to say more, but be less sure about it.
Now, there could be some interesting things that could be done to analyse these numbers further. I think the standard deviation of each codec would be an interesting number (though I think I'd know who's gonna win that contest). I've got a perl script in the works right now to convert the raw data to .csv so that it can be played with in a spreadsheet. I'll post that later tonight.
Second, because I am only willing to take one other VBR test. The critics on a VBR test are such a PITA that I definitely don't even consider conducing several ones.
I'm sure that if you stuck to either a full ABR or full VBR philosophy there would be fewer critics. Either find the settings that produce an average of 128kbps (or whatever target) across a broad spectrum of music, or find the settings that produce that same average for each individual sample. It would be much harder to find flaw with such a test.
The most important thing isn't really to speculate with statistical tools, but to test by yourself, and finally make your choice according to your own subjectivity, training, hearing abilities, etc...
Generally speaking, and according to the posted results, mp3 - created by Fraunhofer for 1/11 encoding - is now completely outdated. Wma standard family is probably out-of-date too.
Now, we can hope some hardware support for some of these new formats. 120-130 kbps encodings are now close to transparency. Therefore, they aren't sufficient at this bitrate range for archiving/HQ listening, but perfectly suitable for comfortable listening on any portable player (impossible with 64 kbps encoding, crap, even with best algorythms). This is very important for flash memory players. Most of people would be happy with these codecs : they will fit more music without suffering from big artifacts (flanging, pre-echo...).
A new era is beginning...
The most important thing isn't really to speculate with statistical tools, but to test by yourself, and finally make your choice according to your own subjectivity, training, hearing abilities, etc...
Generally speaking, and according to the posted results, mp3 - created by Fraunhofer for 1/11 encoding - is now completely outdated. Wma standard family is probably out-of-date too.
Now, we can hope some hardware support for some of these new formats. 120-130 kbps encodings are now close to transparency. Therefore, they aren't sufficient at this bitrate range for archiving/HQ listening, but perfectly suitable for comfortable listening on any portable player (impossible with 64 kbps encoding, crap, even with best algorythms). This is very important for flash memory players. Most of people would be happy with these codecs : they will fit more music without suffering from big artifacts (flanging, pre-echo...).
A new era is beginning...
I wonder if Sony said the same thing about Betamax...
No hardware support without widespread consumer support, and even if a codec surfaced that was perfectly transparent at 32kbps, not a dent would appear in MP3's dominance without... (?) (I'd say "mindshare").
Most important, thanks for the test Roberto! Great work.
Suprising news regarding WMAPro. Any news on when it will be portable compliant? Since I purchased dBPoweramp, I am ready to encode my collection using WMAPro for my portable use. But my flash and HD players cannot read it.
(MP3 APS with LAME still rules for the combination of transparency and portability! - when the 128 restriction is relaxed)
/me needs MPC portable! NOW! ;-)
I've found the test very amusing except thear1 sample - deafening experience.
BTW, I normally listen to similar music too, but all other samples were MUCH quieter and I've set my volume accordingly...
WMA9Pro has some very serious failure modes, but it was to be expected, it's 1.0 release.
Vorbis has constant quality, good, but is easily detectable.
AAC is very good, for me is the only one which could compete with MusePack.
Can't wait for hardware support and freely available better encoders.
I guess I'm fed up with 'MP3 artifacts' - they sound damn nasty to me.
fewtch: I'd say 'without advertising and hardware support'.
Thanks for organising the test Roberto. Excellent stuff!
I was one who flagged a concern about the bitrates, and I am genuinely sorry for any concern or constenation that I may have caused to others at HA. What caught me by surprise was that we had the discussion thread where people posted a shed load of bitrate test results etc, to get the right q numbers, but then when I started getting into the test proper, seeing a mpc sample hit 197 just caught me by surprise.
I don't think that in the end it really matters, as there are now at least three codecs that got close to the money in the 128 kbits range, despite some using more bits to do so with particular samples. We now have some new successors to LAME, and at least two of them have current or hopefully pending hardware support. B)
Thanks again for all your efforts Roberto.
As guru said, and I've also mentioned previously, it still comes down to personal preference, and testing/deciding for yourself. Hopefully this test has opened a few people to alternatives to their own favourite format. B)
I just did my own abx test using the same codecs as in Roberto's test. I used several pieces of music that I am very familiar with. I could NOT abx any of the
encoded files vs the .ape file used as the reference for the test (I used foobar 2000
version .667 for the abx test) except for the lame encoded file, and I could BARELY abx that. This tells me my almost 45 year old ears are past their prime, and I could use any codec for portable use. I still like the idea of storing all my music as lossless though, so I can encode to anything on the fly for portable use (mp4 for ipod, wma9 pro for a pocket pc etc.) . I use dbpoweramp to go from .ape to lossy. It has been working great.
Thank you for the informative test. Since I have proved to myself my ears are crap, I can now sleep beter at night.
OK, as promised, I've compiled all the results into one big spreadsheet. You can download it in zipped XML (OO's native format) here: 128kbps_results.sxc (http://www.phong.org/audio/128kbps_results.sxc)
Wait? Your spreadsheet won't take that? Ok, you can get an .xls here (grumble, grumble): 128kbps_results.xls (http://www.phong.org/audio/128kbps_results.xls)
Let me know if you have problems with that. I don't know how good OpenOffice's .xls export is.
Feel free to do whatever you want to it.
I've included two sections. On the left are all the individual results. It has each listener's individual ratings, and how each codec ranks (i.e. 1st, 2nd, 3rd) for each sample. At the bottom of each sample, there is the average and standard deviation for that sample for each codec. Way down at the bottom are the averages and standard deviations for all the samples.
On the right are all the per user averages. You can see what the average user prefers, and compare that to each individual user. Over here, I've got a couple cells that I left filled in that probably aren't that meaningful (like the standard deviation of all the listeners' standard deviations), so take all the numbers well salted.
Dude, to tell you the truth, I keep wondering if it's not better to keep these results out of /. :-/
Ya, I think you're right. Anymore, slashdotters are hosers, eh. I'm sure they'd all point out the 4.25, 4,25 issue with vorbis. I wish there were a general-purpose geek forum that didn't suck.
Since I have proved to myself my ears are crap, I can now sleep beter at night.
Don't be so sure. I couldn't pick out any at first either (even blade on some!) but after a little training I could ABX the majority. Of course, now all these old 128kbps mp3s are completely unlistenable.
Also, I believe many of the samples chosen were ones known to cause trouble for lossy codecs, so they're more likely to reveal problems than the average piece of music.
Thanks for the vote of confidence phong
I was too busy to do the test itself that Roberto setup, maybe I can download his samples and try the abx for the codecs myself. I still think my hearing is at a disadvantage compared to the 20 something people that heavily populate this forum.
Many spoke of needed hardware support for WMA9 Pro. I bought my wife's Pocket PC back in March 2002, and it has WMP version 8.5 installed on it, which plays back the one pass and two pass VBR wma files just fine (must be a beta version that was installed on that version of Pocket PC OS) so I imagine all the new Pocket PC's should have no trouble with WMA9 Pro VBR files. I can see needing to update portable cd players that decoded wma files, since most of those are prolly based on the older codec.
OK, as promised, I've compiled all the results into one big spreadsheet. You can download it in zipped XML (OO's native format) here: 128kbps_results.sxc (http://www.phong.org/audio/128kbps_results.sxc)
Wait? Your spreadsheet won't take that? Ok, you can get an .xls here (grumble, grumble): 128kbps_results.xls (http://www.phong.org/audio/128kbps_results.xls)
Let me know if you have problems with that. I don't know how good OpenOffice's .xls export is.
Looks like there's some sort of formula error when it calculated how many samples I had listened to (I listened to all 12, but it only showed 11). I'm looking at the .xls version.
ff123
Interesting factoid:
Listener annoyance ranking (who tended to rate the lowest):
proxima 2.55 (7 samples)
Guruboolez 2.71 (12 samples)
gecko 2.73 (12 samples)
kl33per 2.77 (3 samples)
dimkovic 2.92 (1 sample)
ff123
Hmm, I double checked, and I get 12 for you for both versions. It could be OO's fault. The xls file I have there is exported for Excel 97/2000/XP. I just made a version for Excel 95 that you should also be able to import (it may work better): 128kbps_results_95.xls (http://www.phong.org/audio/128kbps_results_95.xls)
Did the other numbers look right? By my count, 8 people should have all 12. What shows as the formula for that cell? (Should be something like "=COUNTIF(B1:B302;Q36)")
Edit: Also, does it show your name identically for all samples (i.e. ff123, lower case, no leading or trailing spaces)?
Hmm, I double checked, and I get 12 for you for both versions. It could be OO's fault. The xls file I have there is exported for Excel 97/2000/XP. I just made a version for Excel 95 that you should also be able to import (it may work better): 128kbps_results_95.xls (http://www.phong.org/audio/128kbps_results_95.xls)
Did the other numbers look right? By my count, 8 people should have all 12. What shows as the formula for that cell? (Should be something like "=COUNTIF(B1:B302;Q36)")
Oops, my fault. I munged it up by sorting.
ff123
Man, LAME really is dated at this bitrate. Hopefully no one misinterprets this test and decides from it that all these codecs perform equally the same at higher bitrates. Please remember that this is only at 128kbps. alt-preset standard, vorbis gt3, musepack standard, etc will be tested later, if that would even be a valid or necessary test. Taking from the difficulty that this test has shown, it would be no wonder even harder to test the higher bitrates.
Even though I know that it is incorrect to make blanket statements, I'm sure that because of this test we will see them. If it wasn't for compatibility, I think you would see people leaving LAME in droves now. While I guess it might be plausible to make judgements from this data that LAME is dated, that other codecs have seen greater development as of late, and then as such decide to use a different codec for all your encoding needs, I don't think it would be a good idea to directly link this test to overall sound quality.
I agree with these sentiments. Since we know that lame --aps is transparent in nearly all cases it seems a bit harsh to say that this test proves that lame is crap. Certainly it is inferior at 128kbps, but since most of us HA-enlightened mp3 users encode with --aps is this really relevant? We already knew that 128kbps mp3 was not that good. With the availability of cheaper storage and faster internet access, why the imperative to have 128kbps files? I totally agree that if there was hardware compatability for other formats then lame mp3 would be redundant, but the main reason for this would be for gapless playback rather than sound quality.
I'm not trying to slag off the tests here. The work everyone put in was fantastic, and the results are important and useful. Congratulations to all involved! I just don't want the wrong conclusions to be drawn. (I am imagining a lot of uninformed people transcoding their --aps mp3s to mpc to get better quality. )
.dd.
Oops, my fault. I munged it up by sorting.
No, my fault. I forgot that you can use $'s in ranges to make them behave. I've uploaded new versions (same urls). Should be able to sort or do other nifty operations now without causing havoc.
Listener annoyance ranking (who tended to rate the lowest):
proxima 2.55 (7 samples)
Guruboolez 2.71 (12 samples)
gecko 2.73 (12 samples)
kl33per 2.77 (3 samples)
dimkovic 2.92 (1 sample)
On looking upon how other people have marked, I think I've marked to hard. Maybe there needs to be more solid criteria then the "Imperceptible, Perceptible but Not Annoying, Slighly Annoying, Annoying, Very Annoying". After all, this sort of marking scheme is very subjective. What's annoying to one person might not be annoying to another, even if they both here the same flaws in the file.
BTW, Roberto, nice test, thanks for holding it. Also my thanks to all those who participated.
I'm playing with the data right now. It's kind of interesting to split the results in half after sorting by average rating. The low scorers (the most sensitive listeners) spread the scores out more. When the high scorers results are averaged in, the winning codec(s) sometimes get squashed back into the group. This appears to be because the less sensitive listeners aren't really able to distinguish at the highest levels, so they aren't giving the best codec(s) their proper due.
For a high bitrate test, this has implications: only the most sensitive listeners should participate because the less sensitive listeners will "squash" the results at the top. Only a dozen people are really needed, but they need to be able to hear the subtlest flaws.
I'll try to post the two different types of graphs later.
ff123
I've got a perl script in the works right now to convert the raw data to .csv so that it can be played with in a spreadsheet. I'll post that later tonight.
I uploaded all the test results in a nice .zip package, so that you don't have to download one by one.
Also inside are the tables ready to be fed to friedman.exe processing (folder \Friedman), and the Anova scores for each sample and for all samples (folder \Scores-Anova)
The package is here:
http://rarewares.hydrogenaudio.org/rja/Results.zip (http://rarewares.hydrogenaudio.org/rja/Results.zip)
I was one who flagged a concern about the bitrates, and I am genuinely sorry for any concern or constenation that I may have caused to others at HA.
Man, it wasn't your fault at all! :B
You actually came politely to me and raised a genuine concern.
What I can't stand is people mailing me with something like "You are using VBR, your test suxxxxxorz!" (not that bad, but close)
Geh, I took too long to upload the results .zip. Sorry phong. :-/
http://ff123.net/export/128exten_split.html (http://ff123.net/export/128exten_split.html)
On individual samples, the split between high scorers and low scorers seems to make a difference, but the overall results are similar. There are no new significant differences, but MPC vs. Ogg comes closer (about 89% confidence for low scorers that mpc is better than ogg).
ff123
Geh, I took too long to upload the results .zip. Sorry phong. :-/
No worries. Perl, bash and wget automated the task by combining their powers, as if they were five giant robot cats coming together to form a super-robot with a big sword for slashing at other robots, except there were only three of them and they don't wear spandex or helmets and...
I'll incorporate those Anova scores as soon as I get home and figure out what they mean.
delete in_wm.dll and add wma to the extension list in in_dshow's configuration
Hey Cool it works ThX
But a Official or a Plugin Support for Winamp would be Cool 2.
On looking upon how other people have marked, I think I've marked too hard. Maybe there needs to be more solid criteria then the "Imperceptible, Perceptible but Not Annoying, Slighly Annoying, Annoying, Very Annoying". After all, this sort of marking scheme is very subjective. What's annoying to one person might not be annoying to another, even if they both here the same flaws in the file.
And what exactly would you propose as an alternative marking scheme? This scheme, although admittedly subjective, serves its purpose just right, because quality is precisely a subjective term. Of course, what one person finds annoying might not be so for another, but we are not looking for an absolute scale because there can´t be any. What we are trying to do is to find the one that sounds best for the majority of people, and if some people are more "sensitive" or "exigent", that of course will lower the average. That, however, doesn´t matter very much, since what we are trying to find are the relative ranking among the codecs. You could delete all numerical values and leave only the graph and it would still tell what it needs to tell.
As a matter of fact, I can´t even think of a more objective marking scheme that makes sense. 5: 0-100 millikleepers of difference with the original, 4: 101-200 millikleepers of difference with the original...?
-Dologan
This topic's a whole othrer thread, but anyway. I don't how it should be changed, just that maybe they're should be more solid criteria for each level. It's not that I think I thought the qualtity of these files was any less then the next person, just that I had no reference on how to score this (apart from the aforementioned five written levels). Anyway, it was just a thought, and I'm not really sure what to do about it, it just seems like I was to critical of the codecs, that's all.
http://ff123.net/export/128exten_split.html (http://ff123.net/export/128exten_split.html)
this could just be my spacial perception, but on the (Overall, high srocers) Ogg seems to win over WMA. so what are those graphs? what does it mean (in laymans terms if possible )?
I just had an idea for another test. Compare the major codecs at the broadly accepted high (but not stupid) quality level, eg lame -aps, ogg -q6, mpc -xtreme etc.
Two reasons for doing this - first, it would be an interesting comparison of the various recommended settings. But more important, imagine the flamefest! The amount of crap Roberto got just because VBR is, well, variable, would fade into insignificance!
Cheers, Paul
I just had an idea for another test. Compare the major codecs at the broadly accepted high (but not stupid) quality level, eg lame -aps, ogg -q6, mpc -xtreme etc.
Yeh, I've thought about this to, but as mentioned, these codecs are pretty close (if not actually) transparent at these bitrates, thus making the ABXing of these files much harder.
Two reasons for doing this - first, it would be an interesting comparison of the various recommended settings. But more important, imagine the flamefest! The amount of crap Roberto got just because VBR is, well, variable, would fade into insignificance!
Poor Roberto, I pity him, he goes and sets up this huge test with 12 samples, 6 codecs, website and all, and then gets flamed for using codecs in their optimal (or only) mode.
http://ff123.net/export/128exten_split.html (http://ff123.net/export/128exten_split.html)
this could just be my spacial perception, but on the (Overall, high srocers) Ogg seems to win over WMA. so what are those graphs? what does it mean (in laymans terms if possible )?
In statistical terms, you can only say that one codec is significantly better than another with 95% confidence if the bottom error bar of the better codec does not overlap with the top error bar of the worse codec. Otherwise, they are considered to be the same quality.
So in the overall high-scorers graph, ogg is not significantly better than wma9pro.
ff123
I couldn't help noticing that at both samples that could be classified as classical (Bach and Saint-Saens), WMA9Pro won with pretty good scores (~0.3 above the next "hardwareable" codec (let's face it - MPC will never get widespread hardware support - if any at all). I wonder whether WMA9Pro has such a good performance with all classical music in general, since if so, it might become my format of choice for the classical collection on my portable . Unfortunately I am not able to test that by myself at the moment, since I am on a trip and all my classical is encoded, and trascoding is something I'd like to avoid for this kind of judgment.
~Dologan
Compare the major codecs at the broadly accepted high (but not stupid) quality level, eg lame -aps, ogg -q6, mpc -xtreme etc.
I think Musepack at -standard is competent enough.
Testing -q6 and above would almost be pointless.
I couldn't help noticing that at both samples that could be classified as classical (Bach and Saint-Saens), WMA9Pro won with pretty good scores (~0.3 above the next "hardwareable" codec (let's face it - MPC will never get widespread hardware support - if any at all). I wonder whether WMA9Pro has such a good performance with all classical music in general, since if so, it might become my format of choice for the classical collection on my portable . Unfortunately I am not able to test that by myself at the moment, since I am on a trip and all my classical is encoded, and trascoding is something I'd like to avoid for this kind of judgment.
~Dologan
I noticed it too. I'm looking for an encoder at low-mid bitrate, for storing some discs on my new portable computer, very limited in space. I tried to found good settings/codec at 70-90 kbps, but too many artifacts remained. HE-AAC isn't very efficient at the moment, and can't be seriously considered (nevertheless, i hope fast improvments). Therefore, 128 kbps seems to be a good compromise.
Two potential challengers here :
AAC MP4 (QT) and
WMA9pro. Quicktime isn't suitable for batch encoding, and tagging with WMA9Pro is something impossible with foobar2000. That's really annoying !
Compromise :
Nero MP4 VBR --streaming, less reliable in my opinion than QT 128, but good quality, and easy to use (transcoding forom lossless is easy).
Vorbis doesn't really interest me : too noisy for my taste, and harpsichord isn't well-reproduced.
MPC ? Hmm... I wasn't convinced for classical music and --radio. And my baroque taste will conduct to high bitrate (close to 150 kbps)
I'm gonna play the next time with WMApro, with VBR and Two-Pass, at mid-bitrate. I didn't have time and will enough for serious comparison, but for the moment, I didn't noticed something really annoying with this encoder, with all tracks and samples I tested. I have to look carefuly on loud music : wma standard often destroyed the sound, and I suspect PRO encoder to share the same flaw (less pronounced) : may be annoying with grand-orchestra.
Anyway, if PRO vbr encoder isn't amazing at low bitrate (try any metal sample at VBR 25-50 : it's obvious), quality seems to be more interesting for classical music (metalic coloration is the most annoying thing) ; bitrate is sometime very low.
Sorry if this gets a bit OT, but I think it's relavent. I'm wondering if, at this bitrate, vorbis would benefit from use of the lowpass switch. Many of the comments complained of the high-frequency noise or brightness of the vorbis samples. While I realize that doing a lowpass isn't the inverse of the high-frequency problem with vorbis, it may help tone down the percieved brightness a bit while at the same time, saving bits that may help with other artifacts (i.e. a higher -q level could be used to achieve the same average bitrate).
Thoughts? Comments? Flames?
Sorry if this gets a bit OT, but I think it's relavent. I'm wondering if, at this bitrate, vorbis would benefit from use of the lowpass switch. Many of the comments complained of the high-frequency noise or brightness of the vorbis samples. While I realize that doing a lowpass isn't the inverse of the high-frequency problem with vorbis, it may help tone down the percieved brightness a bit while at the same time, saving bits that may help with other artifacts (i.e. a higher -q level could be used to achieve the same average bitrate).
Thoughts? Comments? Flames?
I tried to 'tweak' Vorbis by doing that, long time ago. First : bitrate is quite the same (classical music - it may change with louder music). Second : hiss remains, at the same level.
NOTE: What follows is said concerning Korn CD's and them alone, I did not notice such differences in any other CD I have.
Seeing how well WMA seemed to perform in this test, I just have to add something from my own experience (perhaps you may correct a mistake I made somewhere)
First off I used WMA vbr 135 - 215 kbps setting, since I didn't like the size of loosless, and there is nothing that says Pro in WMP XP. Where did you get your encoder from?
In my experience any Korn song which has both base guitar and vocals at the same time, is a killer sample for WMA. The instruments (including the base guitar) are perfect (the bass is gorgeous, I was droolling for like 5min in a trance), but then realization that something is wrong grew in me. Being so amazed about the improvement in bass/drams/guitar performance in WMA over mp3 preset extreme I didn't notice right away that voice of the lead singer was coming from somewhere far-far-away, at times base-guitar would totally cover the voice. It sounded like the mikes were in a room with the instruments, but the singer was in a corridor leading to the room and was singing to the mike from there.
What makes it even more strange is that the voice doesn't seem like it's strained, but in fact it is smooth, clean, and well rehearsed.
So if someone have experienced similar problem in WMA vbr, could you comment if it is something that can be fixed or is it a flaw?
Please tell me it can be fixed, because I tried mp3 extreme, ogg -q6 gt2, and acc QT cbr 192 (or close to it). WMA just has the best bass/drum/base-guitar performance I have heard. others are more ballanced (instruments and voice are on equal level) but a lot of Korn music is based on bass/base guitar, I want that extra kick I get from WMA.
First off I used WMA vbr 135 - 215 kbps setting, since I didn't like the size of loosless, and there is nothing that says Pro in WMP XP. Where did you get your encoder from?
http://www.hydrogenaudio.org/forums/index....50&#entry115951 (http://www.hydrogenaudio.org/forums/index.php?showtopic=11383&st=50&#entry115951)
Right now, you're using WMA9 standard, which sounds worse than WMA9Pro.
ff123
The amount of crap Roberto got just because VBR is, well, variable, would fade into insignificance!
I don't think the problem was so much the VBR as the mix of ABR and VBR modes, like I've said previously (and have not yet had my logic critiqued, hint, hint! ). Sure, under normal circumstances, there's not much difference, but when you're testing, you should stick to a single methodology.
Hello,
Just as a followup to my mini discussion with phong last night, I downloaded two
problem samples that people seem to use around here; fatboy.wav and catanets.wav. When I ran the abx tests using just a couple of codecs (.ogg and wma9 pro) in the 128 kbps vbr range, I could FINALLY abx the encoded files from the wav files. Foobar 2000 .667 abx comparator told me probabilty of guessing was between 1.1-3.2 percent. I guess my ears are not as bad as I feared. I still cannot abx "normal" music with the codecs used in the test-so I can STILL sleep better tonight knowing that I can use 128 vbr with just about any codec out there
(never had the "fortune" to have used blade or xing) for portable use
but when you're testing, you should stick to a single methodology.
I sticked to a single methodology. The methodology of using whatever settings that output the best quality for each codec.
but when you're testing, you should stick to a single methodology.
I sticked to a single methodology. The methodology of using whatever settings that output the best quality for each codec.
Which is good enough for me.
As well as me. I'm very pleased with the manner in which the test was conducted, and was pleased with the results. Thanks Roberto!
OT - can we include some build of ATRAC that we can all get ahold of, in future multiformat blind testing?
Yes, if I gather the courage to conduce another 128kbps test.
It would be interesting to compare WMA9Pro VBR in kinda the same bitrate range as MPC Standard.
and there is nothing that says Pro in WMP XP. Where did you get your encoder from?
dBpowerAMP Music Converter with the WMA 9 Codec.
NOTE: What follows is said concerning Korn CD's and them alone, I did not notice such differences in any other CD I have.
[snip]
As you probably know, if you're going to get answers to question like this then...
a) show you have ABXed the coded vs original versions,
B) provide a sample, and (in this case)
c) don't expect anyone on this board to be able to apply some nice code-level tweaks to the WMA encoder to help you! ;-)
However, if there are audible problems with lame --alt-preset extreme, we should all take a listen.
Cheers,
David.
I will add a table with all the bitrates to the results page later today.
unable to locate such table, where is it? tia.
alt-preset standard, vorbis gt3, musepack standard, etc will be tested later
Will they? When? and by who?
Surely not by me. First, because these bitrates are transparent, we would get most of the codecs reaching score 5. Second, because I am only willing to take one other VBR test. The critics on a VBR test are such a PITA that I definitely don't even consider conducing several ones.
alt-preset standard, vorbis gt3, musepack standard, etc will be tested later, if that would even be a valid or necessary test.
You cut out the most important part of what I said:
if that would even be a valid or necessary test.
a - show you have ABXed the coded vs original versions,
b - provide a sample, and (in this case)
c - don't expect anyone on this board to be able to apply some nice code-level tweaks to the WMA encoder to help you! ;-)
a - first and what exactly stops me from faking those? secod if listening to CD then WMA then mp3 and ogg counts, then yes WMA has more of a CD-type base/drum performance, others are flat (not bad, just no kick after silent moment)
b - sample of copyrighted music?
c - I don't wan't code level tweaks, I was asking where I could get the encoder, since I was clearly using the wrong one.
finally I don't believe I stated that there is a problem with lame preset extreme what I said is that WMA has a more pronounced bass, but
WMA has problem with voice when base is present.
b - sample of copyrighted music?
Samples below 30 seconds are allowed by law. (I think. Read forum rules.)
a - first and what exactly stops me from faking those?
Everybody can do that.
If we ask for ABX test, it isn't really for us, but for you. For exemple, I often had some good feeling (untidy bass, less high frequencies, slight distorsion, some hiss...) when listening directly an encoded file with winamp or foobar. But when I performed a blind comparison, my good feeling and trust disappeared, and I wasn't able to heard something wrong anymore.
That's why ABX test are needed : just to be sure that the problem you noticed really exist.
unable to locate such table, where is it? tia.
Sorry, it took longer than I expected.
Already up, located just after the individual plots.
rjamorim, can you plz also add a zoomed in picture of the results (skipping blade for this would surely help)
Done.
I didn't do that before exactly because of Blade, but since it's the anchor anyway...
thanks!
uah, poor lame
btw. faac reached in the aac test 3,52, lame now 3,66 (ok i know that this cant be compared but... )
btw. faac reached in the aac test 3,52, lame now 3,66 (ok i know that this cant be compared but... )
If I ever compare several MP3 encoders at 128kbps (yes, I'm planning that), I might throw in FAAC just for kicks and giggles :B
Ok doing a simple percentage alignment for any codec over 128Kbps (don't know if that is fair or not, a codec given 64Kbps would be 25% of one given 256Kbps?...):
Adj Result = Result * (128 / Avg Bitrate)
Adjusted Results are Adj Result (Previous results):
AAC 4.38 (4.42)
Lame 3.77 (3.66)
MPC 3.95 (4.51)
Vorbis 3.91 (4.28)
WMA Pro 4.30 (4.30)
Blade 2.22 (2.22)
BTW Was WMA Pro encoded as 24 bit 2 channel? (seems to be the only Pro option that will encode 16bit stereo on mine), if so it would be interesting to feed it 24bit data as opposed 16bit to see if the sound quality improves.
Lies, damn lies and statistics come to mind, in that results can be adjusted up or down...make of it as you will
Yay, a 64kb test should mean I can take part When you're ready I'll finally be there to help you Roberto
Lies, damn lies and statistics come to mind, in that results can be adjusted up or down...make of it as you will
Quoting Congressman Alex Shrub:
"Those statistics are interesting, but like all statistics, they are also irrelevant. "
"Facts, shmacts... you can use facts to prove anything, that even remotely true!" - Homer J. Simpson.
If I ever compare several MP3 encoders at 128kbps (yes, I'm planning that)
Would be quite interesting...
Aren't there any tests yet?
Iam Happy that my Favorite Codec WMA show the rest that he is better and not so Crappy like Everybody said.
But i have One little Problem a wma9pro Encoded File dont be played in Winamp 2.95.
Is there a plugin that wma9pro played without the Windows Media Player.
ThX
Unfortunately, that's the "one little problem" with using wma9pro right now. Although it sounds better than the older wma format, it breaks compatibility with current decoders. I'm sure that will change in due time.
ff123
Let's say I want to play wma9 Pro and older wma encoded files. Do I need the old AND the new codec in order to play it, or does wma9 Pro codec already include the 'old' codec as well? I mean in that case I wouldn't see too many problems...
Do I need the old AND the new codec in order to play it, or does wma9 Pro codec already include the 'old' codec as well?
I think you would need to ask that to Microsoft. :-/
Wma in all its confusing glory:
Basically everything wma that is not pro, lossless or voice (even WMA v9 standard) can be played by all WMA codecs including the first v2 codec.
To play pro, lossless and voice install the wma v9 codecs (which will play everything).
Ok doing a simple percentage alignment for any codec over 128Kbps (don't know if that is fair or not, a codec given 64Kbps would be 25% of one given 256Kbps?...):
Spoon: you are my hero.
But I thought that you people said that the grand total average bitrate of all tracks (i.e. add all average bitrates and divide by # of tracks, 12 I believe, for a specific codec) was tuned to all be -exactly- 128?? Or? If so then spoon is wrong.. (or Im just missing something obvious, wouldn't be the first time)
But if not then this is even more staggering.. (and.. exactly what I mentioned over and over again ).. so byte for byte AAC beats MPC if you extrapolate results from 'bitrate really used' to 'what it would be rated at when bitrate was really 128'?
Cool. and here I was thinking that maybe the overall differences would be too small to influence score, but at least for specific tracks this correction factor could give hints about what codec is best suited for what type of sample. Incidentally I realise that you can't just do this correction without giving the realities of lossy coding a second thought, but this is quite some food for thought.
(Plus next time PuntCodec is going to be submitted in there.. Lossy to the max people..)
Sooo.. reorganising Spoon's list based on adjusted score we get the following ranking:
Adj Result = Result * (128 / Avg Bitrate)
Adjusted Results are  Adj Result  (Previous results):
1: AACÂ Â Â Â 4.38Â Â Â (4.42)
2: WMA Pro 4.30   (4.30)
3: MPCÂ Â Â Â 3.95Â Â Â (4.51)
4: Vorbis   3.91   (4.28)
5: Lame   3.77    (3.66)
6: Blade   2.22    (2.22)
Lies, damn lies and statistics indeed... B)
Incidentally, rjamorim, don't take my criticisms the wrong way, I think the test was pretty solid and the results interesting! Good job.
While we're critisizing though, is there any way to eliminate the 'listening to crap hardware' factor in this kind of test? For example vorbis introduces more noise than 'average' (whatever that is), but if you listen on a stereo that hisses already you might find vorbis the best codec by far..
If you listen to bad headphones you might lose all lows and rate codecs that reproduce bass well way too low..
And well I still would be interested in 160kbps listening tests.. As most people agree, abx'ing -q6 settings will probably be drowned in 'noise' from people (like me probably) who can't hear the difference with the original, but q5 might still be doable. Plus that really comes into the ranges the codecs were 'designed' for, right?
I mean I saw claims that musepack is the #1 codec for 160kbps or more, I know that vorbisenc GT3 mostly has tweaks for -q5 and better and so on..
AAC beats MPC if you extrapolate results from 'bitrate really used' to 'what it would be rated at when bitrate was really 128'
This whole disagreement is incredibly bogus... I thought it had gone away when rjamorim explained how he picked the q ratings for vbr codecs (the average bitrate for your entire collection will be 128kbps even though for these deliberately singled out difficult pieces it is higher).
I think rjamorim should force spoon and puntloos to do a listening test of his entire music collection, just so that the bitrates come out right.
If puntloos still thinks these bogus statistics mean anything at all then I can upload the test encoded with floggy: even if I only scored 0.2 I'd come out on the top of spoon's table, and could claim to have the best 128kbps codec.
But I thought that you people said that the grand total average bitrate of all tracks (i.e. add all average bitrates and divide by # of tracks, 12 I believe, for a specific codec) was tuned to all be -exactly- 128?? Or? If so then spoon is wrong.. (or Im just missing something obvious, wouldn't be the first time)
Note: This post is long and getting buchered, so I've broken it up into several. Sorry.
I again suggest you read the pre-test thread to clarify things. You can do that here (http://www.hydrogenaudio.org/forums/index.php?showtopic=11383). Also, look at this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=11134&hl=) if you want to see, in detail, how the settings were chosen. Several settings were tried over a large number of whole albums. The settings that came closest to averaging 128 over all those albums were the ones used for the test.
The bitrates of the chosen samples are not representative of the bitrates you would get on average over a typical music collection. They are often higher because difficult portions of music are overrepresented.
I don't think anybody ever takes 20-30 second clips of several different pieces of music and encodes them several times at different VBR settings until they get 128kbps. On the other hand, I'd wager almost everyone chooses settings that meet their filesize/quality tradeoff requirements then encodes all their music with the same settings (though their settings might change over time as their needs change).
I maintain that to repeatedly encode your collection tweaking settings on a file-by-file basis so that each song (or album) comes out to 128kbps makes as much sense as encoding longer songs at lower quality levels so they come out to the same file size as shorter songs.
Furthermore, I think adjusting the test scores by multiplying them by the ratio of actual bitrate to target bitrate makes no sense. It can't really represent how well they would actually perform in a CBR/managed bitrate comparision. There is no reason to believe, for example, that if MPC encoded a short sample at 256kbps and got a "5.0" would get a "2.5" if forced to use 128kbps. In fact, there's no basis on which to make ANY assumptions about what score a codec would get at a particular bitrate (other than a higher bitrate probably corresponds to a higher score, and a lower bitrate corresponds to a lower score).
If you would like to run a CBR comparison, you may get the results you're looking for. However, I don't know how much participation you'll get since the only legitimate use of CBR is streaming. A 64kbps CBR test would be interesting, because that is a bitrate well suited to streaming.
Put it this way, no one expected MPC to win this test and when it did you could say an average combined bitrate of 140Kbps over 12 tracks helped, when something like AAC was 128Kbps spot on.
I think it is right to challenge convention, remember a test for 128Kbps is aimed at an end user who is limited by space, if AAC was allowed an average of 140Kbps then the initial results would have been different.
I have nothing against rjamorim, mpc or abx testing, infact I recommend people to use MPC above 160Kbps.
I think it is right to challenge convention, remember a test for 128Kbps is aimed at an end user who is limited by space, if AAC was allowed an average of 140Kbps then the initial results would have been different.
Yes, but if the end user chooses mpc q4 as his default setting, on a large scale the bitrate will average 128kbps. Thus he is using up practically the same amount of space as when using aac cbr @ 128kbps. That's why choosing this setting for the test makes perfect sense.
Noone would recode over and over again until 128kbit is achieved in a realistic scenario. But this is a -test-, isn't it? Phong gave the floggy example (which, c'mon.. would never be given even more than a score 1, even compared to the 5 of Blade)..
Saying 'floggy scores 1.1 at 4kbps and MPC scores 4.5 at 128kbps' so floggy wins if you multiply up to compensate for average bitrate isnt fair, the simple calculation curve I suggested before and Spoon used does not work for encoder settings that are in completely different leagues. Anyway I doubt we (humans) have the capacity to fairly rate and compare a floggy and a mpc-q4 encoding. Would floggy be 1.1? 0.5? 0.05? I really wouldn't know how to fairly rate floggy if it were used as the anchor test instead of blade.
The encoders tested here however are in the same league. They were designed for these kinds of bitrates (perhaps a bit higher) and didnt vary that much in resulting bitrates.
However.
In -this- test, evidentally Q4 on MPC turned out to be 140kbps average. What can we conclude? Nothing solid. But we can at least hypothesise that:
1/ "Q4" for MPC coders meant something slightly different than 'has to average at 128" for one I think Q5 was intended to be 'transparent', and wasnt set while trying to fit an average bitrate (160?). Who says Q4 was intended to be "128"?
2/ Maybe the settings rjamorim and the people who made the test were indeed giving them 128kbit average with their samples, but they didn't for the end result of the test. I think it is safe to say that this means MPC got unlucky and didn't find the test sample package 'averagely easy'. And this will skew the results compared to (say) AAC. Yes, MPC scored higher on sound quality at the expense of bitrate. MPC should not be allowed to get away with this since I refuse to believe we just compare quality, period.
If we did only compare quality, PuntCodec which strips only the wav headers and calls itself lossy should win. This is useless information. The results aren't useless but I say that for fairness purposes it is skewed.
Who says Q4 was intended to be "128"?
I
REALLY think you need to read this again.
Yes, but if the end user chooses mpc q4 as his default setting, on a large scale the bitrate will average 128kbps. Thus he is using up practically the same amount of space as when using aac cbr @ 128kbps. That's why choosing this setting for the test makes perfect sense.
and this
Furthermore, I think adjusting the test scores by multiplying them by the ratio of actual bitrate to target bitrate makes no sense. It can't really represent how well they would actually perform in a CBR/managed bitrate comparision. There is no reason to believe, for example, that if MPC encoded a short sample at 256kbps and got a "5.0" would get a "2.5" if forced to use 128kbps. In fact, there's no basis on which to make ANY assumptions about what score a codec would get at a particular bitrate (other than a higher bitrate probably corresponds to a higher score, and a lower bitrate corresponds to a lower score).
and this one too for good measure.
This whole disagreement is incredibly bogus... I thought it had gone away when rjamorim explained how he picked the q ratings for vbr codecs (the average bitrate for your entire collection will be 128kbps even though for these deliberately singled out difficult pieces it is higher).
edit: added more quotes
In -this- test, evidentally Q4 on MPC turned out to be 140kbps average. What can we conclude? Nothing solid. But we can at least hypothesise that:
1/ "Q4" for MPC coders meant something slightly different than 'has to average at 128" for one I think Q5 was intended to be 'transparent', and wasnt set while trying to fit an average bitrate (160?). Who says Q4 was intended to be "128"?
2/ Maybe the settings rjamorim and the people who made the test were indeed giving them 128kbit average with their samples, but they didn't for the end result of the test. I think it is safe to say that this means MPC got unlucky and didn't find the test sample package 'averagely easy'. And this will skew the results compared to (say) AAC. Yes, MPC scored higher on sound quality at the expense of bitrate. MPC should not be allowed to get away with this since I refuse to believe we just compare quality, period.
Before the test, rjamorim asked people to encode one or more albums at a couple of different q values in this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=11134). Based on those results, he concluded that
for the majority of albums Q4 was most likely to have an ABR of 128. So what if it didn't for the samples in this test? That indicates, as 264556 said, that these pieces are more difficult to encode (and better for use in tests), and that Musepack is a better overall encoder, since it is capable of raising the ABR that high when necessary, while still maintaining an overall ABR of ~128kbps for the large majority of music.
That indicates, as 264556 said, that these pieces are more difficult to encode (and better for use in tests), and that Musepack is a better overall encoder, since it is capable of raising the ABR that high when necessary, while still maintaining an overall ABR of ~128kbps for the large majority of music.
Thanks for putting it EXACTLY the way I wanted to.
Wow, MPC tops the 128 kbps range as well? Impressive stuff. I might start playing with MPC more then.
Actually, I think some credit is due for Vorbis. Even though it didnt do that well in the test, at least it proved true in their claim that Vorbis will do as good, if not better than mp3. Also, if we view this from another perspective, Vorbis is a product of the free open source concept rather than intensive and sophicated research and development by the big companies like Fraunhofer, Dolby, or Microsoft. Thus for it to keep up with the other state-of-the-art codecs is quite an achievement within itself. Take for example gcc. It used to as slow as Borland's compiler but now is achieving pretty impressive performance. However, compared with sophisticated compilers from Intel, it isnt easy for open source gcc to surpass them.
If I can ever understand the Vorbis source code, 128 kbps is the range I'm gonna try tuning for.
Anyway, thanks to Roberto for this very informative test as well as all those listeners.
I don't think anybody ever takes 20-30 second clips of several different pieces of music and encodes them several times at different VBR settings until they get 128kbps. On the other hand, I'd wager almost everyone chooses settings that meet their filesize/quality tradeoff requirements then encodes all their music with the same settings (though their settings might change over time as their needs change).
You should read hedline of this topic again. I quite sure that there is a text "128kbps Extrension Test". There is'nt mentioned anywhere anything about testin certain quality settings of encoders. I thought we should do this test to see how different codeks can handel common problem samples at same average bittrate. I think almost everyone here who encodes music him self with lossy codek will not use 128kbps average size without any special reason.
To test efficiency, we need to have more closely similar bitrates. Regardless of whether MPC is good at choosing higher bitrates for those seldom "difficult" tracks, there should be enough tracks that are NOT this difficult to encode for us to test with. Neither known difficult tracks for certain encoders, nor known easy tracks for one encoder should be used that could skew results. To test efficiency, which i believe is what should be the goal for a 128kps listening test, the bps should be more similar.
The arguments over the test method as it concerns bitrate have been hashed and rehashed many times over now, and the opinions have solidified. Maybe it's time to close this particular topic.
I think that a 64 kbit/s test could be designed to move closer to happy medium, since difficult samples are not be needed to discriminate between codecs. So it would be possible to try to choose samples such that the entire test suite averages about 64 kbit/s for each codec.
For this test, though, there's not much more to say. It's done and over with. I will add that I personally think Roberto's tests are of the highest quality. There's not a comparable test out there on the net for 128 kbit/s codecs (cT's test is a joke by comparison). For those who would have done it differently: the opportunity is still there for you! If you can dish out the criticism, can you stand to take it too?
ff123
...
Do not take my last post as criticism for the last test. I am speaking only for the future. I highly respect the quality of this last test, and I only want to make the future tests even better. This test proved very informative, I only would like to see how much closer it would be if we fine tuned it in the future, that's all.
For this test, though, there's not much more to say. It's done and over with. I will add that I personally think Roberto's tests are of the highest quality. There's not a comparable test out there on the net for 128 kbit/s codecs (cT's test is a joke by comparison). For those who would have done it differently: the opportunity is still there for you! If you can dish out the criticism, can you stand to take it too?
ditto.
@ff123: thanks for the kind words. (And thanks to everyone else that praised the test. That is _really_ welcome)
Edit: post split in two parts due to @#$%& forum bugs.
Vorbis is a product of the free open source concept rather than intensive and sophicated research and development by the big companies like Fraunhofer, Dolby, or Microsoft.
You seem highly biased toward Vorbis. If you're looking for a product of the free open source concept, why not Musepack? The next version is open source, and it won the test.
I thought we should do this test to see how different codeks can handel common problem samples at same average bittrate.
They ARE handling it at the same average bitrate. It might not be the same ABR for the samples, but it is for the overall album or collection. This test is great because it's not just testing a codec's quality, but its ability to exploit the advantages of VBR over CBR as well.
I think almost everyone here who encodes music him self with lossy codek will not use 128kbps average size without any special reason.
Depends on what you call special reason. 128kbps is a very popular choice for broadband streaming, and some of the samples proved it might be a good fit for some portables as well.
To test efficiency, which i believe is what should be the goal for a 128kps listening test, the bps should be more similar.
Efficiency? How do you define efficiency? Speed? Quality? Optimum mixture of the two? It doesn't matter what the answer is, because Roberto's test clearly states on the presentation page (http://audio.ciara.us/test/128extension/presentation.html) "Its purpose is to find what encoder outputs the best quality on bitrates around 128kbps".
I only would like to see how much closer it would be if we fine tuned it in the future, that's all.
If the test's purpose doesn't change, I don't see how your proposed changes would improve the method.
,
Radu
Edit: post split in two parts due to @#$%& forum bugs.
Vorbis is a product of the free open source concept rather than intensive and sophicated research and development by the big companies like Fraunhofer, Dolby, or Microsoft.
You seem highly biased toward Vorbis. If you're looking for a product of the free open source concept, why not Musepack? The next version is open source, and it won the test.
Biased? There is nothing biased about stating facts. The fact is Vorbis source code has been open for quite a while yet I haven't seen musepack source code in the public domain that's for all to see and contribute for as long. Whether it is open source in the next version is something in the future, not the past, thus my statement is accurate.
Anyway I sense this bias against those who make favourable comments about Vorbis. Is it really fundamentally wrong to acknowledge the work put into Vorbis that sees it outperform even the open-source and highly tuned lame mp3 encoder? Is it fundamentally wrong to acknowledge the potential of Vorbis to improve as we've seen in Garf's tunings?
Or maybe we should just throw away every coder in development and use mpc instead? I believe competition is the way to progress and we've all acknowledged (including myself) the greatness of mpc, as demonstrated in this test. mpc has defined the state-of-art and provides the benchmark for other competing coders to emulate and attempt to surpass.
I thought we should do this test to see how different codeks can handel common problem samples at same average bittrate.
They ARE handling it at the same average bitrate. It might not be the same ABR for the samples, but it is for the overall album or collection. This test is great because it's not just testing a codec's quality, but its ability to exploit the advantages of VBR over CBR as well.
Is there a big differece if we encode something with ABR to 128kbps and encode with VBR that same sample in to same size? Should'nt that VBR still be at least as good as ABR encoded sample at least if encoder is same?
I think almost everyone here who encodes music him self with lossy codek will not use 128kbps average size without any special reason.
Depends on what you call special reason. 128kbps is a very popular choice for broadband streaming, and some of the samples proved it might be a good fit for some portables as well.
But does ABR or VBR work well in streaming? I was under that impression that CBR is only choise for streaming? And i guess that we were'nt testing samples for streaming.
And i ment with that special reason for situation where size matters and quality isn't so high priority.
rjamorim: I'm sorry roberto, but i had to extract my frustrated opinnions from this subject. I mean no harm to you. Overall this test was werry well made and handelled.
@rjamorim
Thank you again for the test! I'd appreciate tests like that in the future!
Kind regards, fileman.
But does ABR or VBR work well in streaming?
With buffering (which most steaming audio progs use), and a small safety margin for bandwidth, ABR should work for streaming (VBR would be iffy), assuming the range used isn't TOO big.
Uses for 128kbps are actually pretty common. Lots of people have flash-based mp3 players with limited space, and 128kbps VBR is the right tradeoff level for a lot of people. These sorts of players are tending towards getting embedded in everything (cell phones, PDAs, probably watches before too long).
Also, right now, almost all the on-line music retailers (Apple's iTunes, eMusic, MusicNow, etc.) use 128kbps. Their claim that it's CD quality is obviously a joke (I'd call it fraud), but that doesn't prevent them from using that bitrate. I know everybody here would like mpc --standard or some such, but until that happens (never!) people who want to buy music online are stuck with 128kbps. I'd imagine they'd be interested in what performs the best at that bitrate.
I was thinking about the variability in bitrate, and realised that the people questioning the use of bitrates for the sample other than 128kbps aren't going far enough. What we really need to do is cut each sample into 1 second intervals, and make sure that each of those is exactly 128kbs. That should remove the variability.
No, wait, slice each one into 0.1 second intervals, and make sure that each is exactly 12.8kbs.
no, wait, slice each one into 0.01 second intervals.....
Cheers, Paul
(edit:spelling)
Is there a big differece if we encode something with ABR to 128kbps and encode with VBR that same sample in to same size? Should'nt that VBR still be at least as good as ABR encoded sample at least if encoder is same?
First: How do you expect me to encode something to ABR in Musepack?
If you didn't notice yet, WMA Pro, AAC and Lame are already in ABR mode.
Second: Let's take Vorbis, that is the only format in the test with ABR and an usable VBR mode: How can you be sure that the ABR mode will be at least as good as VBR? Can you imagine the flood of complaints from vorbis zealots if I use anything but the best setting?
Again (hopefully for the last time): This test is about quality, not bitrate, encoding speed, decoding speed, encoder price.. (when commercial). These other factors are of no interest to the final rating. Therefore, I chose the best quality setting for each format that outputs 128kbps on a large average (the quality setting test thread).
... I chose the best quality setting for each format that outputs 128kbps on a large average (the quality setting test thread).
The truth, only the truth and nothing but the truth...
I fail to see why it is so hard to understand for some people...
I agree 100% (or more ) with the methodology, the settings chosen, the procedure to choose them, and everything in the test.
Great test, very good work, and my most honest congratulations, hoping they will help to get you encouraged to go on performing this sort of tests.
People: read the thread in which the quality settings were chosen! All of it! You should be use to reading long threads by now if you are in HA!
How you've not responded to the 1000th complaint without resorting to all out apocalyptic warfare I don't know, you are a man of infinite patience Roberto
I haven't really been involved with the test at all, yet I've been pissed off at people for not taking time out to read how the test was designed.. . aaaagh!
Go RJAmorim!
I haven't really been involved with the test at all, yet I've been pissed off at people for not taking time out to read how the test was designed.. . aaaagh!
Not to mention that these people kept silent when Roberto welcomed criticism and advice BEFORE starting the test. doh!
Perhaps in the next test, all the test samples should be 128kbps more or less exactly, by way of adjusting -q levels for the VBR codecs. More work for roberto, no doubt, but I'm sure this would be an arrangment which more people would be happy with in terms of 'fairness'.
My conclusion for this test would simply be that this test confirms that significant improvements have been made in audio compression since MP3, despite those people claiming that "MP3 should be good enough, we should all continue using it and ignore all those incompatible new formats". MPC, Vorbis, WMAPro and AAC have shown to be significantly better than MP3. Between themselves, there are none which are significantly better than the other, so your choice will probably depend on whether you are an OSS advocate, M$-lover, MPC-diehard or Mpeg standards advocate. Or something like that.
Perhaps in the next test, all the test samples should be 128kbps more or less exactly, by way of adjusting -q levels for the VBR codecs. More work for roberto, no doubt, but I'm sure this would be an arrangment which more people would be happy with in terms of 'fairness'.
This way is more fair, I for one WOULD complain if the test was done the way you purpose.
More work for roberto, no doubt, but I'm sure this would be an arrangment which more people would be happy with in terms of 'fairness'.
"More people" ? No, this will satisfy people who doesn't understand VBR principle in a practical point of vue, and made angree (and probably boycott) all the others.
Other thing to do : use non-difficult sample, in order to avoid VBR encoder's bitrate inflation. Then, mpc, vorbis... will probably reach their statistical average bitrate, without any tweaking. But I'm not sure that many pepole will be able to ABX one of this encoding. As you said, and I'm agree with you, new generation of encoder are really better than the best mp3 one.
Roberto, I would just like to say that this was an excellently run test and I consider its results very credible, great work! I look forward to participating in the next test you put togther (tentative planning it to be all mp3, are you not?). The only thing I would have liked to see in the results would be some sort of transparency data, like "x-number of users found this sample with this encoder totally transparent".
regards,
-Tyler
I look forward to participating in the next test you put togther (tentative planning it to be all mp3, are you not?).
Thanks
Actually, the MP3 codecs test is on low priority now. I already started planning it (I'm thinking of comparing Lame, MP3pro, MP3enc, FhG fastenc and Radium, with Blade as anchor again), but it's really too early to discuss that now.
The next test will be at 64kbps. I plan to compare Vorbis, HE AAC, MP3pro, WMA pro and Real Audio Cook, using 7kHz and 3.5kHz lowpass as anchors. If all goes as planned, I'll start a calling for samples next week and the test starts on September 3rd (with a pre-test thread starting a week before. And this time I expect everyone to participate :-P)
After that, I plan to test several vocoders, with the help of JMValin. The preliminary competitors would be Speex, MPEG4 CELP, Acelp.net (As implemented in Windows Media encoder), GSM and g729. I don't know if such test needs an anchor, if not, I might throw in another encoder, like VoxWare or Qualcomm PureVoice.
The vocodec test will probably be separated in two perts, wideband and narrowband.
Regards;
Me.
@Guruboolez: Fortunately, for the next test, I'll be able to use non-difficult samples, since it's at 64kbps. The only "problem" sample I would keep is Waiting, since that's one of the most interesting samples I've seen - it doesn't look at all like a problem sample, but encoders have a hard time with it (check the aac test, for example)
Regards;
Roberto.
At some point, we probably need to split off for pre 64-kbit/s topics.
Regarding ABC/Hr, it probably needs to be recompiled to change the scale descriptions to better suit a lower bitrate test: i.e., "excellent, good, fair, poor, etc", instead of "imperceptible, perceptible but not annoying, etc"
I'm working right now to add an obscuring feature to hide the contents of the config and results files, but I don't know if that will be ready by the time I take off for a 3 week vacation.
ff123
After that, I plan to test several vocoders, with the help of JMValin. The preliminary competitors would be Speex, MPEG4 CELP, Acelp.net (As implemented in Windows Media encoder), GSM and g729. I don't know if such test needs an anchor, if not, I might throw in another encoder, like VoxWare or Qualcomm PureVoice.
This sounds cool. I have'nt try any speech codek before, exept almost daily use of GSM...
For those that think I'm wrong:
http://www.googlism.com/index.htm?ism=rjamorim&type=1 (http://www.googlism.com/index.htm?ism=rjamorim&type=1)
:-P
Interesting.
Regarding WMA9Pro "1.0": In near future there will be an update to WM9, but it is unclear when (there's a possibility MS might ditch the plan and wait for Longhorn) and if it will just affect the player or also the codecs.
--edit: typos and grammer
The next test will be at 64kbps. I plan to compare Vorbis, HE AAC, MP3pro, WMA pro and Real Audio Cook
i am really interested in a 64kbps test but wouldnt it be better to wait a little bit more as
- nero he-aac is new and probably needs more tuning (next release on 31 august)
- monty already plans to tune vorbis for low bitrates (btw the next official vorbis release will be on 1st september -> 1.0.1 only bugfixes)
and i think you meant normal wma9 as the pro codec cant be used below 128 afaik
i am really interested in a 64kbps test but wouldnt it be better to wait a little bit more as
- nero he-aac is new and probably needs more tuning (next release on 31 august)
- monty already plans to tune vorbis for low bitrates (btw the next official vorbis release will be on 1st september -> 1.0.1 only bugfixes)
Well, the test will start after Nero's next release (it's planed for Sept 3rd), so I think it will be available anyway.
And how long will I have to wait until Vorbis 1.1 is released? If it follows Xiph's release schedule, this test will start around October 6th, 2004 :B
(It took more than an year for a minon-minor new version (1.0.1), how long will it take for a minor version (1.1)?)
Do you have any info from Monty if he at least started working on tuning 1.1 already? If he didn't, I see no point waiting months for only one codec.
and i think you meant normal wma9 as the pro codec cant be used below 128 afaik
Oh, yeah? I sincerely don't know.
Can someone with Windows Media Encoder 9 check out if you can get a 64kbps two pass VBR encode out of it in WMA pro?
I could chack it, but it's already 5:15AM here :-/
Thanks for the info, Bond.
Regards;
Roberto.
OMG! I must stop that! I want to take a break, not start talking about the 64kbps test already.
people: Nevermind the request about Windows Media. I'll have time to verify that later, there are still almost 4 weeks until the test starts.
Please, let's not discuss the 64kbps test here.
The test will start to be discussed about 10 days before the test start.
Thanks for your comprehension.
R.
and i think you meant normal wma9 as the pro codec cant be used below 128 afaik
Oh, yeah? I sincerely don't know.
Can someone with Windows Media Encoder 9 check out if you can get a 64kbps two pass VBR encode out of it in WMA pro?
Nope. CBR minimum bitrate is 128kbit. The only way to get it below that is using the Quality VBR mode, Q10 and Q25 give bitrates below 128kbit, maybe even Q50, but the bitrates these modes spit out depend on the input you feed the codec.
this is an interesting test and I'm glad it was (and will continue to be) done, but as a scientist I have to ask : What makes this a double-blind test? shouldn't the samples be compiled on the fly for each person from a large bank of random music so as to eliminate subconscious prejudice from those selecting the samples?
After all the principal of the double-blind is that neither the experimentee or the experimenter knows what they are being subjected to until the results are in... blind tests have demonstrated homeopathy works, whereas double blind ones then dismissed the claim...
EDIT : not that I think that it would make much of difference in this case, as the samples are supposed to be deliberately selected to not highlight any particular flaw...
this is an interesting test and I'm glad it was (and will continue to be) done, but as a scientist I have to ask : What makes this a double-blind test? shouldn't the samples be compiled on the fly for each person from a large bank of random music so as to eliminate subconscious prejudice from those selecting the samples?
It is double-blind in the sense that neither the administrator nor the listener knows which codec is being listened to at any given time (unless there is only one codec being compared).
If random samples were chosen each time, there would be no way to make a group comparison, since preferences vary by sample as well as by person.
However, you have a point about how the samples are selected in the first place. I selected the original group of samples for the 64 kbit/s test, after calling for people to send in short clips of music that they liked to listen to. This process was definitely not random. I culled according to my own judgment. The idea was to obtain a mix of genres, with vocals both male and female, a variety of acoustic instruments. I chose what I personally thought was interesting-sounding music (eg., I chose not to include several Japanese pop-music selections).
For the 128 kbit/s tests, Roberto substituted in a couple of "problem" samples, with lots of transients, known to give codecs trouble.
How this might have biased the test is unknown, but the caveat is clearly made that the results of this test are valid for this particular mix of music and the particular group of people who listened to it.
After all the principal of the double-blind is that neither the experimentee or the experimenter knows what they are being subjected to until the results are in... blind tests have demonstrated homeopathy works, whereas double blind ones then dismissed the claim...
This is not so different from a drug test, in which the drug under test is known, but whether the drug or the placebo is being administered to any given person is not known.
ff123
Well, as i said I don't think it's too important in this case, as the testing is very specifically targeted, and the results are just as meaningful... what i'm suggesting would probably be a great del more difficult to implement, for only marginal gain.
after all, using targeted samples has it's place too.
I'm just suggesting a broader, possibly more meaningful test criterion, in that the purpose of the double-blind test is to erase any prejudice on behalf of the tester, as well as the testee. randomising the selection process, of course, is an integral part of this. Otherwise it's more like the less thourough Pepsi Challenge style blind test.
re: the drug test example. I feel providing truly random audio samples would be like the tester not knowing who gets what drug in double blind test. rather than matching the patient to the test drug or the placebo. true the test is blind, but if the testing is unknowingly potentially tailored for the "best" result then inevitably the trial is tainted.
Meanwhile any one aberrant result caused by the random sampling should be overruled by the mean result and is, at least more truly random.
The test IS double-blind - you don't know which sample you're listening to, you can only discern between coded and not coded.
Double-blind testing was invented to remove some nonverbal information from uncovering what the sample is.
There is no such problem with computers.
First of all, I'd like to apologise for the way I've put things in my last posts, as I think I'm coming across as a jerk. but hey, that's the 'net. it's easy to offend, unless you always agree with everyone...
AstralStorm, my main point is that the interaction between the testers and the subjects is in the audio selection. if the music they encode is very randomly chosen on a user by user basis (say a random 30 secs out of an hour, chosen and encoded on-demand) the results might be different.
I don't think they would be. but they might.
imagine: "hey, our test says J-pop in 24kbs mp3 sounds the same as raw!"
I wasn't annoyed by your post. Just tried to dispell some FUD. Oh well...
Lots of people would find hardly any artifacts in normal (easy) track encoded by recent codec at this bitrate.
Even best would have problems.
This would diminish the differences between them.
rjamorim, great work! Three thumbs up!
I think some ppl take these test a bit too seriously. The main point came out clearly: mp3 is outdated with 128 kbps. It is up to your software and hardware choices what to use of the winners.
Hopefully all the mobile device developers also read your test and move on from mp3 Can't wait the 64 kpbs test results.
Best wishes,
Jore
Wma in all its confusing glory:
Basically everything wma that is not pro, lossless or voice (even WMA v9 standard) can be played by all WMA codecs including the first v2 codec.
To play pro, lossless and voice install the wma v9 codecs (which will play everything).
So if I have a Nomad Jukebox that supports wma it will NOT play file encoded (lossy or lossless) with the Pro encoder?
Also, what is the real world scenario that the 64kbps test is evaluating? I would think with all the "golden ears" here, noone would use that low of a bitrate. Is it to evaluate codecs for the purpose of streaming content?
So if I have a Nomad Jukebox that supports wma it will NOT play file encoded (lossy or lossless) with the Pro encoder?
Right, at least until creative releases a firmware update (if ever)
Also, what is the real world scenario that the 64kbps test is evaluating? I would think with all the "golden ears" here, noone would use that low of a bitrate. Is it to evaluate codecs for the purpose of streaming content?
That, and flash players, and probably because the other bitrate ranges either have already been tested (128) or are untestable (160+)
For the WMA Codecs on protables, the source code comes from Microsoft themselves. I have heard they have yet to release a PRO or lossless version for portables to the various manufacturers.
and probably because the other bitrate ranges either have already been tested (128) or are untestable (160+)
Is this because at 160 or above some codecs are mostly transparent? I would think that the presence of HD based players for mp3, wma (though not pro), ogg (Rio Karma), aac, and the whispers of the possibility of mpc, would make more people interested in seeing the results at 160 and 192. I, for example, have been using lossless while waiting to make a decision on my lossy codec choice. The relative performance of these codec at higer bitrates would really help me with that decision as well as the decision of what bitrate to encode files for portable use. It would help even if some of them tie due to transparency. For example if three codecs get 5s at 160 I can quit worrying and choose whichever portable I like and encode at 160. Am I missing something?
And thanks guys for the wma clarification.
Great test Roberto.
I think from my point of view, I'd prefer an encoder that doesn't trip up very badly very often, even if its average score were a little lower.
Now, WMA Pro tripped up badly once. Perhaps it was bad luck and with other samples another codec would trip up, so statistical information isn't perfect.
However, I tabulated the mean scores (read from your graphs) and estimated the standard deviation.
Assuming all test samples are similarly distributed in terms of encoder variability, and assuming a "normal" or "gaussian" distribution, the average minus one sigma and average minus two sigma give a guide to the worst behaviour we're likely to see:
Track AAC Lame MPC Vorbis WMAPro Blade
41_30sec 4.36 3.3 4.33 4.2 3.97 1.4
ATrain 4.41 3.78 4.37 4.17 4.48 3.05
Bachpsic 4.5 3.41 4.66 4.51 4.8 2.9
Blackwat 4.62 3.92 4.71 4.38 4.56 2.18
death2 4.35 3.62 4.67 4.18 2.7 1.27
flooress 4.08 3.68 4.52 4.57 4.25 1.7
layla 4.15 3.59 4.4 4.24 4.45 1.83
macabre 4.59 4.06 4.55 4.54 4.86 3.16
midnight 4.56 3.42 4.43 4.26 4.38 2.39
thear1 4.69 4.16 4.48 4.11 4.44 2.41
thesourc 4.61 4.33 4.62 4.43 4.87 2.36
waiting 4.13 2.71 4.35 3.78 3.88 1.99
AvgScore 4.42 3.67 4.51 4.28 4.30 2.22
Std.Dev 0.21 0.44 0.13 0.22 0.59 0.62
-1 sigma 4.21 3.23 4.37 4.06 3.71 1.60
-2 sigma 4.00 2.79 4.24 3.84 3.11 0.99
-3 sigma 3.79 2.35 4.10 3.61 2.52 0.37
-1 sigma pt = 84.13% p(new sample < this value)
-2 sigma pt = 97.72% p(new sample < this value)
-3 sigma pt = 99.87% p(new sample < this value)
sd/sqrt12 0.06 0.13 0.04 0.06 0.17 0.18
errorbar 0.12 0.25 0.08 0.13 0.34 0.36
The probabilities at the end refer to the inverse normal distribution and the chances of getting a value worse than the -1 sigma point etc. if you chose a new sample at random and had the same listeners test it.
This is the result at the average minus 2-sigma point:
(http://members.lycos.co.uk/bhafool1/avg-2sigma128.png)
The errorbar line is based on the estimated error in the mean score, which I'd use to find the best rated codec overall on a mean score basis = 2*(Std Dev / Sqrt(12))
(http://members.lycos.co.uk/bhafool1/avgscore128.png)
Just my thoughts. Many thanks to those who tested (I didn't have time, or probably the artifact training to join in)
By my criterion, of not failing badly, MPC wins over AAC, Vorbis, WMAPro, LAME, Blade.
(Edit: Note, I posted the wrong image originally, so please refresh if the top graph doesn't match the scores or this order)
DickD
P.S. Hmm, I wonder if WMAPro did badly only because it was using 2 passes to aim at 128 kbps for the specific short sample tested. Perhaps it's fairer to use it in a one-pass mode that averages at 128 kbps over many albums.
The errorbar line is based on the estimated error in the mean score, which I'd use to find the best rated codec overall on a mean score basis = 2*(Std Dev / Sqrt(12))
The graph showing the standard error of the mean is an interesting one, as it shows the variability of quality across the codecs. I think that's a useful graph which should be included in future test analyses.
Edit: your graph shows twice the standard error of the mean; I think a more conventional graph would show just the standard error. Still, it's a good graph.
P.S. Hmm, I wonder if WMAPro did badly only because it was using 2 passes to aim at 128 kbps for the specific short sample tested. Perhaps it's fairer to use it in a one-pass mode that averages at 128 kbps over many albums.
1-pass VBR was way too variable in average bitrates across albums. See the bitrate thread for this test. So we chucked it in favor of 2-pass VBR.
ff123
Is this because at 160 or above some codecs are mostly transparent? I would think that the presence of HD based players for mp3, wma (though not pro), ogg (Rio Karma), aac, and the whispers of the possibility of mpc, would make more people interested in seeing the results at 160 and 192. I, for example, have been using lossless while waiting to make a decision on my lossy codec choice. The relative performance of these codec at higer bitrates would really help me with that decision as well as the decision of what bitrate to encode files for portable use. It would help even if some of them tie due to transparency. For example if three codecs get 5s at 160 I can quit worrying and choose whichever portable I like and encode at 160. Am I missing something?
Well, you can try to extrapolate the results of the 128kbps test. At 128kbps, MPC, AAC, WMA and Ogg ended up with average 4.35 points. At 160, I'm pretty sure all of them would reach very close to 5, maybe with some results from golden ears making the scores go down a little.
Unless I use very problematic samples, but then, it won't be really representative of several musical styles.
Finally, there's the point that it would be a VBR test. And I only have the courage to conduce one more VBR test (the 64kbps test)