Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: 128kbps Extension Test - FINISHED (Read 73391 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

128kbps Extension Test - FINISHED

Reply #50 –
I just did my own abx test using the same codecs as in Roberto's test. I used several pieces of music that I am very familiar with. I could NOT abx any of the
encoded files vs the .ape file used as the reference for the test (I used foobar 2000
version .667 for the abx test) except for the lame encoded file, and I could BARELY abx that. This tells me my almost 45 year old ears are past their prime, and I could use any codec for portable use. I still like the idea of storing all my music as lossless though, so I can encode to anything on the fly for portable use (mp4 for ipod, wma9 pro for a pocket pc etc.) . I use dbpoweramp to go from .ape to lossy. It has been working great.

Thank you for the informative test.  Since I have proved to myself my ears are crap, I can now sleep beter at night.
you will make mp3's for compatibility reasons.

128kbps Extension Test - FINISHED

Reply #51 –
OK, as promised, I've compiled all the results into one big spreadsheet.  You can download it in zipped XML (OO's native format) here: 128kbps_results.sxc

Wait?  Your spreadsheet won't take that?  Ok, you can get an .xls here (grumble, grumble): 128kbps_results.xls
Let me know if you have problems with that.  I don't know how good OpenOffice's .xls export is.

Feel free to do whatever you want to it.

I've included two sections.  On the left are all the individual results.  It has each listener's individual ratings, and how each codec ranks (i.e. 1st, 2nd, 3rd) for each sample.  At the bottom of each sample, there is the average and standard deviation for that sample for each codec.  Way down at the bottom are the averages and standard deviations for all the samples.

On the right are all the per user averages.  You can see what the average user prefers, and compare that to each individual user.  Over here, I've got a couple cells that I left filled in that probably aren't that meaningful (like the standard deviation of all the listeners' standard deviations), so take all the numbers well salted.

Quote
Dude, to tell you the truth, I keep wondering if it's not better to keep these results out of /. :-/

Ya, I think you're right.  Anymore, slashdotters are hosers, eh.  I'm sure they'd all point out the 4.25, 4,25 issue with vorbis.    I wish there were a general-purpose geek forum that didn't suck.

Quote
Since I have proved to myself my ears are crap, I can now sleep beter at night.

Don't be so sure.  I couldn't pick out any at first either (even blade on some!) but after a little training I could ABX the majority.  Of course, now all these old 128kbps mp3s are completely unlistenable.

Also, I believe many of the samples chosen were ones known to cause trouble for lossy codecs, so they're more likely to reveal problems than the average piece of music.
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

128kbps Extension Test - FINISHED

Reply #52 –
Thanks for the vote of confidence phong 

I was too busy to do the test itself that Roberto setup, maybe I can download his samples and try the abx for the codecs myself. I still think my hearing is at a disadvantage compared to the 20 something people that heavily populate this forum.

Many spoke of needed hardware support for WMA9 Pro. I bought my wife's Pocket PC back in March 2002, and it has WMP version 8.5 installed on it, which plays back the one pass and two pass VBR wma files just fine (must be a beta version that was installed on that version of Pocket PC OS) so I imagine all the new Pocket PC's should have no trouble with WMA9 Pro VBR files. I can see needing to update portable cd players that decoded wma files, since most of those are prolly based on the older codec.
you will make mp3's for compatibility reasons.

128kbps Extension Test - FINISHED

Reply #53 –
Quote
OK, as promised, I've compiled all the results into one big spreadsheet.  You can download it in zipped XML (OO's native format) here: 128kbps_results.sxc

Wait?  Your spreadsheet won't take that?  Ok, you can get an .xls here (grumble, grumble): 128kbps_results.xls
Let me know if you have problems with that.  I don't know how good OpenOffice's .xls export is.

Looks like there's some sort of formula error when it calculated how many samples I had listened to (I listened to all 12, but it only showed 11).  I'm looking at the .xls version.

ff123

128kbps Extension Test - FINISHED

Reply #54 –
Interesting factoid:

Listener annoyance ranking (who tended to rate the lowest):

proxima        2.55  (7 samples)
Guruboolez    2.71  (12 samples)
gecko          2.73  (12 samples)
kl33per          2.77  (3 samples)
dimkovic        2.92    (1 sample)

ff123

128kbps Extension Test - FINISHED

Reply #55 –
Hmm, I double checked, and I get 12 for you for both versions.  It could be OO's fault.  The xls file I have there is exported for Excel 97/2000/XP.  I just made a version for Excel 95 that you should also be able to import (it may work better): 128kbps_results_95.xls

Did the other numbers look right?  By my count, 8 people should have all 12.  What shows as the formula for that cell?  (Should be something like "=COUNTIF(B1:B302;Q36)")

Edit: Also, does it show your name identically for all samples (i.e. ff123, lower case, no leading or trailing spaces)?
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

128kbps Extension Test - FINISHED

Reply #56 –
Quote
Hmm, I double checked, and I get 12 for you for both versions.  It could be OO's fault.  The xls file I have there is exported for Excel 97/2000/XP.  I just made a version for Excel 95 that you should also be able to import (it may work better): 128kbps_results_95.xls

Did the other numbers look right?  By my count, 8 people should have all 12.  What shows as the formula for that cell?   (Should be something like "=COUNTIF(B1:B302;Q36)")

Oops, my fault.  I munged it up by sorting.

ff123

128kbps Extension Test - FINISHED

Reply #57 –
Quote
Man, LAME really is dated at this bitrate.  Hopefully no one misinterprets this test and decides from it  that all these codecs perform equally the same at higher bitrates.  Please remember that this is only at 128kbps.  alt-preset standard, vorbis gt3, musepack standard, etc will be tested later, if that would even be a valid or necessary test.  Taking from the difficulty that this test has shown, it would be no wonder even harder to test the higher bitrates.

Even though I know that it is incorrect to make blanket statements, I'm sure that because of this test we will see them.  If it wasn't for compatibility, I think you would see people leaving LAME in droves now.  While I guess it might be plausible to make judgements from this data that LAME is dated, that other codecs have seen greater development as of late, and then as such decide to use a different codec for all your encoding needs, I don't think it would be a good idea to directly link this test to overall sound quality.

I agree with these sentiments. Since we know that lame --aps is transparent in nearly all cases it seems a bit harsh to say that this test proves that lame is crap. Certainly it is inferior at 128kbps, but since most of us HA-enlightened mp3 users encode with --aps is this really relevant? We already knew that 128kbps mp3 was not that good. With the availability of cheaper storage and faster internet access, why the imperative to have 128kbps files? I totally agree that if there was hardware compatability for other formats then lame mp3 would be redundant, but the main reason for this would be for gapless playback rather than sound quality.

I'm not trying to slag off the tests here. The work everyone put in was fantastic, and the results are important and useful. Congratulations to all involved!    I just don't want the wrong conclusions to be drawn. (I am imagining a lot of uninformed people transcoding their --aps mp3s to mpc to get better quality.  )

.dd.

128kbps Extension Test - FINISHED

Reply #58 –
Quote
Oops, my fault. I munged it up by sorting.

No, my fault.  I forgot that you can use $'s in ranges to make them behave.  I've uploaded new versions (same urls).  Should be able to sort or do other nifty operations now without causing havoc.
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

128kbps Extension Test - FINISHED

Reply #59 –
Quote
Listener annoyance ranking (who tended to rate the lowest):

proxima 2.55 (7 samples)
Guruboolez 2.71 (12 samples)
gecko 2.73 (12 samples)
kl33per 2.77 (3 samples)
dimkovic 2.92 (1 sample)


On looking upon how other people have marked, I think I've marked to hard.  Maybe there needs to be more solid criteria then the "Imperceptible, Perceptible but Not Annoying, Slighly Annoying, Annoying, Very Annoying".  After all, this sort of marking scheme is very subjective.  What's annoying to one person might not be annoying to another, even if they both here the same flaws in the file.

BTW, Roberto, nice test, thanks for holding it.  Also my thanks to all those who participated.

128kbps Extension Test - FINISHED

Reply #60 –
I'm playing with the data right now.  It's kind of interesting to split the results in half after sorting by average rating.  The low scorers (the most sensitive listeners) spread the scores out more.  When the high scorers results are averaged in, the winning codec(s) sometimes get squashed back into the group.  This appears to be because the less sensitive listeners aren't really able to distinguish at the highest levels, so they aren't giving the best codec(s) their proper due.

For a high bitrate test, this has implications:  only the most sensitive listeners should participate because the less sensitive listeners will "squash" the results at the top.  Only a dozen people are really needed, but they need to be able to hear the subtlest flaws.

I'll try to post the two different types of graphs later.

ff123

128kbps Extension Test - FINISHED

Reply #61 –
Quote
I've got a perl script in the works right now to convert the raw data to .csv so that it can be played with in a spreadsheet.  I'll post that later tonight.

I uploaded all the test results in a nice .zip package, so that you don't have to download one by one.

Also inside are the tables ready to be fed to friedman.exe processing (folder \Friedman), and the Anova scores for each sample and for all samples (folder \Scores-Anova)

The package is here:
http://rarewares.hydrogenaudio.org/rja/Results.zip

Quote
I was one who flagged a concern about the bitrates, and I am genuinely sorry for any concern or constenation that I may have caused to others at HA.


Man, it wasn't your fault at all! :B

You actually came politely to me and raised a genuine concern.

What I can't stand is people mailing me with something like "You are using VBR, your test suxxxxxorz!" (not that bad, but close)

128kbps Extension Test - FINISHED

Reply #62 –
Geh, I took too long to upload the results .zip. Sorry phong. :-/

128kbps Extension Test - FINISHED

Reply #63 –
http://ff123.net/export/128exten_split.html

On individual samples, the split between high scorers and low scorers seems to make a difference, but the overall results are similar.  There are no new significant differences, but MPC vs. Ogg comes closer (about 89% confidence for low scorers that mpc is better than ogg).

ff123

128kbps Extension Test - FINISHED

Reply #64 –
Quote
Geh, I took too long to upload the results .zip. Sorry phong. :-/

No worries.  Perl, bash and wget automated the task by combining their powers, as if they were five giant robot cats coming together to form a super-robot with a big sword for slashing at other robots, except there were only three of them and they don't wear spandex or helmets and...

I'll incorporate those Anova scores as soon as I get home and figure out what they mean. 
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

128kbps Extension Test - FINISHED

Reply #65 –
Quote
delete in_wm.dll and add wma to the extension list in in_dshow's configuration


Hey Cool it works ThX

But a Official or a Plugin Support for Winamp would be Cool 2.

128kbps Extension Test - FINISHED

Reply #66 –
Quote
On looking upon how other people have marked, I think I've marked too hard.  Maybe there needs to be more solid criteria then the "Imperceptible, Perceptible but Not Annoying, Slighly Annoying, Annoying, Very Annoying".  After all, this sort of marking scheme is very subjective.  What's annoying to one person might not be annoying to another, even if they both here the same flaws in the file.

And what exactly would you propose as an alternative marking scheme? This scheme, although admittedly subjective, serves its purpose just right, because quality is precisely a subjective term. Of course, what one person finds annoying might not be so for another, but we are not looking for an absolute scale because there can´t be any. What we are trying to do is to find the one that sounds best for the majority of people, and if some people are more "sensitive" or "exigent", that of course will lower the average. That, however, doesn´t matter very much, since what we are trying to find are the relative ranking among the codecs. You could delete all numerical values and leave only the graph and it would still tell what it needs to tell.
As a matter of fact, I can´t even think of a more objective marking scheme that makes sense.  5: 0-100 millikleepers of difference with the original, 4: 101-200 millikleepers of difference with the original...? 

-Dologan

128kbps Extension Test - FINISHED

Reply #67 –
This topic's a whole othrer thread, but anyway.  I don't how it should be changed, just that maybe they're should be more solid criteria for each level.  It's not that I think I thought the qualtity of these files was any less then the next person, just that I had no reference on how to score this (apart from the aforementioned five written levels).  Anyway, it was just a thought, and I'm not really sure what to do about it, it just seems like I was to critical of the codecs, that's all.

128kbps Extension Test - FINISHED

Reply #68 –
Quote
http://ff123.net/export/128exten_split.html

this could just be my spacial perception, but on the (Overall, high srocers) Ogg seems to win over WMA. so what are those graphs? what does it mean (in laymans terms if possible  )?
The Plan Within Plans

128kbps Extension Test - FINISHED

Reply #69 –
I just had an idea for another test.  Compare the major codecs at the broadly accepted high (but not stupid) quality level, eg lame -aps, ogg -q6, mpc -xtreme etc.

Two reasons for doing this - first, it would be an interesting comparison of the various recommended settings.  But more important, imagine the flamefest!  The amount of crap Roberto got just because VBR is, well, variable, would fade into insignificance!



Cheers, Paul

128kbps Extension Test - FINISHED

Reply #70 –
Quote
I just had an idea for another test. Compare the major codecs at the broadly accepted high (but not stupid) quality level, eg lame -aps, ogg -q6, mpc -xtreme etc.

Yeh, I've thought about this to, but as mentioned, these codecs are pretty close (if not actually) transparent at these bitrates, thus making the ABXing of these files much harder.

Quote
Two reasons for doing this - first, it would be an interesting comparison of the various recommended settings. But more important, imagine the flamefest! The amount of crap Roberto got just because VBR is, well, variable, would fade into insignificance!

Poor Roberto, I pity him, he goes and sets up this huge test with 12 samples, 6 codecs, website and all, and then gets flamed for using codecs in their optimal (or only) mode.

128kbps Extension Test - FINISHED

Reply #71 –
Quote
Quote
http://ff123.net/export/128exten_split.html

this could just be my spacial perception, but on the (Overall, high srocers) Ogg seems to win over WMA. so what are those graphs? what does it mean (in laymans terms if possible  )?

In statistical terms, you can only say that one codec is significantly better than another with 95% confidence if the bottom error bar of the better codec does not overlap with the top error bar of the worse codec.  Otherwise, they are considered to be the same quality.

So in the overall high-scorers graph, ogg is not significantly better than wma9pro.

ff123

128kbps Extension Test - FINISHED

Reply #72 –
I couldn't help noticing that at both samples that could be classified as classical (Bach and Saint-Saens), WMA9Pro won with pretty good scores (~0.3 above the next "hardwareable" codec (let's face it - MPC will never get widespread hardware support - if any at all). I wonder whether WMA9Pro has such a good performance with all classical music in general, since if so, it might become my format of choice for the classical collection on my portable . Unfortunately I am not able to test that by myself at the moment, since I am on a trip and all my classical is encoded, and trascoding is something I'd like to avoid for this kind of judgment.

~Dologan

128kbps Extension Test - FINISHED

Reply #73 –
Quote
Compare the major codecs at the broadly accepted high (but not stupid) quality level, eg lame -aps, ogg -q6, mpc -xtreme etc.

I think Musepack at -standard is competent enough.
Testing -q6 and above would almost be pointless.

128kbps Extension Test - FINISHED

Reply #74 –
Quote
I couldn't help noticing that at both samples that could be classified as classical (Bach and Saint-Saens), WMA9Pro won with pretty good scores (~0.3 above the next "hardwareable" codec (let's face it - MPC will never get widespread hardware support - if any at all). I wonder whether WMA9Pro has such a good performance with all classical music in general, since if so, it might become my format of choice for the classical collection on my portable . Unfortunately I am not able to test that by myself at the moment, since I am on a trip and all my classical is encoded, and trascoding is something I'd like to avoid for this kind of judgment.

~Dologan

I noticed it too. I'm looking for an encoder at low-mid bitrate, for storing some discs on my new portable computer, very limited in space. I tried to found good settings/codec at 70-90 kbps, but too many artifacts remained. HE-AAC isn't very efficient at the moment, and can't be seriously considered (nevertheless, i hope fast improvments). Therefore, 128 kbps seems to be a good compromise.
Two potential challengers here : AAC MP4 (QT) and WMA9pro. Quicktime isn't suitable for batch encoding, and tagging with WMA9Pro is something impossible with foobar2000. That's really annoying !
Compromise : Nero MP4 VBR --streaming, less reliable in my opinion than QT 128, but good quality, and easy to use (transcoding forom lossless is easy). Vorbis doesn't really interest me : too noisy for my taste, and harpsichord isn't well-reproduced. MPC ? Hmm... I wasn't convinced for classical music and --radio. And my baroque taste will conduct to high bitrate (close to 150 kbps)

I'm gonna play the next time with WMApro, with VBR and Two-Pass, at mid-bitrate. I didn't have time and will enough for serious comparison, but for the moment, I didn't noticed something really annoying with this encoder, with all tracks and samples I tested. I have to look carefuly on loud music : wma standard often destroyed the sound, and I suspect PRO encoder to share the same flaw (less pronounced) : may be annoying with grand-orchestra.
Anyway, if PRO vbr encoder isn't amazing at low bitrate (try any metal sample at VBR 25-50 : it's obvious), quality seems to be more interesting for classical music (metalic coloration is the most annoying thing) ; bitrate is sometime very low.