Skip to main content

Topic: Multiformat@128kbps listening test - FINISHED (Read 160409 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Multiformat@128kbps listening test - FINISHED
Reply #25
Some comments:

1. mpc encoded debussy.wav at too low of a bitrate (98 kbit/s), apparently, because multiple people commented on a distorted sound, and its low rating on this sample (3.53) hurt it in comparison with vorbis.  Note that problem samples are not synonymous with high bitrate!  I would hope Frank could look into what's going on with mpc on this sample.

2. It's not clear that the all of the samples which didn't show significant differences (there were 4) would have benefited much with a larger listener sample size.  The Bartok_strings2.wav and OrdinaryWorld.wav samples in particular are pretty evenly rated across the board.

Roberto did a separate analysis omitting these 4 samples and the overall results were very similar to the results with all 18 samples, except that with 18 samples the confidence level increased.  So I'd say they helped out, even if individually they didn't show significant differences between codecs.

3. The absolute ratings of iTunes is remarkably stable in the tests it's been featured in (4.39, 4.42, 4.20, and 4.26 on this one), even though the tests are not strictly comparable.

4.  MPC should have been expected (and it did appear) to be slightly better this time around than the last multiformat test since its quality setting was tweaked up slightly (from 4 to 4.15).

5.  Excellent job on AoTuVb2, Ayumi and everybody else who was involved.  Seeing such a high score in the test shouldn't have been a real big surprise since those virtuoso tuning ears were rating the beta2 version at around 4.0 overall.

6.  Lame is still improving.  Good job Gabriel and [proxima]

ff123

Edit:  After checking, I see that MPC's absolute score went down from 4.51 to 4.47, so comment 4 is not consistent with what actually happened.  But then again, it's not strictly correct to compare scores on one test with scores on another.
  • Last Edit: 24 May, 2004, 04:28:06 AM by ff123

  • Raptus
  • [*][*]
Multiformat@128kbps listening test - FINISHED
Reply #26
Quote
Quote
How many results were discarded because of ranked refs?

54

Mind you that I didn't discard results that ranked the reference but on that sample pair ABXd the samples to a pval of 0.05 or less.

Ok.
Thats around 15% of the results... And for me it still doesn't feel right to take them as irrelevant for the stats...

What about all the /.ers? 
Seems they were just interested in wasting bandwidth after all 

  • Grease
  • [*]
Multiformat@128kbps listening test - FINISHED
Reply #27
I found my chanchan listening test result wrongly classified as a NewYorkCity result.


-Grease

  • rjamorim
  • [*][*][*][*][*]
Multiformat@128kbps listening test - FINISHED
Reply #28
Quote
What about all the /.ers?  
Seems they were just interested in wasting bandwidth after all 

More than 500 people downloaded the samples through bittorrent only - not counting HTTP downloads! :B

I won't ever understand these people. 
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Multiformat@128kbps listening test - FINISHED
Reply #29
Quote
Quote
Quote
How many results were discarded because of ranked refs?

54

Mind you that I didn't discard results that ranked the reference but on that sample pair ABXd the samples to a pval of 0.05 or less.

Ok.
Thats around 15% of the results... And for me it still doesn't feel right to take them as irrelevant for the stats...

What about all the /.ers? 
Seems they were just interested in wasting bandwidth after all 

Some results with ranked refs are worse than others.  Roberto showed me results from one person whose listening results I wouldn't trust at all, they were so bad (meaning lots of ranked refs).

There is always a question about how these results should be treated, and there are probably multiple ways of handling them.  The fairest and simplest way seems to be to just throw them away if you have enough results that you can afford to do that, which in this case seems to be true.

ff123

  • rjamorim
  • [*][*][*][*][*]
Multiformat@128kbps listening test - FINISHED
Reply #30
Quote
I found my chanchan listening test result wrongly classified as a NewYorkCity result.

No worries, that classification happened while uploading. I'll move it back to the correct folder later.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Multiformat@128kbps listening test - FINISHED
Reply #31
Quote
1. mpc encoded debussy.wav at too low of a bitrate (98 kbit/s), apparently, because multiple people commented on a distorted sound, and its low rating on this sample (3.53) hurt it in comparison with vorbis.  Note that problem samples are not synonymous with high bitrate!  I would hope Frank could look into what's going on with mpc on this sample.

The problem seems to be low-volume. MPC --radio have some troubles with low-volume sample, especially when there's a slight amout of noise. Debussy.wav is just an exemple amoung hundred of this problem.
Problem is shoking if playback volume is exceptionnaly high, but is probably less annoying on normal playback conditions (which explain maybe the overall relative good notation of the encoding - I expected to be lower).

Note that standard preset also suffers from this problem, but it's less critical...
  • Last Edit: 24 May, 2004, 04:43:28 AM by guruboolez

  • amano
  • [*][*][*][*]
Multiformat@128kbps listening test - FINISHED
Reply #32
Wow. That is interesting. LAME with the --athaa-sensitivity switch and aoTuv being that strong.

Thanks to all participating and - of course - to all these great codec developers and exspecially to Roberto himself!!!

  • XXX
  • [*]
  • Banned
Multiformat@128kbps listening test - FINISHED
Reply #33
This particular test should be called, "The 128 kbps test for iTunes/WMA, and the low-130 test for AC3 and LAME, and the close-to-160 test for MPC/Vorbious.

Leahy   iTunes   MPC   Vorbis   Lame   WMA   Atrac3

bitrate   128   155   149   133   128   132

Score   4.34   4.41   4.68   4.11   4.37   3.76

I am aware of the rationalization. I am aware of the overall average. But let this be a n"oh?" to those that don't and aren't.

  • amano
  • [*][*][*][*]
Multiformat@128kbps listening test - FINISHED
Reply #34
AC3??? Vorbious???

Get some sleep.

  • Lyx
  • [*][*][*][*][*]
Multiformat@128kbps listening test - FINISHED
Reply #35
maybe it would make sence to rename "iTunes" to "iTunes AAC" in the summary chart, so that people do not mistake the iTunes result with its lousy mp3-encoder?

- Lyx
I am arrogant and I can afford it because I deliver.

  • cuan
  • [*]
Multiformat@128kbps listening test - FINISHED
Reply #36
lame's result is fairly amazing. I was about to begin encoding my cd collection into iTunes aac for an iPod im about to purchase. I think ill just stick with lame now. It's level of quality combined with it's compatiblity between mp3 players is an unbeatable combination.

  • bond
  • [*][*][*][*][*]
Multiformat@128kbps listening test - FINISHED
Reply #37
rjamorim, can you plz make a zoomed "music store codecs only" chart too (aac, wma9, atrac3), i think it would be very interesting and important to have such a chart handy for showing people that when they have to choose where they should buy songs from, that not only the prices, but also the quality is very important and varries a lot

btw did i already thank you for your great test? thanks a lot!

Quote
maybe it would make sence to rename "iTunes" to "iTunes AAC" in the summary chart, so that people do not mistake the iTunes result with its lousy mp3-encoder?

yepa and maybe add "mp3" to lame too, (and maybe ogg to vorbis) at least in the final chart to exclude all possible misunderstandings
I know, that I know nothing (Socrates)

  • QuantumKnot
  • [*][*][*][*][*]
  • Developer
Multiformat@128kbps listening test - FINISHED
Reply #38
A big thank you to Roberto for his efforts in conducting this test.  Let's hope that it is not the last too 

  • SebastianG
  • [*][*][*][*][*]
  • Developer
Multiformat@128kbps listening test - FINISHED
Reply #39
Quote
This particular test should be called, "The 128 kbps test for iTunes/WMA, and the low-130 test for Atrac3 and LAME, and the close-to-160 test for MPC/Vorbis.


Yup, it's hard to compare CBR encoders with VBR encoders.
Everything you do is wrong

Usually all encders tend to produce files at around 128 kbps on an "average" sound file with the same settings. That's why I think it's ok to compare these codecs with these settings. Many test samples were chosen to be hard-to-encode (weren't they?). VBR encoders use higher bitrates in those complex situations. CBR encoders don't.
Bad Luck for the CBR encoders.

So... you can ask yourself: Is the choice of test samples fair ?
I don't know...

bye,
Sebastian

  • JeanLuc
  • [*][*][*][*][*]
Multiformat@128kbps listening test - FINISHED
Reply #40
Quote
This particular test should be called, "The 128 kbps test for iTunes/WMA, and the low-130 test for AC3 and LAME, and the close-to-160 test for MPC/Vorbious.

Leahy   iTunes   MPC   Vorbis   Lame   WMA   Atrac3

bitrate   128   155   149   133   128   132

Score   4.34   4.41   4.68   4.11   4.37   3.76

I am aware of the rationalization. I am aware of the overall average. But let this be a n"oh?" to those that don't and aren't.

That's why I suggested to put the bitrates into the score graphs for each sample ... so everyone can see at which average bitrate the codec's result has been obtained.
The name was Plex The Ripper, not Jack The Ripper

  • JohnV
  • [*][*][*][*][*]
  • Developer
Multiformat@128kbps listening test - FINISHED
Reply #41
Quote
lame's result is fairly amazing. I was about to begin encoding my cd collection into iTunes aac for an iPod im about to purchase. I think ill just stick with lame now. It's level of quality combined with it's compatiblity between mp3 players is an unbeatable combination.

I suggest you do also your own tests concentrading for example on pre-echo etc.  (I'm not saying that either one is better, I have not compared LAME 3.96 -V5 --athaa-sensitivity 1 against iTunes 4.2 with pre-echo).
Remember however that these are average results of a group with restricted amount of samples and listeners with different abilities. It shows pretty well the quality on average, but doesn't necessarely show some of the details which might be interesting for you.
Also I think that Lame 3.96 -V5 --athaa-sensitivity 1 is not tested enough to say it doesn't fail (badly) in certain cases even pretty often. Imo iTunes 4.2 AAC in this sense is more safe.

But, if it's not so big deal, that Lame setting does seem on average pretty good.
  • Last Edit: 24 May, 2004, 07:43:29 AM by JohnV
Juha Laaksonheimo

  • Digga
  • [*][*][*][*][*]
Multiformat@128kbps listening test - FINISHED
Reply #42
Quote
A big thank you to Roberto for his efforts in conducting this test.  Let's hope that it is not the last too 

second the thanks to Roberto and everyone elso involved (including all the testers).

Roberto: come on, be honest, you would realy miss all the hick-hack and nag-nag going hand in hand with the tests, wouldn't you   
Nothing but a Heartache - Since I found my Baby ;)

  • diskvask
  • [*]
Multiformat@128kbps listening test - FINISHED
Reply #43
Quote
Quote
What about all the /.ers?  :rolleyes:
Seems they were just interested in wasting bandwidth after all  :lol:

More than 500 people downloaded the samples through bittorrent only - not counting HTTP downloads! :B

I won't ever understand these people.  :frustrated:

I think a lot of people thought that the test was going to be very easy (me included), "Come on, it's 128kbit! That sounds like crap, everybody knows that.".

...only to find out that there couldn't be found any major imperfections in the couple of  samples tried. Sample 1 looks like it was one of the hardest ones to abx; very tough start, especially for someone who had set his mind on the assumtion above.

And besides, abx is an exhausting way of testing and it can be very frustrating/unmotivating if you don't get the results you're expecting ;).

  • Jojo
  • [*][*][*][*][*]
Multiformat@128kbps listening test - FINISHED
Reply #44
Quote
woow, now thats what i not expected

- wma9: lol, worse than mp3! (and i even wonder that it got rated that high, even at 128 it had this metallic sound sometimes) -> go away m$

it's a pitty that wma9 Pro was included in the test ...last test it was included it performed quite well
--alt-presets are there for a reason! These other switches DO NOT work better than it, trust me on this.
LAME + Joint Stereo doesn't destroy 'Stereo'

  • JohnV
  • [*][*][*][*][*]
  • Developer
Multiformat@128kbps listening test - FINISHED
Reply #45
Quote
Quote
woow, now thats what i not expected

- wma9: lol, worse than mp3! (and i even wonder that it got rated that high, even at 128 it had this metallic sound sometimes) -> go away m$

it's a pitty that wma9 Pro was included in the test ...last test it was included it performed quite well

Answer why wma9 pro was not included is here: http://www.hydrogenaudio.org/forums/index....ndpost&p=199103
  • Last Edit: 24 May, 2004, 08:28:20 AM by JohnV
Juha Laaksonheimo

  • dev0
  • [*][*][*][*][*]
  • Developer
Multiformat@128kbps listening test - FINISHED
Reply #46
Quote
This particular test should be called, "The 128 kbps test for iTunes/WMA, and the low-130 test for AC3 and LAME, and the close-to-160 test for MPC/Vorbious.

Leahy   iTunes   MPC   Vorbis   Lame   WMA   Atrac3

bitrate   128   155   149   133   128   132

Score   4.34   4.41   4.68   4.11   4.37   3.76

I am aware of the rationalization. I am aware of the overall average. But let this be a n"oh?" to those that don't and aren't.

Where did you get those numbers from?
"To understand me, you'll have to swallow a world." Or maybe your words.

  • echo
  • [*][*][*]
Multiformat@128kbps listening test - FINISHED
Reply #47
@ Roberto

A big thanks for making this test possible. I hope you reconsider making more tests in the future.

About the test results, I noticed that for some samples there are no confidence intervals on the graphs (bartok_strings, leahy, mahler, ordinary world). Did everybody score exactly the same on these samples, or maybe you just forgot to put the intervals on the graphs?

  • JohnV
  • [*][*][*][*][*]
  • Developer
Multiformat@128kbps listening test - FINISHED
Reply #48
Quote
This particular test should be called, "The 128 kbps test for iTunes/WMA, and the low-130 test for AC3 and LAME, and the close-to-160 test for MPC/Vorbious.

Leahy   iTunes   MPC   Vorbis   Lame   WMA   Atrac3

bitrate   128   155   149   133   128   132

Score   4.34   4.41   4.68   4.11   4.37   3.76

I am aware of the rationalization. I am aware of the overall average. But let this be a n"oh?" to those that don't and aren't.

See here how the average bitrates were decided for this test (personally I'm not absolutely sure if it was enough). Obviously those settings in the table close to 128 were used:
http://www.hydrogenaudio.org/forums/index....ndpost&p=207203

Also the correct average bitrates for the 18 samples tested are (instead of what you said):
Code: [Select]
iTunes MPC   aoTuV  Lame    WMA  Atrac3
128    136     135   134    128    132
  • Last Edit: 24 May, 2004, 09:26:50 AM by JohnV
Juha Laaksonheimo

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Multiformat@128kbps listening test - FINISHED
Reply #49
Roberto> what software did you used to obtain wma9 files? Is it VBR-2 pass 128 kbps? What decoder? I've tried to reproduce the same wavform with different settings, and I wasn't able to do it.