Skip to main content

Topic: Multiformat Listening Test @ 128 kbps - FINISHED (Read 141630 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
Multiformat Listening Test @ 128 kbps - FINISHED
The much awaited results of the Public, Multiformat Listening Test @ 128 kbps are ready.

Here is the results page: http://www.maresweb.de/listening-tests/mf-128-1/results.htm



Edit 1: A description of the mentioned Nero problem can be found here: http://www.maresweb.de/nero-problem

Edit 2: For people who want to decrypt their results, here are the encoder IDs:

1 = iTunes
2 = LAME
3 = Nero
4 = Shine
5 = AoTuV
6 = WMA Professiona
  • Last Edit: 14 January, 2006, 05:57:02 PM by Sebastian Mares

  • minisu
  • [*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #1
Great job!

*goes decrypting*
  • Last Edit: 14 January, 2006, 04:06:45 PM by minisu
Opera bookmark synchronizer: http://osync.sf.net

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #2
Just a quick info regarding the Nero bug - unfortunately it was found too late, as the encoder was given away couple of days  before the test (*** too short ***) and it was completely "rushed in" to be ready for the test, even few months before the complete release.  In the whole "rush" process, the bug was overlooked in the internal Nero Digital Audio QA as well by the external people testing the codecs (Guru and few others)

This bug reflects quality in the unpredictable way (as it does not allocate bits according to the psychoacoustics but drains the bit reservoir) - but in general we believe that the quality difference would not be significant (in fact it is my belief that it would be better without the bug as the extra bits would be allocated according to psychoacoustic model) - however Sebastian decided to exclude the Nero codec from the test, which is IMO unfortunate, but I can understand his decision.

Fortunately, the bug has been fixed (thanks to Guruboolez and his very good hearing) - and this kind of behavior will be included in pre-test screening of the codecs to seek for such obvious bugs.

I am dissapointed that this bug was found out too late, but hopefully for the next tests we won't be having such problems.
  • Last Edit: 14 January, 2006, 04:14:21 PM by Ivan Dimkovic

  • skelly831
  • [*][*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #3
Wow! awesome results, it's interesting to see how close iTunes and AoTuv are in the overall rating but they're somewhat disparate in the bitrate table.
we was young an' full of beans

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #4
How about making a hyperlink to the page describing the nero bug?  Good explanation of what went wrong.

ff123

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #5
By the way, results that were invalid (didn't meet ABX minimums) were not uploaded. Since I posted the encryption key, you can decrypt the results yourself if you are wondering why your result is not counted.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #6
Great job to everyone who participated to the test - especially to the conducer!

The first immediate lesson of this test is that ~132...135 kbps main encoders are very, very good. Even LAME at -V5 is close to transparency. I hope that -V2/--preset standard will progressively cease to be the automatic recommendation when new members are asking for good quality MP3 encodings: the first step to excellency is below the historical presets!

Developers must be celebrated: Apple's and Nero's developers for leading AAC to the best places; Aoyumi for having resurrecting Vorbis; Microsoft's developers for -at last- offering a very good encoding and free encoding tool; and of course the whole LAME team who are making MP3 better and better even if people are sometimes not realizing it or believing that MP3 couldn't be improved!

Last, people could compare the collective results to my individual test. I've conclude on aoTuV and iTunes superiority, with both Nero and LAME slightly lower. The group results are concluding on exactly the same order, with only less significance and an higher notation. It looks that my hearing thus my listening tests may be very representative from the group's one
  • Last Edit: 14 January, 2006, 07:35:13 PM by guruboolez

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #7
By the way, Francis noticed another problem. The sample "Yello" submitted by Alex B was not lossless. By mistake, Alex B uploaded a file transcoded from a high bitrate MP3.

Guru noticed that the reference file he was listening to sounded too much like an encoded file. After looking at the spectral view of the track with CE, he noticed what he calls "an adaptive lowpass with some spectral 'holes'" which is typical to lossy encoders. I contacted Alex B about the problem and received the following mail today:

Quote
Hi again!

Here's what I found. It is a long story, but explains how accidents can happen when dealing with a large number of test files, even if you try to [be] careful.

I checked the reference and compared it with my ripped CD archive file, which is in Monkey's Audio disc image file & cue format and noticed that there really is a difference.

However, at first I couldn't understand why. It could not be the WMA 2-pass version I made for testing the bitrates. WMA 128 kbps has a lower lowpass. I have also a Musepack Q8 version on my home audio server, but that does not have such a lowpass.

Then I remembered that I cut the sample from the 25 files bitrate test archive I have stored. I have the separated track already there so it was faster than converting the big disc image ape file.

I selected these 25 files originally for testing LAME 3.97 VBR bitrates in September. When I made the different VBR sets I had the original cue files loaded in foobar and I converted each VBR set from the same playlist. Later when I cleaned the resulting 500 MP3 files I converted the 25 original cue tracks to separate lossless tracks for future use.

It appears that I have somehow accidentally loaded a high bitrate MP3 file instead of the cue to a foobar playlist and converted it to Monkey's Audio. The other 24 Monkey's Audio tracks seem to be fine, only this one is unfortunately different.

When I made this particular sample for testing WMA 2-pass in November I converted that ape file to wave and cut it. By looking the lowpass I believe the MP3 file was encoded at -V0. There is no way to tell if it was VBR new or old.

I am sorry about the mistake.

I think this sample will accidentally show that these new encoders are not bad transcoders, at least when the source is this kind of loud track. It is a different thing if you like to publish this information. It could be only meaningless clutter. It was never exactly mentioned how the source files were obtained. The test compared these reference files and the encoded versions.
  • Last Edit: 14 January, 2006, 04:38:30 PM by Sebastian Mares

  • Shade[ST]
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #8
The URL in the image is incorrect (www.maresweb.de/nero-problem.txt) -- might as well make the redirect at /nero-problem, too, anyways...

In any case, It's a shame I misunderstood the testing scenario : I did not know I needed to ABX the codecs before rating them :-/

All my results are invalid :-(

However, it's surprising to learn that all popular codecs around 128 kbps are of high enough quality to be practically imperceptible to everyone.

Great!

edit : de / net, whatever.
  • Last Edit: 14 January, 2006, 04:41:46 PM by Shade[ST]

  • Lyx
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #9
Pointing out an old issue with listening-test presentations: neither the plot, nor the detailed results page, mention clearly that "iTunes" means "iTunes AAC Encoder". Taking into account that results of ha.org listening tests are often posted elsewhere without all the necessary info, people may once more mistake iTunes (good) AAC performance with its (bad) MP3 performance.
I am arrogant and I can afford it because I deliver.

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #10
Quote
The URL in the image is incorrect (www.maresweb.net/nero-problem.txt) -- might as well make the redirect at /nero-problem, too, anyways...
[a href=\"index.php?act=findpost&pid=357127\"][{POST_SNAPBACK}][/a]

maresweb.de is the correct domain. .net and .org are only redirecting to the .de domain. Also, I tested the URL from the picture and Apache redirects the users to the correct file. No problem I guess.

Quote
In any case, It's a shame I misunderstood the testing scenario : I did not know I needed to ABX the codecs before rating them :-/

All my results are invalid :-(

However, it's surprising to learn that all popular codecs around 128 kbps are of high enough quality to be practically imperceptible to everyone.

Great!
[a href=\"index.php?act=findpost&pid=357127\"][{POST_SNAPBACK}][/a]

No, no, no... You don't have to ABX all files. ABX logs are only required when you ranked a reference.
IIRC, I used a large part of your results.

And BTW, this was a funny result:

Code: [Select]
 ABC/HR for Java, Version 0.5b, 06 december 2005 
 Testname: DontLetMeBeMisunderstood
 
 Tester: 
 
 1R = Sample05\DontLetMeBeMisunderstood_1.wav
 2L = Sample05\DontLetMeBeMisunderstood_2.wav
 3L = Sample05\DontLetMeBeMisunderstood_3.wav
 4L = Sample05\DontLetMeBeMisunderstood_6.wav
 5R = Sample05\DontLetMeBeMisunderstood_4.wav
 6R = Sample05\DontLetMeBeMisunderstood_5.wav
 
 ---------------------------------------
 General Comments: Focus on 4.16 - 6.46
 ---------------------------------------
 1L File: Sample05\DontLetMeBeMisunderstood.wav
 1L Rating: 4.8
 1L Comment: 
 ---------------------------------------
 5L File: Sample05\DontLetMeBeMisunderstood.wav
 5L Rating: 1.0
 5L Comment: 
 ---------------------------------------
 
 ABX Results:

  • Shade[ST]
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #11
Also a question / comment.  Is Itunes' encoder actually in iTunes, or is it in quicktime?  What versions were used?  Maybe you should list those on the results page (versions of each encoder..)

  • Shade[ST]
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #12
Quote
Code: [Select]
Testname: DontLetMeBeMisunderstood 
Tester:  

 1R = Sample05\DontLetMeBeMisunderstood_1.wav
 2L = Sample05\DontLetMeBeMisunderstood_2.wav
 3L = Sample05\DontLetMeBeMisunderstood_3.wav
 4L = Sample05\DontLetMeBeMisunderstood_6.wav
 5R = Sample05\DontLetMeBeMisunderstood_4.wav
 6R = Sample05\DontLetMeBeMisunderstood_5.wav
  
 ---------------------------------------
 General Comments: Focus on 4.16 - 6.46
 ---------------------------------------
 1L File: Sample05\DontLetMeBeMisunderstood.wav
 1L Rating: 4.8
 1L Comment:  
 ---------------------------------------
 5L File: Sample05\DontLetMeBeMisunderstood.wav
 5L Rating: 1.0
 5L Comment:  
 ---------------------------------------
Wuups.. Thanks for hiding my tester name
  • Last Edit: 14 January, 2006, 04:45:40 PM by Shade[ST]

  • minisu
  • [*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #13
Ok, so this is one of my decrypted results... http://www.maresweb.de/listening-tests/mf-...12-result02.txt

How to know which encoders that matches which sample number?
  • Last Edit: 14 January, 2006, 04:46:24 PM by minisu
Opera bookmark synchronizer: http://osync.sf.net

  • Shade[ST]
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #14
More comments : http://www.maresweb.de/listening-tests/mf-128-1/ Maybe you should update this page to point to the results.

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #15
Quote
Also a question / comment.  Is Itunes' encoder actually in iTunes, or is it in quicktime?  What versions were used?  Maybe you should list those on the results page (versions of each encoder..)
[{POST_SNAPBACK}][/a]

Well, they're on the presentation page which I think is enough.

Quote
Quote
Code: [Select]
Testname: DontLetMeBeMisunderstood 
Tester: 

 1R = Sample05\DontLetMeBeMisunderstood_1.wav
 2L = Sample05\DontLetMeBeMisunderstood_2.wav
 3L = Sample05\DontLetMeBeMisunderstood_3.wav
 4L = Sample05\DontLetMeBeMisunderstood_6.wav
 5R = Sample05\DontLetMeBeMisunderstood_4.wav
 6R = Sample05\DontLetMeBeMisunderstood_5.wav
 
 ---------------------------------------
 General Comments: Focus on 4.16 - 6.46
 ---------------------------------------
 1L File: Sample05\DontLetMeBeMisunderstood.wav
 1L Rating: 4.8
 1L Comment: 
 ---------------------------------------
 5L File: Sample05\DontLetMeBeMisunderstood.wav
 5L Rating: 1.0
 5L Comment: 
 ---------------------------------------
Wuups.. Thanks for hiding my tester name
[a href=\"index.php?act=findpost&pid=357132\"][{POST_SNAPBACK}][/a]

I didn't say it was your result. Or to be more precise, it wasn't your result. It was from anonymous user.

Quote
Ok, so this is one of my decrypted results... [a href=\"http://www.maresweb.de/listening-tests/mf-128-1/miscellaneous/results/Sample02/anon12-result02.txt]http://www.maresweb.de/listening-tests/mf-...12-result02.txt[/url]

How to know which encoders that matches which sample number?
[a href=\"index.php?act=findpost&pid=357133\"][{POST_SNAPBACK}][/a]

1 = iTunes
2 = LAME
3 = Nero
4 = Shine
5 = AoTuV
6 = WMA Professional

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #16
Quote
More comments : http://www.maresweb.de/listening-tests/mf-128-1/ Maybe you should update this page to point to the results.
[a href="index.php?act=findpost&pid=357134"][{POST_SNAPBACK}][/a]


If you go one lever higher, you come to the listening tests page which lists all tests as pairs of presentation page and results page. AFAIK, Roberto does it the same way, too. But no problem, I can edit that page when the FTP server works again.

  • fpi
  • [*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #17
Quote
maresweb.de is the correct domain. .net and .org are only redirecting to the .de domain. Also, I tested the URL from the picture and Apache redirects the users to the correct file. No problem I guess.

You missed the .de on the graph.
Can you also add more complete info of the encoder in the graph? e.g.: AoTuv -> Vorbis AoTuV 4.51, Nero -> AAC Nero 3.1.0.2, etc... I prefer first the format, then vendor and version. Many sites link only to the image and can give a confusing idea of which encoder was used. Also on that image should be a link to the full explanation of the results.
  • Last Edit: 14 January, 2006, 05:29:02 PM by fpi

  • IgorC
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #18
First of all great test. Thanks, Sebastian.

However it's sad what happens with Nero encoder.
Maybe it will be a fair idea to cut 2/3 of encoded samples and see how much size of nero's samples is bigger. However it's not correctly 100% because of  VBR distribution. I've already learned that the bigger/smaller size is not always the indicator  of quality  . However if distribution 150-130 (140) kbps is a good idea to check it. 10 kbit/s isn't issue 7% of total bitrate.  It was admited that Itunes had 10 kbit/s extra of real bitrate  that's ok because of VBR.

However I think it was right decesion to keep us informed  about this issue.
  • Last Edit: 14 January, 2006, 05:31:57 PM by IgorC

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #19
Ah, damn, you're right. Now I see what you guys mean. OK, going to PSP the graphs again... >_<

  • Shade[ST]
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #20
Quote
Quote
,Jan 14 2006, 10:42 PM]Also a question / comment.  Is Itunes' encoder actually in iTunes, or is it in quicktime?  What versions were used?  Maybe you should list those on the results page (versions of each encoder..)

Well, they're on the presentation page which I think is enough.

Actually, nowhere is that page linked on here... Also, If you wish, I can set up a nice 'design' image to show the results;  Could you post the numbers (ranges), PM them to me or email them?  I'll also include the necessary links and info in the image.  And I can give you a PDF format of it, if you like.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #21
Quote
I've already learned that the bigger/smaller size is not always the indicator  of quality  
[a href="index.php?act=findpost&pid=357143"][{POST_SNAPBACK}][/a]

That's right. But in this case, a careful listening reveals (revealed in my case) that this bug has an audible impact, leading to a brutal drop in perceived quality. The impact on quality varies from nothing to considerable.

Quote
However if distribution 150-130 (140) kbps is a good idea to check it. 10 kbit/s isn't issue 7% of total bitrate.  It was admited that Itunes had 10 kbit/s extra of real bitrate  that's ok because of VBR.

Even if bitrate stays within the tolerence of 10%, there's still the problem of unrepresentativity (the same one which decided to not use WMA Std 2-pass with short samples). The tested samples content and quality is different from what a user would get.

  • JeanLuc
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #22
Damn ... looking at the bitrate table and regarding the overall ranking, iTunes AAC seems so effective.

It is good to see, though that users can today chose between different formats at comparable bitrates without having to ask about a possible sacrifice in quality.
The name was Plex The Ripper, not Jack The Ripper

  • IgorC
  • [*][*][*][*][*]
Multiformat Listening Test @ 128 kbps - FINISHED
Reply #23
Quote
there's still the problem of unrepresentativity (the same one which decided to not use WMA Std 2-pass with short samples). The tested samples content and quality is different from what a user would get.

yes , I had fear to it.

Multiformat Listening Test @ 128 kbps - FINISHED
Reply #24
Quote
Quote
maresweb.de is the correct domain. .net and .org are only redirecting to the .de domain. Also, I tested the URL from the picture and Apache redirects the users to the correct file. No problem I guess.

You missed the .de on the graph.
Can you also add more complete info of the encoder in the graph? e.g.: AoTuv -> Vorbis AoTuV 4.51, Nero -> AAC Nero 3.1.0.2, etc... I prefer first the format, then vendor and version. Many sites link only to the image and can give a confusing idea of which encoder was used. Also on that image should be a link to the full explanation of the results.
[a href="index.php?act=findpost&pid=357142"][{POST_SNAPBACK}][/a]


Well, if someone posts the image, he should also post to results page.
Adding the full encoder version / information is useless IMHO - it's stated already on the presentation page (which can be accessed if you are on the results page, that is supposed to be posted together with the plot).

Quote
,Jan 14 2006, 11:31 PM]
Quote
Quote
,Jan 14 2006, 10:42 PM]Also a question / comment.  Is Itunes' encoder actually in iTunes, or is it in quicktime?  What versions were used?  Maybe you should list those on the results page (versions of each encoder..)

Well, they're on the presentation page which I think is enough.

Actually, nowhere is that page linked on here... Also, If you wish, I can set up a nice 'design' image to show the results;  Could you post the numbers (ranges), PM them to me or email them?  I'll also include the necessary links and info in the image.  And I can give you a PDF format of it, if you like.
[a href="index.php?act=findpost&pid=357145"][{POST_SNAPBACK}][/a]


Sent! Thanks.

BTW, I only changed iTunes and Nero to iTuness AAC and Nero AAC on the final plots. The sample plots still have iTunes only. I hope it's not such a big deal since those images are almost never posted alone.

That's it guys - a warm bed is waiting for me.

BTW... Gambit is screwed if his neighbors read his comments.
  • Last Edit: 14 January, 2006, 06:03:03 PM by Sebastian Mares