Skip to main content

Topic: MP3 Listening Test at 128 kbps (Read 163056 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Alex B
  • [*][*][*][*][*]
MP3 Listening Test at 128 kbps
Reply #350
While we wait the test to begin it might be useful to revisit the comments that were posted in the 64kbps multiformat test's announce thread:

In that thread I made some suggestions about how the test presentation and instructions could be developed further:


In addition, the comments in the post-test thread make a good read:
  • Last Edit: 09 October, 2008, 06:28:25 AM by Alex B

MP3 Listening Test at 128 kbps
Reply #351
I am afraid that a major "overhaul" of the listening test page and the instruction is not going to be ready in time for this test. I am planning to change the listening test pages and make them somewhat more attractive, but 1. I am not really talented in designing so I have to see who can help me and 2. this is going to take some time so it will be only available for the next teast at earliest. I hope to have less ranked references in this test. Most of the people who perform listening tests are HA readers anyways so I was actually assuming that they know perform a "correct" test.

  • Alex B
  • [*][*][*][*][*]
MP3 Listening Test at 128 kbps
Reply #352
I didn't want to put more pressure on you.

I merely wanted to keep this discussion alive and posted the links for newcomers so that they can get a better impression of the complete testing process.

MP3 Listening Test at 128 kbps
Reply #353
Small status update: I am trying to get in touch with schnofler for some changes in ABC/HR. Everything is pretty much done except for the ABC/HR configuration files and is ready to be uploaded.

MP3 Listening Test at 128 kbps
Reply #354
Here is the updated bitrate table that also contains the encoding speeds (except for iTunes that has to be tested once the fixed version is out).
I didn't want to redo all tests on the single-core machine and comparing iTunes' single-core results to the dual-core results of the other encoders is not fair (at least if any of the other encoders makes use of multi-threading).

Code: [Select]
                Sample 1     Sample 2     Sample 3     Sample 4     Sample 5     Sample 6     Sample 7     Sample 8     Sample 9     Sample 10     Sample 11     Sample 12     Sample 13     Sample 14     Max     Min     Avg     Speed 1)
LAME 3.97    |  97          126          138          149          146          150          95          147          109          158          149          194          133          159          194    95      139    18x
LAME 3.98.2  |  107          149          143          140          138          143          109          136          118          152          145          214          146          156          214    107    143    27x
FhG          |  119          121          139          140          144          149          134          128          129          147          134          212          151          163          212    119    144    45x
iTunes      |  118          117          139          145          133          125          141          120          126          148          151          192          159          158          192    117    141    N/A 2)
Helix        |  114          110          126          151          149          151          131          138          97          152          117          228          143          173          228    97      141    90x
Low Anchor  |  128          128          128          128          128          128          128          128          128          128          128          128          128          128          128    128    128    1,63x
Max          |  119          149          143          151          149          151          141          147          129          158          151          228          159          173
Min          |  97          110          126          140          133          125          95          120          97          147          117          192          133          156
Avg          |  111          125          137          145          142          144          122          134          116          151          139          208          146          162                          142
Max - Min    |  22          39          17          11          16          26          46          27          32          11            34            36            26            17

1) These results are only an approximate value and were obtained on a dual-core Intel E6550 with 2 GB RAM running on Windows Vista SP1 32-bit
2) Encoding speed will be measured with iTunes once the bug-fixed version was released. The buggy version encoded at 18x but produced lower bitrates.

Edit 1: IMPORTANT: Please notice that the low anchor wasn't taken into consideration for building the min, max, delta and average bitrates.
Edit 2: Sorry for breaking the layout - I used a codebox so I don't exactly know what is going wrong here.
  • Last Edit: 15 October, 2008, 05:47:56 PM by Sebastian Mares

  • Polar
  • [*][*][*][*]
MP3 Listening Test at 128 kbps
Reply #355
Sorry to flutter the dovecotes perhaps, but wouldn't it be fairer to rename the test into 140k then, in stead of 128k?

  • Alex B
  • [*][*][*][*][*]
MP3 Listening Test at 128 kbps
Reply #356
Sorry to flutter the dovecotes perhaps, but wouldn't it be fairer to rename the test into 140k then, in stead of 128k?

Short test samples do not represent the average bitrate behavior of complete tracks of various genres in a media library. In my tests the overall average was 127.8 kbps. I think it is close enough.

Since this thread is huge and things can easily get lost in it I'll replicate my test results here (once again):

I finally had time to continue my bitrate tests with classsical music as I promised earlier in this thread (a thing called summer got in the way...)

After browsing through my lossless classical library I picked 25 "reference" tracks that should be quite representative. I avoided the extremely low and high lossless bitrates and tried to select tracks that have quite varied qualities.

Apparently iTunes has changed radically since my last test. Back then the 128 kbps VBR setting was suitable, but the 7.7 version uses bitrates in a more relaxed way and the 128 kbps VBR setting produces higher bitrates than before. Fortunately the 112 kbps VBR setting appears to be suitable for our test.
EDIT: As explained earlier in this thread, the fundamental difference in the bitrate behavior was found out to be caused by a bug in iTunes. However, the "iTunes 7.7" bitrates in my test are now assumed to be correct. Since my test Apple has released iTunes 8.0, but apparently its MP3 encoder has not changed. iTunes 7.7 and 8.0 create identical MP3 files.

In addition, I retested the "various" bitrates with the latest encoder versions when applicable.

Since I didn't have the old iTunes version installed I couldn't test the "classical" bitrates with it (which would have been unnecessary anyway).

FhG, iTunes and LAME 3.97 have only one suitable VBR setting for this test so I'd suggest to calculate the average bitrate of these encoders and adjust the Helix and LAME 3.98 settings to match this average. Helix -V60 and LAME -V5.7 appear to be pretty close to this average with my test tracks.

Here are the new results:


Various - table and chart

Classical - table and chart


I forgot to mention that if anyone wants to test FhG's bitrate behavior the bitrates must be measured correctly. Most programs don't show accurate bitrate values because FhG doesn't write Xing headers to VBR files.

I used EncSpot Pro's "full scan" option.
  • Last Edit: 16 October, 2008, 08:18:36 AM by Alex B

MP3 Listening Test at 128 kbps
Reply #357
Alex, I think this is the 4th time you quoted yourself.

  • Alex B
  • [*][*][*][*][*]
MP3 Listening Test at 128 kbps
Reply #358
Is it? And they still don't learn...

Maybe I should put the test results presentation in my signature. 

(Don't take this personally, Polar. It's just that the "test samples vs. complete library" bitrate difference issue has been discussed over and over again in the listening test threads.)

MP3 Listening Test at 128 kbps
Reply #359

Please report any major bugs. I will make the public announcement tomorrow (actually today since it's after midnight already ).

BTW, many thanks to rjamorim and Polar for hosting the samples!
  • Last Edit: 16 October, 2008, 07:15:10 PM by Sebastian Mares

  • hellokeith
  • [*][*][*][*]
MP3 Listening Test at 128 kbps
Reply #360
Just a grammar/punctuation nitpick:

The purpose of this test is to find out which popular MP3 VBR encoder outputs the best quality on bitrates around 128 kbps.

(removed comma and added MP3 )

  • caligae
  • [*][*][*]
MP3 Listening Test at 128 kbps
Reply #361
From the readme:
Linux users are asked to use Wine with "wine wcmd /c DecodeXX.bat" from the "bin" directory.

Apparently "wcmd" was renamed to "cmd" in wine 0.9.21 which was released 2 years ago.

Is there any description of the samples available? E.g. if I want to test a specific genre I'd have to download all samples. Also, if one likes a track, it might be interesting what's the artist/title.

MP3 Listening Test at 128 kbps
Reply #362
Thanks for the wmcd feedback. I will edit the readmes once I get home this evening.

As for the sample names - they will be published only after the test. With past tests I had the problem, that a lot of people focused on several samples for which I then had lots of results while other samples remained untested.

If a lot of people complain about this, I might publish them, though.
  • Last Edit: 17 October, 2008, 03:39:18 AM by Sebastian Mares