Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: MP3 Listening Test at 128 kbps (Read 209032 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

MP3 Listening Test at 128 kbps

Reply #125 –
Polar,

This MP3 test has been discussed a lot. We did a long pretest discssion about a year ago and we decided to postpone this test in favor of lower bitrate multiformat tests (48 and 64 kbps). The last about ~130 kbps Mp3 test is outdated and now it is a very good time to do this test for the discussed reasons. (Check also the the last year's 14-page discussion).

Sebastian has said that the next test in his list is a 80 kbps multi-format test and I hope we can continue testing codecs in a faster cycle so perhaps an about 100 kbps test could follow soon after that. Then we would have tested all important bitrate classes between 48 and 128 kbps.

MP3 Listening Test at 128 kbps

Reply #127 –
As a spectator and (very grateful) user of these tests, can I explain why I should like to see the iTunes MP3 encoder tested?

I use a Mac, and have just got an iPod. To navigate on an iPod, tags are very important. It's my understanding that iTunes uses a better database than Max to get its tag information, so I should prefer to use iTunes. I also believe that the script that allows one to use Lame in iTunes uses a now somewhat out-of-date version.

So I'd like to get some kind of handle on how much quality I might lose if I switched from Max+Lame to iTunes with native encoder (assuming that Lame is superior, by some degree to be determined). Therefore I'd like to see iTunes tested at best quality settings.

I could, of course, test it myself, but I don't want to become artifact-aware, and a test like this is more systematic than anything I'd set up myself.

I'm very grateful to the people who organise and take part in these tests. One of their virtues to ordinary punters like me is in discovering those things that it is NOT worth testing for oneself.

MP3 Listening Test at 128 kbps

Reply #128 –
I remember in a previous test iTunes did quite badly, and I don't think they would have upgraded the codec since then - they've abandoned it since they switched to using AAC. I think it's worth to note though, the default mp3 bitrate iTunes is 160kbps (where every other program defaults to 128) so i'm sure they knew it wasn't great.

edit: I would really like to see 128k AAC as a high anchor in this test. I know we've already discussed having a high anchor but I think it would be useful to see exactly how much of an edge AAC has over MP3.
 

MP3 Listening Test at 128 kbps

Reply #129 –
As I promised, I tried to do a "quick pretest" with a few samples using foobar's ABX module, but that didn't work. The codecs' quality didn't have immediately obvious differences and the results would have been almost arbitary depending on the used samples. So I felt kind of obligated to do a full-scale personal test.

I used the following codec settings:
FhGc = FIIS 3.4 CBR 128 kbps
FhGv = FIIS 4.1 VBR -br 0 -m 4 -vbri -ofl
FhGv1 = FIIS 4.1 VBR -br 0 -m 4 -q 1 -vbri -ofl
GoG = GoGo 3.13 ABR -a -b 137
GoG2 = GoGo 3.13 ABR -a -b 137 -q 2

Edit: I forgot to mention that I used FhG Mp3sdecoder v. 1.3 for decoding the samples. (Looks like the -ofl option worked. ABC-HR Java didn't adjust the starting position of the FhG VBR samples)

After seeing the results of my previous speed and bitrate test I upped the GoGo ABR bitrate setting from 135 to 137 so that it would better match the codec average. FhG VBR produced still a bit higher average bitrate with the now tested new samples (134 kbps vs. 131 kbps).

I tested 30 varied samples. All samples are created by myself and most of them are brand new. About half of them are from Vangelis' Alexander soundtrack album, which has a lot of interesting music (it's a mixture of classical, electronic and ethnic genres) and I wanted to try if that material would be useful for testing codecs.

The reference samples and my ABC-HR result files are available in this link:
http://rapidshare.com/files/55027379/alexb_test.zip  (about 76 MB)

Here are the test results:
Code: [Select]
% Result file produced by chunky-0.8.4-beta
% Chunky -d C:\alexb_test\results -r sample -f C:\alexb_test\codecs.txt
%
% Sample Averages:

S.#    FhGc    FhGv    FhGv1    GoG    GoG2
#01    2.90    3.50    3.50    4.00    3.50
#02    3.90    3.30    3.30    2.40    3.30
#03    4.00    3.60    3.90    3.60    4.00
#04    4.00    4.00    4.00    4.00    4.00
#05    3.00    4.10    3.20    3.70    3.40
#06    4.00    3.60    3.30    4.50    2.90
#07    4.00    3.50    4.10    3.30    4.50
#08    4.00    3.80    4.40    4.40    4.20
#09    4.10    3.80    3.20    4.20    3.50
#10    4.30    4.30    3.70    4.50    3.40
#11    3.00    2.50    2.40    3.00    3.00
#12    3.70    4.10    4.00    4.20    3.50
#13    3.20    3.20    3.90    3.70    2.80
#14    3.40    2.80    2.20    3.50    2.50
#15    4.00    4.30    3.90    3.90    4.20
#16    3.20    3.70    3.90    3.70    3.00
#17    3.90    3.10    3.00    4.00    4.00
#18    3.80    4.10    3.90    4.10    3.80
#19    3.80    2.90    3.10    3.90    3.90
#20    3.10    3.10    3.70    3.90    1.90
#21    2.30    2.70    3.00    2.20    2.40
#22    3.90    3.90    3.90    3.90    4.30
#23    3.90    4.00    4.00    4.00    3.80
#24    4.20    4.20    4.20    3.90    4.20
#25    4.20    4.00    4.10    4.00    3.60
#26    3.90    3.70    4.00    3.70    3.70
#27    3.80    4.10    4.00    3.80    3.80
#28    3.50    3.60    3.60    4.00    3.20
#29    2.60    2.80    3.00    1.60    1.50
#30    3.70    3.70    3.50    3.70    3.30

%%%    Codec averages:
%%%    3.64    3.60    3.60    3.71    3.44


Even though I did the test properly, I wouldn't trust the results too much.

Mostly I was able to differentiate the contenders without ABXing because of the lowpass. All encoders seem to use an about 15-16 kHz lowpass frequency at this bitrate/quality level. Beyond that the test was a tough call. The codecs were generally quite good and the slight lowpass didn't make the samples annoying (to me, it seems to be detectable only in a direct comparison with the reference).

I didn't want to spend several days with this test so I rated the samples rather quickly and it is possible that a new test with the same samples would produce slightly different results.

GoGo failed misarably on one killer sample, the sample number 29 (also FhG was bad with this sample, but better than GoGo). On the other hand FhG's VBR has obvious problems with some other samples.

I can make the following conclusions:
All encoders are tied and the higher quality options didn't make the encoders better.

Edit: Fixed a couple of typos and added the MP3 decoder info.

MP3 Listening Test at 128 kbps

Reply #130 –
Thank you for your exhausting test.

At least it shows that

a) Gogo should be used without the -q2 switch
b) Gogo is competitive with FhG

Unfortunately it doesn't show much about how to use FhG.
lame3995o -Q1.7 --lowpass 17

MP3 Listening Test at 128 kbps

Reply #131 –
Quote from: halb27 on
Thank you for your exhausting test.

At least it shows that

a) Gogo should be used without the -q2 switch
b) Gogo is competitive with FhG

Unfortunately it doesn't show much about how to use FhG.

I would ignore any difference that is smaller than 0.5 in my test. As you know, when you hear small artifacts, which may not be the same (i.e. when each encoder produces more or less different artifacts in different positions), it is quite difficult to put the contenders in an annoyance order. On each listening time you may rate the encoders differently. If you spend more time with each test sample the opinion may became more consistent, but as I said, I had to do the test rather quickly.

With some samples I found clear winners and  losers, but also within this subgroup the winners and losers varied from sample to another.

So I think my test can prove only that I didn't find any of the contenders to be clearly worse or better in this specific test.

I wonder why the quality switches didn't work as one would like to expect? Is it possible that when the encoder does a more comprehensive analysis it can also go wrong in some cases and a more straightforward faster approach can actually produce better results?

However, I too would select the faster setting for GoGo, because I would like to get more proof of its usability in the faster default mode. *)

Regarding FhG, I think the newer 4.1 version & VBR mode should be selected, because my test didn't show any clear regression when compared with the 3.4 version. This would also make the test a VBR-only test and the average bitrates of the test contenders would be closer to each other. According to my test it would be safe to select the faster default -q value, but I really would like to see results from other forum members. Perhaps my test samples didn't reveal everything.


Edit: *) In addition, before selecting a specific -q value for GoGo we probably should pretest -q 0, -q 1, -q 3 etc too. Otherwise we would be just quessing the best value. It is better to use the default setting similarly like with Helix. FhG 4.1 VBR is a bit different case because it has only two possible -q values, which can be pretested easier.

MP3 Listening Test at 128 kbps

Reply #132 –
Thanks for your test. From my understanding FHG 4.1 vbr is ABR at M4 / M5 ?

MP3 Listening Test at 128 kbps

Reply #134 –
Does it really make sense to include GoGo? I think not:

1) Practically nobody uses it
2) Basically it's just an ancient version of Lame (3.92 I think) with some speed optimisations
Proverb for Paranoids: "If they can get you asking the wrong questions, they don't have to worry about answers."
-T. Pynchon (Gravity's Rainbow)

MP3 Listening Test at 128 kbps

Reply #135 –
Quote from: sTisTi on
Does it really make sense to include GoGo? I think not:

1) Practically nobody uses it
2) Basically it's just an ancient version of Lame (3.92 I think) with some speed optimisations

So you didn't want to contribute this discussion by suggesting which encoders should be tested and especially which FhG settings (or version) should be selected. Just bashing GoGo does not really advance this discussion.

1) How do you know that? What encoder people usually select when they need a faster MP3 encoder than e.g. LAME? (Naturally I mean only those people who know about the options. It is obvious that most users on this planet do not change the used encoder. They just wait longer or if that is not possible they don't get everything encoded in time.)

2) It is based on LAME 3.88 and much of its code is completely rewritten in assemply language. In my test it was about 5x faster than LAME. This what the encoder displays about its version:
Quote
GOGO-no-coda ver. 3.13 ( May. 20 2004 ) is a mp3 encoder based on lame 3.88,
which is distributed under LGPL on http://www.mp3dev.org/mp3/ .
See http://member.nifty.ne.jp/~pen/ , http://homepage1.nifty.com/herumi/gogo_e.html .

After my listening test I'm even more interested about GoGo. If it really is compentive against FhG it deserves to be tested in a public test. We are trying to find the best encoders that are available for encoding MP3 files just now. AFAIK, GoGo has no technical problems. The resulting MP3 files work fine. It doesn't matter for practical purposes if the development continues or not.

MP3 Listening Test at 128 kbps

Reply #136 –
Quote from: Alex B on
2) It is based on LAME 3.88 and much of its code is completely rewritten in assemply language. In my test it was about 5x faster than LAME. This what the encoder displays about its version:
Quote
GOGO-no-coda ver. 3.13 ( May. 20 2004 ) is a mp3 encoder based on lame 3.88,
which is distributed under LGPL on http://www.mp3dev.org/mp3/ .
See http://member.nifty.ne.jp/~pen/ , http://homepage1.nifty.com/herumi/gogo_e.html .

After my listening test I'm even more interested about GoGo. If it really is compentive against FhG it deserves to be tested in a public test. We are trying to find the best encoders that are available for encoding MP3 files just now. AFAIK, GoGo has no technical problems. The resulting MP3 files work fine. It doesn't matter for practical purposes if the development continues or not.

Well, GoGo was tested years ago in this test and came out clearly worse than the current Lame version (3.95) then. So anyone concerned about quality will avoid it. And that was years ago. How many people really need their MP3s so fast that they are ready to sacrifice quality today with all the superfast computers around? Sorry, I really don't see an audience for such a codec anymore. Just look at the almost complete lack of discussion here on HA about using GoGo for serious encoding work.

If it's really based on Lame 3.88 it's even more ridiculous to include it. Just think of the quality tuning work of 6 1/2 years that went into Lame since then, including Dibrom's Presets and Gabriel's complete reworking of the -V n modes.

I think it's a waste of time to test GoGo if there are more interesting and more widely used codecs around that can be included in the test. As the test will be very hard for most participants due to the hight quality of the codecs, I think we should limit the number of codecs as far as possible.
Proverb for Paranoids: "If they can get you asking the wrong questions, they don't have to worry about answers."
-T. Pynchon (Gravity's Rainbow)

MP3 Listening Test at 128 kbps

Reply #137 –
Quote from: Sebastian Mares on
I think after the MP3 test we will have an AAC test with both LC and HE-AAC at 80 kbps. The winner of that test will be included in the multiformat 80 kbps test.


I was hoping 96 or 128 AAC would be tested as these are more commonly used. Are you going about this systematically from lowest bitrates to high?

MP3 Listening Test at 128 kbps

Reply #138 –
Quote from: sTisTi on
Well, GoGo was tested years ago in this test and came out clearly worse than the current Lame version (3.95) then. So anyone concerned about quality will avoid it. And that was years ago. How many people really need their MP3s so fast that they are ready to sacrifice quality today with all the superfast computers around? Sorry, I really don't see an audience for such a codec anymore. Just look at the almost complete lack of discussion here on HA about using GoGo for serious encoding work.

If it's really based on Lame 3.88 it's even more ridiculous to include it. Just think of the quality tuning work of 6 1/2 years that went into Lame since then, including Dibrom's Presets and Gabriel's complete reworking of the -V n modes.

I think it's a waste of time to test GoGo if there are more interesting and more widely used codecs around that can be included in the test. As the test will be very hard for most participants due to the hight quality of the codecs, I think we should limit the number of codecs as far as possible.


I don't think any of the now tested other encoders can be competitive against LAME. In Roberto's test GoGo was tied with all other MP3 encoders except LAME:



I have a hunch that the situation has not changed much since then.

However, as you said that test is old and the encoders and suitable settings are going to be different now.

One of the reasons to do this test now was to check if the considerably faster encoders can produce usable quality when compared with LAME. The faster options seem to be FhG, Helix and GoGo. iTunes is not practically faster, but it should be included for other reasons as discussed in this thread.

The faster encoders are useful in many situations. One example is when you want to quickly create more or less temporary files for a small portable. Another application could be on-demand WAN or LAN distribution.


It would be nice to finally select the contenders after these 14- and 6-page discussions and start the test samples discussion. In Roberto's MP3 test and in my personal test the results varied clearly from sample to another. Actually, I think that a test like this could be easily manipulated to produce certain results by selecting certain test samples. The sample selection is going to be crucial.

MP3 Listening Test at 128 kbps

Reply #139 –
@sebastian, can we also include lastest Coding Technologies AAC from Winamp. both LC an HE on 80kbps test?

MP3 Listening Test at 128 kbps

Reply #141 –
Sorry for not showing up here lately, but I had a car accident and had to sort up some things. In the meanwhile I've got my car back from the workshop and hired a lawyer to deal with the insurance company of the guy who drove into the back of my car during a traffic jam.

So, where are we... We need to decide what encoders to feature and what settings to use. One (IMO) important question: how many contenders should we have?

MP3 Listening Test at 128 kbps

Reply #142 –
Summary:


List of possible and pertinent (IMO) encoders:

• Fraunhofer surround
• GoGo
• Helix
• iTunes
• LAME


List of possible ambiguities:

• Fraunhofer surround : -q0 (fast & defaut) vs -q1 (slower but labelled as "high quality mode") // ABR vs VBR
• GoGo : -q2 or not
• Helix : no ambiguity (seems that nobody suggested ABR over VBR nor any magic commandline)
• iTunes : not discussed that much IIRC ; 128 VBR would maybe work (I mean bitrate-wise)
• LAME : no ambiguity (3.98b5 -V5)


List of possible goals of this test:

• best possible MP3
• best fastest MP3 encoder outside LAME
• best MP3 among "alive" encoders
• LAME vs major companies (iTunes, Fraunhofer, Real)


Correct me if I missed something.
___


I would personally go for a LAME + Fhg + iTunes + Helix match, set with VBR when possible and with settings we would trust to be the best ones (according to the community or the developers themselves).
Wavpack Hybrid: one encoder for all scenarios
WavPack -c4.5hx6 (44100Hz & 48000Hz) ≈ 390 kbps + correction file
WavPack -c4hx6 (96000Hz) ≈ 768 kbps + correction file
WavPack -h (SACD & DSD) ≈ 2400 kbps at 2.8224 MHz

MP3 Listening Test at 128 kbps

Reply #143 –
So, since we have no high anchor, we could go for 5 contenders + 1 low anchor or 4 contenders + 1 low anchor. If 5 contenders are OK, we can test all 5 suggested MP3 encoders including Gogo. Otherwise, we have to decide which encoder we drop.

MP3 Listening Test at 128 kbps

Reply #144 –
Quote from: kennedyb4 on
Quote from: Sebastian Mares on

I think after the MP3 test we will have an AAC test with both LC and HE-AAC at 80 kbps. The winner of that test will be included in the multiformat 80 kbps test.


I was hoping 96 or 128 AAC would be tested as these are more commonly used. Are you going about this systematically from lowest bitrates to high?

No please, 80 kbps is excellent!  With Vorbis, it is *the* threshold bitrate that I use for my portable.  The quality is good enough and not the least bit irritating (unlike 64 kbps).  I think this is an essential bitrate to test and I'm glad Sebastian is testing it.  Sorry for the OT, just want to show my support for the future planned tests.

Quote from: Sebastian Mares on
So, since we have no high anchor, we could go for 5 contenders + 1 low anchor or 4 contenders + 1 low anchor. If 5 contenders are OK, we can test all 5 suggested MP3 encoders including Gogo. Otherwise, we have to decide which encoder we drop.

As difficult as the last 128 kbps multi-format test proved to be, I think the fewer contenders the better, though this theoretically won't be as difficult since different implementations of the same (older) format are being tested here.

MP3 Listening Test at 128 kbps

Reply #145 –
Quote from: Sebastian Mares on
So, since we have no high anchor, we could go for 5 contenders + 1 low anchor or 4 contenders + 1 low anchor. If 5 contenders are OK, we can test all 5 suggested MP3 encoders including Gogo. Otherwise, we have to decide which encoder we drop.

I agree. The encoders are good enough, that the high anchor can be dropped.

MP3 Listening Test at 128 kbps

Reply #148 –
Quote from: Sebastian Mares on
Are 5 contenders too many?

I would prefer 4 but I'm not against this 5+1 configuration if it helps us to unlock the situation a bit
Wavpack Hybrid: one encoder for all scenarios
WavPack -c4.5hx6 (44100Hz & 48000Hz) ≈ 390 kbps + correction file
WavPack -c4hx6 (96000Hz) ≈ 768 kbps + correction file
WavPack -h (SACD & DSD) ≈ 2400 kbps at 2.8224 MHz

MP3 Listening Test at 128 kbps

Reply #149 –
Let's move on to the settings...

Is --vb-new default in LAME 3.98? Should we choose the highest quality setting in iTunes? FhG -q0 or -q1? Gogo ABR or VBR, any -q setting? Helix CLI encoder or Real, what setting?