Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: MP3 Listening Test at 128 kbps (Read 207856 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

MP3 Listening Test at 128 kbps

Reply #201 –
I am not a Lame dev and I think that its a bad idea for a public test. Too difficult for most people - harder than a normal test 130 k test which is already hard enough for many.

MP3 Listening Test at 128 kbps

Reply #202 –
As I said in a previous post - given the fact that Gogo is not developed any longer (at least quality-wise) and that VBR was not mature in 3.88 according to Gabriel if I recall correctly and therefore not recommended, I fancy the idea to test FhG CBR just to have a comparison between high quality encoders (LAME, FhG with Q switch and VBR, iTunes using best quality settings) and fast encoders (Helix VBR and FhG CBR without Q switch).

MP3 Listening Test at 128 kbps

Reply #203 –
Quote from: Sebastian Mares on
My understanding of ABR is that the the encoder also "memorizes" what bitrate it used on previous frames and allocates bits for future frames taking this "memory" into consideration - sort of like the bit reservoir works, just that it allows larger fluctuations. The problem I see is that an encoded sample might have another bit allocation than if you were to encode the whole track and then cut out the part you want to test.

IMHO, if you don't test the first second of encoded audio, that should be enough to let the encoder adapt itself correctly for Lame-based encoders. Within Lame (up to now), the ABR window is quite short.

Quote from: Sebastian Mares on
Do we need to test 3.97 vs. 3.98 in a public listening test? That was never considered in this test. What do LAME developers think?

I'd say no. People might do some side tests about 3.98 vs 3.97 in order to help Lame's development, but I don't see the purpose within a multi-encoders listening test.

MP3 Listening Test at 128 kbps

Reply #204 –
Quote from: Gabriel on
Quote from: Sebastian Mares on

My understanding of ABR is that the the encoder also "memorizes" what bitrate it used on previous frames and allocates bits for future frames taking this "memory" into consideration - sort of like the bit reservoir works, just that it allows larger fluctuations. The problem I see is that an encoded sample might have another bit allocation than if you were to encode the whole track and then cut out the part you want to test.

IMHO, if you don't test the first second of encoded audio, that should be enough to let the encoder adapt itself correctly for Lame-based encoders. Within Lame (up to now), the ABR window is quite short.

Quote from: Sebastian Mares on
Do we need to test 3.97 vs. 3.98 in a public listening test? That was never considered in this test. What do LAME developers think?

I'd say no. People might do some side tests about 3.98 vs 3.97 in order to help Lame's development, but I don't see the purpose within a multi-encoders listening test.


I'd say yes. LAME is one of the most important mp3 encoders ever made, so let's see if 3.98 is a step up from 3.97. This is probably one of the best opportunities to do so.  It is a mp3 listening test after all ...

MP3 Listening Test at 128 kbps

Reply #205 –
Quote from: Gabriel on
Quote from: Sebastian Mares on

My understanding of ABR is that the the encoder also "memorizes" what bitrate it used on previous frames and allocates bits for future frames taking this "memory" into consideration - sort of like the bit reservoir works, just that it allows larger fluctuations. The problem I see is that an encoded sample might have another bit allocation than if you were to encode the whole track and then cut out the part you want to test.

IMHO, if you don't test the first second of encoded audio, that should be enough to let the encoder adapt itself correctly for Lame-based encoders. Within Lame (up to now), the ABR window is quite short.

I would like to add, that all modes LAME has to offer use the same information:
- psymodel: uses info from two previous granues (one MPEG-1 frame, two MPEG-2 frames).
- quantization loops: use info from one previous frame
So, ABR doesn't do anything special compared to CBR or VBR here.
LAME's ABR doesn't even observe the actual average bitrate.
LAME's ABR really is working like its CBR, but:
- ABR bit allocation uses:
  * base amount from target bitrate, base_bits
  * bits from 320 kbps frame plus bitreservoir minus base_bits, depending on PE
- CBR bit allocation uses:
  * base amount from target bitrate, base_bits
  * bits from actual frame plus bitreservoir minus base_bits, depending on PE

MP3 Listening Test at 128 kbps

Reply #206 –
Maybe I did the same mistake that some others have done when they ask to include a specific codec or setting. I asked to include an additional encoder (LAME 3.97) without carefully considering if that would serve the purpose of this test. I just saw a great oppurtunity to actually compare 3.97 and 3.98 in a public test. I would like to make sure that it is safe to use 3.98 instead of 3.97. /mnt's report might indicate that with some samples the quality may be regressed (especially because /mnt used LAME 3.98 with a setting that gives a bitrate advantage. LAME 3.98 -V5 appears to produce higher bitrates than 3.97 -V5 --vbr-new).

In general, I have not seen many personal reports of 3.98 at "about 130-135 kbps VBR" during the development cycle. However, as I already said, it should be easy to me and others to personally test LAME 3.97 using the samples from the public test.

Quote from: Sebastian Mares on
As I said in a previous post - given the fact that Gogo is not developed any longer (at least quality-wise) and that VBR was not mature in 3.88 according to Gabriel if I recall correctly and therefore not recommended, I fancy the idea to test FhG CBR just to have a comparison between high quality encoders (LAME, FhG with Q switch and VBR, iTunes using best quality settings) and fast encoders (Helix VBR and FhG CBR without Q switch).

Do you have a reason to believe that FhG VBR -q 1 would provide better quality than FhG VBR -q 0 (aka default) ? My personal test didn't indicate anything like that, but naturally it is only a limited proof.

Personally I am not convinced that adding a CBR encoder would be a good idea, but I am not strictly against it. I have no opinion about FHG v. 4.1 CBR because I have not tested it in the CBR mode.

After all, I would be good with only four VBR contenders and the low anchor. It would make the test easier.

MP3 Listening Test at 128 kbps

Reply #208 –
Quote from: Alex B on
Do you have a reason to believe that FhG VBR -q 1 would provide better quality than FhG VBR -q 0 (aka default) ?


Yes, it is labled as high quality switch so I have the right to assume that. I am doing it just to be on the safe side since I don't have the ability to contact the developers and ask for whatever setting they recommend.

MP3 Listening Test at 128 kbps

Reply #209 –
Quote from: Sebastian Mares on
Yes, it is labled as high quality switch so I have the right to assume that. I am doing it just to be on the safe side since I don't have the ability to contact the developers and ask for whatever setting they recommend.

Unfortunately other users have not posted ABX results. Personally I can only trust my ABX results (TOS 8). Actually, is it impossible to reach the FhG developers? Surely they should be interested in this matter.


From the LAME 3.98 "news" thread:
Quote from: Sebastian Mares on
Do you guys recommend -V5.7 to be used in my test?

LAME 3.98 -V5 results 141.7 kbps with my test track set.

Unless Robert and Gabriel suggest something else I think we should proceed in the usual way:
1. Take the contenders that do not allow intermediate quality settings and find the average bitrate of all of these contenders.
2. Test and find the suitable settings for the other contenders. The settings that provide the closest bitrate match with the average bitrate from the "step 1" should be selected.

EDIT

A possible CBR encoder should be excluded from the step 1. Including it would lower the "step 1" average and give slight unfair advantage to the "step 1" VBR encoders.

MP3 Listening Test at 128 kbps

Reply #210 –
I think also this is more on topic here:

Quote from: Serge Smirnoff on
Quote from: Sebastian Mares on

Do you guys recommend -V5.7 to be used in my test?

I think NO coz there is no possibility to make all participating coders to produce equal resulting bit rates. So the use of real-life settings (resulting in acceptable bit rates) seems to be a reasonable compromise.

Hmm... what you mean by "real-life"?

The developers have obviously decided to let the average bitrate of LAME -V5 increase. I can't believe they would have not constantly tested the resulting bitrates. Someone could call this change cheating, but I think they have just redesigned the settings scale. Why should we not use the new scale and do what should be considered fair?

MP3 Listening Test at 128 kbps

Reply #211 –
Quote from: Alex B on
Hmm... what you mean by "real-life"?

The developers have obviously decided to let the average bitrate of LAME -V5 increase. I can't believe they would have not constantly tested the resulting bitrates. Someone could call this change cheating, but I think they have just redesigned the settings scale. Why should we not use the new scale and do what should be considered fair?


I meant that most people use integer values in practice ....

But right you are, - every listening test has its own fair policy. If this one already has some agreed procedure of choosing encoder settings, it should be kept, regardless of any "integer" values for sure.
keeping audio clear together - soundexpert.org

MP3 Listening Test at 128 kbps

Reply #212 –
Quote from: Serge Smirnoff on
... But right you are, - every listening test has its own fair policy. If this one already has some agreed procedure of choosing encoder settings, it should be kept, regardless of any "integer" values for sure.

I checked the three previous public listening tests:
Ogg Vorbis AoTuV 4.51 Beta -q 4.25 and Nero HE-AAC Jul 20 2007 -q 0.24
have been chosen in similar situations and the used procedure for finding the correct setting was like I suggested now.

If LAME is now supposed to be similarly "stepless" an intermediate setting can be used when it is closer to the test target.

EDIT

The test's "target bitrate" should probably be announced as 130-135 kbps or so.

Another option would be to pick LAME 3.98 -V5 as a reference and try to match the other encoder's settings with it. The test target would then probably be ~140 kbps.

MP3 Listening Test at 128 kbps

Reply #213 –
Quote from: Alex B on
Unfortunately other users have not posted ABX results. Personally I can only trust my ABX results (TOS 8). Actually, is it impossible to reach the FhG developers? Surely they should be interested in this matter.


I am very thankful for the time you invested in testing and although your conclusion is that there is no difference between the two settings, I am going to use the quality switch because of the aforementioned reasons. According to the Fraunhofer IIS page, FhG recommends using the maximum quality setting available in the encoder interface:

Quote
How do I achieve perfect audio quality using MP3?

[...]

2) [...]If you use a variable bitrate setting, choose the maximum quality setting available in the encoder interface.[...]

MP3 Listening Test at 128 kbps

Reply #214 –
IMO the bitrate difference should not be bigger than +-2.5 kbps.
I found that the only scenarios that will work are those two:

135-140 kbps, if there should be iTunes VBR and Lame -V5:

1) Lame 3.98: -V5
137 kbps

2) FhG 1.4 (Encoder-Library V04.01.00): -br 0 -m 3 -q 1
138 kbps

3) iTunes 7.6.2.9: http://img77.imageshack.us/img77/9634/itunesmp3nf1.png
135 kbps

4) Helix v5.1 2005.08.09: -V72 -X2 -SBT450 -TX0 -HF2
137 kbps

(lame 3.97)
127 (-V5 --vbr-new); 150 (-V4 --vbr-new)
those settings would disqualify lame 3.97



Or the ~128 kbps range:

1) Lame 3.98: -V5.8
128 kbps

2) FhG 1.4 (Encoder-Library V04.01.00): -br 0 -m 4 -q 1
128 kbps

3) iTunes 7.6.2.9: settings from above but CBR
128 kbps
hitting the 128 range with maximum VBR is not possible due to the lower limit at 128 kbps . Maximum VBR with 112 kbps doesn't work either (bitrate too low).

4) Helix v5.1 2005.08.09: -V62 -X2 -SBT450 -TX0 -HF2
128 kbps

5) Lame 3.97: -V5 --vbr-new
127 kbps


I'm not sure about the Helix settings at that low bitrate though.

MP3 Listening Test at 128 kbps

Reply #215 –
As most people so far want to see Lame 3.97 as the 5th candidate yielding 128 kbps seems to be the way to go.
lame3995o -Q1.7 --lowpass 17

MP3 Listening Test at 128 kbps

Reply #216 –
As always, I am going to build a bitrate table containing the average bitrate an encoder gave to all samples. If the difference between the encoder with the lowest average bitrate and the encoder with the highest average bitrate is greater than 10% of the target bitrate, I will think about either exchanging some of the samples or choosing different encoding parameters. The initial parameters will be chosen based on recommendations from developers and based on a large set of music tracks that should average to our target bitrate.

By the way, this means that in Raiden's example, the first scenario with LAME 3.97 -V5 --vbr-new would still be a valid solution. -V4 is also within the 10% range, but in this case, I would choose -V5 in order to compare the same quality setting between 3.97 and 3.98 and also because 127 kbps is closer to the average bitrate of the four oher encoders than 150 is.
 

MP3 Listening Test at 128 kbps

Reply #217 –
Raiden,

What test tracks did you use and what tool did you use for checking the bitrates?

MP3 Listening Test at 128 kbps

Reply #218 –
I would like to see lame 3.98 as the new recommended encoder. And to do so, tests has to be done. *vote for lame 3.97 AND 3.98*

MP3 Listening Test at 128 kbps

Reply #219 –
Quote from: Alex B on
Raiden,

What test tracks did you use and what tool did you use for checking the bitrates?

ooops... sorry! I should have put that info in the post above.
I haven't got many different genres on my PC (mostly electronic music) so this was just a quick test with a small 'representative' selection (just six albums, 375 minutes): Some electronic music/IDM, ambient, Jazz/NDW, Rock, and a radio play with lots of effects. I guess for modern, compressed music one has to add ~5 kbps to the values.
I've read the bitrate with foobar2000 (after fixing the VBR headers on the FhG files), averaged over time...

So no big conclusions, but there are a few:
- lame 3.98 MP3s are ~10 kbps bigger than their 3.97 counterparts at all quality levels, so direct comparison (i.e. 3.97 -V5 vs. 3.98 -V5) is not so good.
- FhG's VBR scale is not fine at all, whereas with Lame and Helix there is no problem. So FhG would likely be the 'bitrate reference'.
- iTunes VBR is very difficult. 112-VBR results in a bitrate of ~120 kbps, and 128-VBR in ~135 kbps, but I have not found an inbetween VBR setting for 128 kbps.

I'm very curious about bitrates from bigger testcases!

MP3 Listening Test at 128 kbps

Reply #220 –
As I said already, if the target bitrate is 128 kbps (which it is), a difference of 12.8 (or let's round to 13) kbps is allowed between contenders (again, on average, not per sample - per sample the difference can be higher).

MP3 Listening Test at 128 kbps

Reply #221 –
LAME developers, in case the bitrate difference between LAME 3.97 and LAME 3.98 is too big, is there anything I can do to increase the bitrate of LAME 3.97 without having a negative impact on audio quality?

Alex B, any chance you could post a bitrate table for your samples with LAME 3.97 -V5 --vbr-new to have a direct comparison?

MP3 Listening Test at 128 kbps

Reply #222 –
Quote from: Sebastian Mares on
LAME developers, in case the bitrate difference between LAME 3.97 and LAME 3.98 is too big, is there anything I can do to increase the bitrate of LAME 3.97 without having a negative impact on audio quality?
Or else, still include tried 'n' true 3.97 at -V5 --vbr-new, and play around with the new stepless quality settings on 3.98, to encode at - say - V5.5?

MP3 Listening Test at 128 kbps

Reply #223 –
Problem is that all other encoders also produce bitrates near the 140 kbps range. The only encoder that (according to Raiden - I didn't had time to test myself that is why I asked Alex B if he can also test) encodes even "lower" than the target bitrate is LAME 3.97.

Personally, I wouldn't mind a bitrate difference of 10 kbps (which is even less than 10%) since I assume the difference to be very low between the two LAME versions and if it's not, it is most likely caused by other factors not only the bitrate.

MP3 Listening Test at 128 kbps

Reply #224 –
Quote from: Sebastian Mares on
Alex B, any chance you could post a bitrate table for your samples with LAME 3.97 -V5 --vbr-new to have a direct comparison?

I have already tested it. LAME 3.97 -V5 --vbr-new produces about 134 kbps with my sample set. This has not changed since I tested 3.97 beta 2 for the 128 kbps multiformat test. As you can see from my previous table also 3.98 beta 5 was quite similar. After that the scale has changed and now 3.98 is different. The obvious solution would be to adjust LAME 3.98 to exactly match 3.97 or the test average, but as you said, we have some tolerance and naturally we should not use a setting that would decrease the quality if the next minor setting "step" should produce clearly better quality.

I would like to ask Robert and Gabriel if there are some fundamental changes when, for instance, the setting is changed from -V5 to -V5.5 or -V5.7. In other words, should the integer steps still be preferred for some reason or is the -V scale now truly stepless? It would be good to have info about that before the encoding setting for LAME 3.98 is decided.

I am going run a few additional tests and post a new table soon. My test set has usually been quite good in representing the "various" genre aka an average of pop/rock/jazz/electronic genres even when it has been compared with big encoded libraries, but in addition, I am gathering a modest set of classical tracks and I'll  publish those results in separate table.