HydrogenAudio

Lossy Audio Compression => MP3 => MP3 - Tech => Topic started by: Megaman on 2002-11-23 09:19:46

Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Megaman on 2002-11-23 09:19:46
I've made a short test with two LAME binaries:  Dibrom's 3.90.2 ICL latest compile and Mitiok's 3.92 latest compile.

I've uploaded a report here: LAME test (http://usuarios.advance.com.ar/jdoliva/LAME/Test.htm).

Read at your own risk (!) , I'm not much more than a newbie so my conclusions might be completely wrong , however I tried to do my best.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Dibrom on 2002-11-23 09:57:44
The only thing I'd be a little wary about is the conclusion that they don't sound different.  I've mentioned this before, but one of the changes made  after 3.90.2 was that /qrcd was no longer used in the Makefile.  Granted, it is "bad form" (or whatever you want to refer to it as) because it changes the rounding behavior of the code slightly, but even so, I have noticed on at least one occassion that there can be slight degredations in quality without it (I noticed this on fatboy).  I don't believe the LAME developers did any significant testing after removing this command from the makefile to make sure it did not impact quality.  This is one of the reasons I don't recommend using 3.92 right now.. since there is more room for quality degredation because of this slight change in the compile time options.  It's probably true that this won't affect most situations significantly, and that most people won't hear the difference anyway, but I still wouldn't make the conclusion that there is no detectable difference at all.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Megaman on 2002-11-23 10:14:29
Quote
The only thing I'd be a little wary about is the conclusion that they don't sound different.......It's probably true that this won't affect most situations significantly, and that most people won't hear the difference anyway, but I still wouldn't make the conclusion that there is no detectable difference at all.

Right....maybe the difference would be noticed only on very critical samples but of course if you want the best possible quality under any possible circumstances no matter if highly critical or not , it would be reasonable to choose your compile , assuming that you heard that slight quality degradation with Mitiok's compile.
I will check out the fatboy sample myself although I doubt that I have the adequate gear to do any quality ABX testing at all , so I guess in the end it will be a matter of trust .
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: robert on 2002-11-23 11:11:59
I cannot recommend the use of 3.90.2 if that binary was compiled with the /qrcd switch turned on. It simply uses the wrong rounding mode.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Dibrom on 2002-11-23 11:39:57
Quote
I cannot recommend the use of 3.90.2 if that binary was compiled with the /qrcd switch turned on. It simply uses the wrong rounding mode.

This is what I was talking about.

Ironically, this statement is probably not made with any basis in quality testing.

It's just "wrong", so it doesn't matter if it sounds better or not.  From a development point of view, this might be interesting, but from an end user point of view (something which LAME does not address very well.. tons of unnecessary experimental switches are an example of this), it is not.  Most of the people on this site who are looking for recommended compiles and who looking at the --alt-presets are doing so because they want the best quality.  They want something which has been verified via listening tests.  I doubt they would really care whether or not it uses the "intended" rounding modes if the actual end result is better.

EDIT: I do understand the need for code to behave a certain way, but with the way LAME is right now (basically no quality testing or control),  I don't think we can take anything for granted without testing its effects on quality -- this includes rounding behavior (in that, the behavior of the ICL compiles was changed after 3.90.2, but I don't think any quality testing was done by the people making this change).  The 3.93 release should be a lesson in this...

So basically what I'm trying to say is that regardless of how 3.90.2 works at a code level, regardless of rounding modes or anything which should work a certain way, I'm only making a judgement on how it actually sounds in practice -- I'm not going to make any assumptions.  I think this is, at least right now, the only true way to determine the actual quality of the output.  3.90.2 has been verified to be good.. 3.92 hasn't, and 3.93 has been verified to not be good.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: ErikS on 2002-11-23 12:20:32
Quote
The only thing I'd be a little wary about is the conclusion that they don't sound different.  I've mentioned this before, but one of the changes made  after 3.90.2 was that /qrcd was no longer used in the Makefile.  Granted, it is "bad form" (or whatever you want to refer to it as) because it changes the rounding behavior of the code slightly, but even so, I have noticed on at least one occassion that there can be slight degredations in quality without it (I noticed this on fatboy).  I don't believe the LAME developers did any significant testing after removing this command from the makefile to make sure it did not impact quality.  This is one of the reasons I don't recommend using 3.92 right now.. since there is more room for quality degredation because of this slight change in the compile time options.  It's probably true that this won't affect most situations significantly, and that most people won't hear the difference anyway, but I still wouldn't make the conclusion that there is no detectable difference at all.

I assume you made an ABX test... I can't hear this myself, but are there other persons who can pass an ABX on this sample? Can you repeat the test yourself and pass it again? What I mean: both Takehiro and Robert have now discouraged the use of this switch and it seems you are the only one who claims the opposite with one(?) abx test made some time ago. It would be good if your results on the ABX test was repeatable.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: JohnV on 2002-11-23 13:31:39
Quote
What I mean: both Takehiro and Robert have now discouraged the use of this switch and it seems you are the only one who claims the opposite with one(?) abx test made some time ago. It would be good if your results on the ABX test was repeatable.

Hmm, you seem to forget that the whole development of --aps was made using Dib's compiles. It was all based on listening tests. Granted, I think 3.92 is pretty much the same "quality level" though.

It's quite ironic that the Lame devs are so concerned about some rounding issue, and even recommend something based on it, but then the real problems seem to go unnoticed..
http://www.geocrawler.com/lists/3/SourceFo...323/0/10244994/ (http://www.geocrawler.com/lists/3/SourceForge/7323/0/10244994/)
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Megaman on 2002-11-23 17:49:20
I agree with Dibrom's statement: what the end user wants is the highest possible perceptual quality no matter the rounding mode.Of course , if the wrong rounding mode results in a binary which delivers files of perceptually superior quality (even if only on critical samples) is something that needs to be backed up with several , seriously made listening tests.

Edit: I'm absolutely paranoid about my english
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: ErikS on 2002-11-24 09:21:55
Quote
Hmm, you seem to forget that the whole development of --aps was made using Dib's compiles. It was all based on listening tests. Granted, I think 3.92 is pretty much the same "quality level" though.

So?

1. Listening tests don't mean anything if they are not reproducable....  Can you hear any improvement with this compiler switch?

2. If the switch was used throughout the development of the alt-preset:s then it is very possible that something else was tuned to compensate for that "wrong" rounding mode, and when you switch back to the "correct" rounding mode it overcompensates and throws something else off. If this is the case the right approach is to fix lame so it can use the correct rounding mode. Many people don't use icl compiles - why should they suffer?
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Megaman on 2002-11-24 09:31:03
I think that 99% out there use ICL compiles.
Mitiok's official compile is compiled with ICL (am I right?).I guess at least 98% of non-HA members use Mitiok's compile.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: ErikS on 2002-11-24 09:54:00
So 99% of the people use windows? I doubt it.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Dibrom on 2002-11-24 10:09:47
Quote
Quote
Hmm, you seem to forget that the whole development of --aps was made using Dib's compiles. It was all based on listening tests. Granted, I think 3.92 is pretty much the same "quality level" though.

So?

1. Listening tests don't mean anything if they are not reproducable....  Can you hear any improvement with this compiler switch?

2. If the switch was used throughout the development of the alt-preset:s then it is very possible that something else was tuned to compensate for that "wrong" rounding mode, and when you switch back to the "correct" rounding mode it overcompensates and throws something else off. If this is the case the right approach is to fix lame so it can use the correct rounding mode. Many people don't use icl compiles - why should they suffer?

I'm not really going to spend much effort on this for the following reasons.

1.  I don't have time.
2.  I'm tired of arguing pointlessly about LAME.
3.  I simply don't care enough anymore.

To address your first point, I agree with you that listening tests are pointless if they are not reproducable.  I do not plan on reproducing this particular listening test anytime soon though.  You can ask the other LAME developers if they would like to do so though.

As to whether or not my statement means anything aside from that in regards to this matter, well, I've done more reproducable listening tests for LAME than any of the other developers have ever done.  Most of them were blind, and nearly all of my results have been verified by other parties.  The fruits of my labor in regard to past listening tests can be heard through the --alt-presets.

Additionally, Robert and Takehiro are not authorities on quality.  It was commits from both of them to CVS which have resulted in 3.93's broken quality.  Given that, it's understandable for me to question their statements about what is quality and what isn't.  Furthermore, I've never seen an effort from either of them to do comprehensive blind testing with results verified by any sort of community.  This is not a reflection on their character, so please don't take this out of context.  I get along with these guys fine on a personal level, I'm only attempting to show why I don't believe their comments on this matter hold a lot of weight.

As for your second point... if they feel that there is a problem with the rounding mode in /qrcd, and that the --alt-presets have been tuned with this in mind (which is probably true, except for the possibility of a "problem" with the rounding mode.. this is unclear.. especially if the rounding mode led to higher quality), then they are more than welcome to correct the problem themselves.  Unfortunately, I don't think this will be easy without some extensive listening tests, which I don't think either of them has the time or desire to do -- it's easier to just claim that it's "wrong" and be done with it.

Nobody is made to "suffer" with the --alt-presets.  I am not forcing anyone to use anything, I am only making a recommendation.  I tuned the --alt-presets according to the code that was already in LAME (the /qrcd switch was put there by the LAME developers, not I.  Only after the --alt-presets were implemented was it removed), so it is not my responsibility to make sure that they still work after someone else changes their functionality.  It should be the responsibility of those who have made the change to put forth the effort to verify that quality has not taken a hit.  If that means asking me to help, that's fine.. but I was not asked to help.  Some may say that it is my responsibility, because I'm a LAME developer.  I would disagree with this.  LAME is, accordingly, driven by consensus.  My input into this consensus has always been ignored.  Because of this, and because of that fact that I don't think the other developers take me seriously (and thus, don't really consider me a peer), I don't consider myself to be a LAME developer.  To be honest, I'm not even sure why the --alt-presets ever were included in the first place, other than it was just another thing that could be added to LAME.

Ultimately I agree with your assessment that if /qrcd improves quality, then changes in LAME itself should be made.  Who is going to do this though?  I don't think the people complaining about /qrcd are going to do it.  Complaining to me about it isn't going to accomplish anything, because I have no control over LAME.  Furthermore, there is very little initiative or motivation in the LAME project to improve quality at such fine levels.  In addition, even if we could prove that there was a beneficial change that could be incorporated into LAME, you'd likely meet with much resistance to implementing your idea (basically, it'd never happen).

I recently had a discussion with Takehiro about the future of LAME and about large scale changes that should be made to the code.  He basically told me that LAME is too mature and too much of the code is interdepedant for there to be any more large scale modifications at this point.  He mentioned that he didn't believe LAME would ever improve or change significantly at this point.  Although I don't agree with the ideas that change can't happen, I do agree with his assessment of this particular situation, given all that has happened (and that which hasn't) over the past year.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Dibrom on 2002-11-24 10:11:02
Quote
So 99% of the people use windows? I doubt it.

I guess you haven't done much research.  ICL is available for free on Linux, so that thows the "only windows" bit out.  I don't know if it compiles LAME, but the possibility it certainly there.

And, FWIW, gcc is not a very good compiler speed wise.  I get much better results running the 3.90.2 compile under WINE then I do compiling it with gcc.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: dev0 on 2002-11-24 10:16:31
Did you try GCC 3.2? From what I've heard it's almost as fast as ICL.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Benjamin Lebsanft on 2002-11-24 10:39:18
are there win32 builds ?
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Dibrom on 2002-11-24 10:40:34
Quote
Did you try GCC 3.2? From what I've heard it's almost as fast as ICL.

No, I've tried GCC 3 or 3.1, forget which.

I'd be very surprised if GCC 3.2 was as fast as ICL though.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: john33 on 2002-11-24 12:46:38
Quote
Quote
Did you try GCC 3.2? From what I've heard it's almost as fast as ICL.

No, I've tried GCC 3 or 3.1, forget which.

I'd be very surprised if GCC 3.2 was as fast as ICL though.

It seems to depend rather on the application. The GCC3.2 compiles I made of the OggVorbis apps ran as fast, if not marginally faster in some cases, than the ICL compiles. However, I did produce 3.2 compiles of LAME for my own testing and they were significantly slower than the ICL compiles. So I guess the only conclusion to be drawn is that it depends on how the code is written.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: ErikS on 2002-11-24 19:43:59
Quote
I guess you haven't done much research.  ICL is available for free on Linux, so that thows the "only windows" bit out.  I don't know if it compiles LAME, but the possibility it certainly there.

My bad. I assumed you needed MSDev to use ICL, plus I never heard that neither MSDev nor ICL exitsted for any other OS. But that is because I didn't look.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Gabriel on 2002-11-25 07:36:28
There is ICL for linux, but I'm not sure that it's free. Under win32, at least, it's not free at all...
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Dibrom on 2002-11-25 08:31:42
Quote
There is ICL for linux, but I'm not sure that it's free. Under win32, at least, it's not free at all...

It's free and "unsupported", but I think you do have to register at their developer website to download it.  Gentoo, which is a source based distro, has been working on ICL support (to compile the entire distro) for quite some time.. and I'm pretty sure they wouldn't be doing this if it wasn't available freely.

And I'm aware that the win32 version is non-free.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Gabriel on 2002-11-25 09:20:53
Free under linux...That's interesting...
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: john33 on 2002-11-25 12:14:37
Quote
Free under linux...That's interesting...

And they only want your name, email address and country to download!
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: joro_09 on 2002-11-25 12:26:32
Hi,
I've been reading different forums and recommendations about quality MP3 encoding for about 2 months up to now, but I still hesitate what switches to use. To be clear, my intention is to encode all my music in MP3 archive with best possible quality, but to be able (if I decide for some reason to do so) to decompress it again and write it to Audio Discs, without furthermore loss of quality.
So here are some questions for you:
1. Some people say --alt-preset insane is the best quality you can get from MP3 format (at least for LAME 3.92). Other people say extreme is better because of its intelligence during coding simple and harder parts of a song (e.g. extreme is better than insane in passing on the stereo panorama). What is the truth?!
2. Is there any official place where developers classify, in order of their quality, the LATEST, TESTED and STABLE switches (presets) for command  line? It is very confusing to me to see different developers recommending their own switches and to choose between them… Which is best!?
3. These days I understood LAME 3.93 have been released. Is it safe to use it, or should I wait for further tuning? Or is the lame3.90.2 better?
Thanks in advance for helping me.

Sincerely, George
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: KikeG on 2002-11-25 12:54:23
- According to my experience and others, --alt-preset insane is the best quality you can achieve, better than with alt-preset extreme.

- 3.93 is not advised, better use 3.90.2
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Chun-Yu on 2002-11-25 13:30:06
BTW: The Intel C++ Compiler 7.0 has recently been released (but I haven't had time to try it yet).  The best thing about it is that they've added a "student license" that's only $30 - "Student pricing is for non-commercial licenses only. The licenses are individual, unsupported user licenses without a timeout on the license."
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: tangent on 2002-11-25 14:13:10
Quote
Quote
Free under linux...That's interesting...

And they only want your name, email address and country to download!

Some Mr Gaylord Fooker  with email thisisso@fake.com  from Zimbabwe just downloaded it.
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: john33 on 2002-11-25 16:03:18
Quote
Quote
Quote
Free under linux...That's interesting...

And they only want your name, email address and country to download!

Some Mr Gaylord Fooker  with email thisisso@fake.com  from Zimbabwe just downloaded it.

Know him well!!
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Dibrom on 2002-11-25 20:49:36
Quote
1. Some people say --alt-preset insane is the best quality you can get from MP3 format (at least for LAME 3.92). Other people say extreme is better because of its intelligence during coding simple and harder parts of a song (e.g. extreme is better than insane in passing on the stereo panorama). What is the truth?!


Extreme should never be better than insane.

Quote
2. Is there any official place where developers classify, in order of their quality, the LATEST, TESTED and STABLE switches (presets) for command  line? It is very confusing to me to see different developers recommending their own switches and to choose between them… Which is best!?


No.. and this is one of the most hotly contested issues lately.  There is no framework or structure for anything like this in place from the official developers, and it doesn't seem as if there is a lot of motivation for it either.  HA provides recommended command line switches and recommended compiles though.  These have been very extensively tested and are surely your best bet.  They are not "official" recommendations though from the actual LAME development team.  I doubt you'll see something like this anytime soon (if ever) though because it does not appear to be a focus of the LAME development ideaology.  If this is the type of thing you're really after in an encoder, I suggest looking at something like MPC if you aren't tied to the MP3 format.

Quote
3. These days I understood LAME 3.93 have been released. Is it safe to use it, or should I wait for further tuning? Or is the lame3.90.2 better?


No.  3.93 has had some quality problems due to a lack of proper testing before release.  The best compile to use is 3.90.2 (which is an unofficial compile, btw).
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: joro_09 on 2002-11-27 15:08:39
Thanks a lot to all for helping me with my hesitation !
But is the v3.90.2 better even then v3.92 (that I have and is marked as stable) ?  Because I can't find a link to it anymore !

Tks, George
Title: LAME compiles test: Dibrom's & Mitiok's
Post by: Volcano on 2002-11-27 15:58:18
It's not necessarily that much "better", but it is tested properly, so you can be certain the alt-presets work the way they should.

There's a link here (http://www.hydrogenaudio.org/forums/index.php?s=&act=ST&f=15&t=478) (read the sticky threads! ).