I would like to ask you to submit suggestions what MP3 encoder to include in my upcoming test, as well as suitable settings to produce ~128 kbps MP3 files. I would like to use only one setting per contender, so no suggestions like "LAME with -V 5 and -b 128 to test VBR vs. CBR". If that is really needed, a possible solution could be having -b 128 as contender and -V 5 or a higher setting as high anchor. Low anchor could be Shine, Blade or some very (very) old FhG encoder.
The goal of the test is to find out if fast encoders are really worse than LAME and if yes, how much worse. Some ideas for contenders: Gogo, LAME, iTunes, Helix, some FhG from Nero or Adobe or MMJB (in case I can still download it - heard that Y! stopped developing it) or WMP...
Also, since MP3 supports real CBR, do you think I should favor CBR over VBR for possible portability reasons (iPods not dealing well with LAME VBR)?
Bear in mind that LAME devs are the only persons to ask for recommended settings. I tried getting in touch with Helix folks some months ago but I had no luck (they didn't know what to recommend). Obtaining recommendations from FhG isn't possible neither.
Edit: One more thing - the max no. of contenders should be 4 + 2 anchors.
Helix has a vbr quality mode for 128k, but FHG lowest setting is 150..160k (m3) which rules it out. Alternative for FHG is ABR mode for 135..140k (m4), but would that bitrate be unfair in a 128k test ?. So we can do FHG / Helix CBR 128k :
mp3sencoder -br128000
hmp3 -B64
gogo -b128
ABR:
mp3sencoder -br0 -m4
hmp3 - ?????
gogo -b135 -a
Note that Nero is using multiple MP3 encoders depending on the application. The one from FhG we use in one place is AFAIK the same as what they have as demo on their website (unless that one is somehow limited, but doesn't look like it).
I would like to see AAC 96 kbps CBR LC (used in previous test as high anchor) as a low anchor in this test.
This would be helpful to have some connection of results for 48 kbps and 64 kbps with this test.
But I suppose most people will be against it.
I would like to see AAC 96 kbps CBR LC (used in previous test as high anchor) as a low anchor in this test.
This would be helpful to have some connection of results for 48 kbps and 64 kbps with this test.
But I suppose most people will be against it.
Yes this would be great, also because I hear the phrase "AAC 96k is equal to MP3 128k" thrown around a bit and it would be interesting to see if it's in any way true.
I would like to see AAC 96 kbps CBR LC (used in previous test as high anchor) as a low anchor in this test.
This would be helpful to have some connection of results for 48 kbps and 64 kbps with this test.
But I suppose most people will be against it.
Yes this would be great, also because I hear the phrase "AAC 96k is equal to MP3 128k" thrown around a bit and it would be interesting to see if it's in any way true.
People have the wrong idea regarding this. it depends where the quality bump is. For mp3 there is a big one from 96 > 128k , smaller from 128 > 160k and yet smaller from 160 > 192k. The other codecs are similar , converging more or less @ 128k but drop quality more gracefully @ bitrate < 100 k.
So saying ogg/aac 96k = mp3 128k is a possibility (IMO 96k will be worse than LAME -V5). We can't use these assumptions for high bitrates: 170=200, 256=320 etc etc, because once you pass the big quality bump you get less and less as you bump up the bitrate.
Anyway, I would like to see the mp3s encoder from FhG as we are using this since recently and it would be good to see how it performs compared to LAME. We only use CBR at the moment.
Anyway, I would like to see the mp3s encoder from FhG as we are using this since recently and it would be good to see how it performs compared to LAME. We only use CBR at the moment.
Me too.
96 kbps LC-AAC might be too good as low anchor. Consider that the low anchor has to be significantly worse than the rest.
IMO we should use with any encoder that setting which is a priori supposed to yield the best result. This can be quite disputable but we should try it.
Helix has a good VBR mode, so with Helix I suggest to use -V55 or -V60 which according to a listening test I did some time ago matches 128 kbps on average quite well. According to that (restricted) test we should use the default options (with the exception of -X2).
The problem with Lame is: 3.97 or 3.98b5+? With this only exception I suggest we use both versions because of their practical importance and in order to see what things are like with these versions.
With FhG IMO we shouldn't use VBR. According to my experience FhG's VBR mode is not attractive qualitywise.
Sorry, but I maybe missed previous discussions where it was discussed why low anchor has to be significantly worse. Can you please give a rationale about this?
IMO it is enough that it is worse than almost all contenders in almost all samples. There was also a case in 64 kbps where I would prefer low anchor over one of contenders.
96 kbps LC-AAC might be too good as low anchor. Consider that the low anchor has to be significantly worse than the rest.
I agree to that one. Considering the results of the 96 kbit/s AAC anchors of the 48 kbit/s and 64 kbit/s multiformat tests, it definitely won't be an appropriate low anchor in this case. I vote for Shine @128 kbit/s as low anchor.
Reason: Shine's an extremely simple encoder, which could be called a reference implementation for people starting to develop own encoders. It would be a good indicator to see how much better optimized encoders perform compared to a very basic one like Shine. Lately I had a few arguments with technically unexperienced people who refused believing me that the quality of MP3s can highly vary, depending on the encoders used for creating them. For these guys MP3 encoding is a fix matter that can't be altered/optimized in any way. I guess that's a major reason for the many 128 kbit/s full stereo Blade MP3s still found throughout the internet.
IMO it is enough that it is worse than almost all contenders in almost all samples.
Judging by the multiformat test results I mentioned there's no clear proof that 96 kbit/s AAC is actually worse than the contenders for this test. In this one (http://www.listening-tests.info/mf-64-1/results.htm) it resulted at 4.59, i.e. on many samples it was transparent for a lot of listeners. How should you distinguish a transparent low anchor from the contenders and the high anchor?
Sorry, but I maybe missed previous discussions where it was discussed why low anchor has to be significantly worse. Can you please give a rationale about this?
IMO it is enough that it is worse than almost all contenders in almost all samples. There was also a case in 64 kbps where I would prefer low anchor over one of contenders.
If the low anchor is on par with the contenders, or only slightly worse, then there is no need for an anchor at all. An anchor should be clearly distinguishable from the rest.
Now, doesn't this look really sweet: http://www.rjamorim.com/rrw/l3enc.html (http://www.rjamorim.com/rrw/l3enc.html)
what about the mp3 encoder from iTunes ? dunno if it's FhG-based...
what about the mp3 encoder from iTunes ? dunno if it's FhG-based...
I should like to see the iTunes encoder included, not for any technical reason but because it is so prevalent. It would be convenient for me to use it, instead of Max + Lame, and I'd like to know how the quality stacks up -- and I guess I'm not the only one.
IMO we should use with any encoder that setting which is a priori supposed to yield the best result. This can be quite disputable but we should try it.
Helix has a good VBR mode, so with Helix I suggest to use -V55 or -V60 which according to a listening test I did some time ago matches 128 kbps on average quite well. According to that (restricted) test we should use the default options (with the exception of -X2).
The problem with Lame is: 3.97 or 3.98b5+? With this only exception I suggest we use both versions because of their practical importance and in order to see what things are like with these versions.
With FhG IMO we shouldn't use VBR. According to my experience FhG's VBR mode is not attractive qualitywise.
I say 3.98 and FHG @ 128 vbr mode is really ABR. The FHG cbr is also like a freeformat.
Here's my two cents' worth,
FhG, Nero or WMP11 ?
- muaddib, please make some in-house queries about the exact details and version of FhG. Has anyone inside Nero actually compared the surround version against the presumably older "standard" version which included in WMP11?
Helix, VBR -- the exact -V value must be tried with a large amount of complete tracks (I can contribute).
- what is the last released source code version? AFAIK, the compile at Rarewares has not changed since it was put online.
LAME, VBR (whatever version and setting the developers recommend)
- 3.97 vs. 3.98 should be tested in private tests. The differences are probably not significant unless 3.98 is seriously regressed because of the developers' mistakes. I can promise to do a private test after this public test using the same samples. It would be relatively easy after doing the public test and finding out which kind of problems and where exactly the selected samples produce (which is going to be tough).
GoGo, ABR ~135 kbps (would produce a similar bitrate with LAME -V5)
- according to my limited personal testing this setting is slightly better than CBR 128. Perhaps I, halb27, shadowking, muaddib and others interested could select a few samples and compare these encoding modes in a quick "semi-public" pretest...
iTunes, CBR or VBR 128 kbps
- must be quite popular, is quite fast and has supposedly lower quality than LAME. Is anyone else interested?
(to kurtnoise: it is not FhG)
Low anchor, Shine 128 kbps
- this worked fine in the previous 128 kbps multi-format test. Even a very old FHG at 128 kbps would probably be too good. iTunes AAC at 96 kbps would most certainly be too good.
(To muaddib: The intention of a poor quality low anchor is to introduce the testers with typical encoding artifacts so that they can more easily try to pinpoint similar, but less pronounced artifacts that the contenders possibly produce.)
High Anchor, a surprise suggestion: FhG, CBR 192 kbps
- if it is good enough (I have been privately testing LAME -V2 variants too much lately and I would like to know how good FhG is at higher bitrates. Would it be transparent for the majority of testers?)
Edit: several typos...
High Anchor, a surprise suggestion: FhG, CBR 192 kbps
- if it is good enough (I have been privately testing LAME -V2 variants too much lately and I would like to know how good FhG is at higher bitrates. Would it be transparent for the majority of testers?)
Sounds like a good choice to me, even with FhG already being included as a contender. The two FhG results could be very interesting for the many users out there, who rip their CDs with widely available commercial software, e.g. Windows Media Player. I don't have any figures available, but to me it doesn't look like the majority of people goes for alternative software, that uses a bundled version of LAME for MP3 encoding.
Alternatively, if Nero's/WMP's FhG encoder was used as the High Anchor, we could also go for the MP3 Surround encoder as a contender instead.
Because of the meaning of Lame I still think it's worth while to test 3.97 as well as most current 3.98.
But if I'm the only one who thinks like that I suggest to use 3.98 cause it's the latest version and close to final.
As for iTunes mp3 I also like to see it tested.
FhG, Nero or WMP11 ?
- muaddib, please make some in-house queries about the exact details and version of FhG. Has anyone inside Nero actually compared the surround version against the presumably older "standard" version which included in WMP11?
I'm pretty sure WMP does not use the new 4.0 series of FhG encoders. Nero does, but I'm not sure since what version (or if we even included that already). FhG claims their 4.x version is completely new compared to 3.x.
As said, I do not think the command line evaluation from the FhG website uses anything different than the SDK Nero uses, but I might be mistaken. If you want me to check it, let me know.
It support CBR and since recently also VBR (not enabled in Nero), there's also a quality/fast switch.
We are also still using the CodingTechnologies MP3Pro encoder in some places. I haven't heard anyone about that one yet (as it also does normal MP3).
The WMP11 encoder is currently this:
MPEG Layer-3 Codec Version 3.4.0
Fraunhofer IIS MPEG Layer-3 Codec (professional)
Copyright (C) 1996-2004 Fraunhofer IIS
The great thing about this codec is that it can used through Nyaochi's "acmenc" command line tool (after a small registry edit). Acmenc can include a LAME info tag style header which contains info for gapless playback. It is easy to configure foobar to encode through acmenc. I have used it as a faster option to LAME when I am in a hurry.
I wonder if Nyaochi could create a modified version which could be used with FhG's exe encoder (for creating a compatible gapless playback info header instead of FhG's quite useless newly introduced header).
*Low anchor:
AAC-LC @96 is not a suitable low anchor for such a test, IMHO. (anchor should not be used as a contender)
Regarding Shine, well, now that everyone seen how lame is Shine I think that it's perhaps a too low anchor. I would rather suggest either Blade or L3enc 2.72
*CBR vs VBR: hardware compatibility is a non-issue now. Every player should be able to properly play mp3 streams (even Apple is reported to having fixed Ipod's buffering issues with vbr)
*Lame version: I'd suggest 3.98 rather than 3.97. 3.98 is representative of current Lame, while 3.97 is from 1 year ago, and is unlikely to be updated/maintained by anyone.
dBPoweramp uses the Fhg IIS 4.0.3 encoder as well. Is this the surround sound encoder? Does Nero use 4.0.3 as well? What versions of Nero offer this encoder? WMP, as someone showed, uses Fhg 3.x.
I'm not sure of any Mac applications that can use Fhg 4.0.3. I asked Fraunhofer and they replied Cubase. Well this "only" costs about $1000!!!
dBpowerAMP could be an option, indeed. However, it seems to support only CBR encoding.
I thought the consensus was that Fhg only does CBR very well and that VBR still left much to be desired? I may be wrong though.
A bit off-topic, but still valid to the concerns of those doing the listening test -
which encoders can do gapless on iTunes and the iPod. Apple's own MP3, iTunes AAC, LAME MP3 can. What about Fhg MP3, Nero AAC?
dBPoweramp uses the Fhg IIS 4.0.3 encoder as well. Is this the surround sound encoder? Does Nero use 4.0.3 as well? What versions of Nero offer this encoder? WMP, as someone showed, uses Fhg 3.x.
What do you mean by IIS 4.0.3 ? Is it available somewhere?
The current command line encoder reports this version info:
********************************************************************
* *
* Fraunhofer IIS MP3 Surround Commandline Encoder V1.4 *
* *
* *
* Encoder-Library V04.01.00 (build 2007-05-18) *
* *
* *
* (c) 1996 - 2007 Fraunhofer IIS *
* (c) 2004 Fraunhofer IIS and Agere Systems Inc. *
* All Rights Reserved. *
* This software and/or program is protected by copyright law and *
* international treaties. Any reproduction or distribution of *
* this software and/or program, or any portion of it, may result *
* in severe civil and criminal penalties, and will be prosecuted *
* to the maximum extent possible under law. *
* *
********************************************************************
EDIT
I found that Illustrate advertises the encoder as 4.0.3:
http://www.dbpoweramp.com/codec-central-mp3-fraunhofer.htm (http://www.dbpoweramp.com/codec-central-mp3-fraunhofer.htm)
It looks like Fraunhofer does not provide their latest codec version here:
http://www.all4mp3.com/tools/sw_fhg_cl.html (http://www.all4mp3.com/tools/sw_fhg_cl.html)
In general, I dont think we have any reliable information about the FhG v.3.4.0 vs. v.4.0.x quality differences. Has Fraunhofer actually done any development work with the standard stereo mode since the IIS 3.4 version?
I guess we can't expect Fraunhofer to answer.
Perhaps Spoon could say something. He must have some insider information.
I don't know the exact version numbering, but we have something that starts with 4 and ends with 10 and VBR was only supported since the version that starts with 4 and ends with 4... IIRC
I just noticed that the command line encoder is 4.01.00 not 4.0.1. So perhaps FhG means 4.1.0.
I'd suggest that we disqualify Fraunhofer because of misleading version numbering.
Here's my two cents' worth..
I pretty much agree here. I'd like to see FhG (no particular version preference), LAME (3.98 preferable over 3.97) and iTunes. Shine seems like it would do well as the low anchor (though perhaps at a lower bitrate than 128kbps -- I'm not familiar enough with Shine to have an idea how it sounds at 128kbps), and FhG at 192kbps sounds like a fitting high anchor. I think it might be more fitting to use both VBR and CBR formats in this test, when appropriate. Naturally, opinion may vary on this to a high degree, but I agree with others that the hardware issues are being eliminated at a fairly brisk pace, and each encoder should use whatever method it's stronger with and/or whatever method is more widely used.
... Shine seems like it would do well as the low anchor (though perhaps at a lower bitrate than 128kbps -- I'm not familiar enough with Shine to have an idea how it sounds at 128kbps) ...
This is what the creator of the Shine encoder says:
*Low anchor:
AAC-LC @96 is not a suitable low anchor for such a test, IMHO. (anchor should not be used as a contender)
Regarding Shine, well, now that everyone seen how lame is Shine I think that it's perhaps a too low anchor. I would rather suggest either Blade or L3enc 2.72...
According to Gabriel Shine is intended to be the most basic MP3 encoder that can produce standard conforming MP3 files. It is just an educational tool.
I must check how it was ranked in the 128 kbps multi-format test. I don't recall it being too bad, but perhaps I remember wrong.
In any case, I am afraid that even the oldest release version of FhG is not suitable at 128 kbps. It wasn't that bad. I think it already included advanced things like bitreservoir, joint/intensity stereo and short/long block switching. It kind of created the myth of CD quality MP3 at 128 kbps.
Perhaps Blade would be a better low anchor, but even Blade may be surprisingly good with tonal samples that don't have sharp attacks.
In that case, I'll try to do some evaluations with both Shine and Blade tonight versus LAME before forming a real opinion.
Given my understanding of the purpose of the low anchor, so long as it's is easily identifiable, it doesn't truly matter how poor it sounds. According to Sebastian, way back in 2006, in fact, "..a low anchor is supposed to sound really bad, it should be the worst", so Shine would seem suitable (i.e., "really bad") for that if quality at 128kbps is as poor as you say.
In any case, I'll give 'em both a listen and see how I feel after doing so.
In that case, I'll try to do some evaluations with both Shine and Blade tonight versus LAME before forming a real opinion.
Given my understanding of the purpose of the low anchor, so long as it's is easily identifiable, it doesn't truly matter how poor it sounds. According to Sebastian, way back in 2006, in fact, "..a low anchor is supposed to sound really bad, it should be the worst", so Shine would seem suitable (i.e., "really bad") for that if quality at 128kbps is as poor as you say.
In any case, I'll give 'em both a listen and see how I feel after doing so.
Shine was the low anchor in this test:
http://www.listening-tests.info/mf-128-1/results.htm (http://www.listening-tests.info/mf-128-1/results.htm)
Instead of wasting your time testing Shine vs. Blade you could try GoGo 3.13 (http://www.rarewares.org/mp3-others.php) with the following settings and post your findings here.
-b 135 -a -q 2
-b 135 -a
-b 128 -q 2
-b 128-b 135 -a = an ABR setting for an average bitrate of 133 kbps = about the average bitrate of LAME -V5
-b 128 = CBR 128
-q 2 = a slightly slower and possibly higher quality settingBTW,
We had a 14-page discussion last year before this 128 kbps MP3 test was postponed:
http://www.hydrogenaudio.org/forums/index....showtopic=47313 (http://www.hydrogenaudio.org/forums/index.php?showtopic=47313)
During the discussion Sebastian decided to not use a high anchor. Perhaps without a high anchor we could have five contenders (LAME, GoGo, FhG, Helix and iTunes).
What do the others think about the idea of dropping the high anchor?
Well, being an anchor, it wouldn't be that much of a nuisance, would it? FhG @192kbps (maybe even higher, like 224/256) seems like a good choice, since it can serve its goal well and probably showcase the drastic leap in quality between it and LAME.
(Then again, I visit HA so rarely nowadays so my opinion doesn't really matter.)
I think an ancient yet reasonably popular historic encoder like Radium or SoloH could serve well as a potential low anchor even at the same bitrates. I am interested in seeing how far along MP3 has come in the past 7-8 years. I see Blade's been suggested, which isn't bad except it was never realistically considered a high quality solution at any time. It was just free and quickly available after ISO's code release. Radium was *the* MP3 codec for some time, for longer than it had any right to be. I suggest that if there's hesitation about its possible competitiveness with respect to being a low anchor, then we aren't convinced whether MP3 has truly made gains since the late 90s. In any case, I think the choice should be something that mattered at some time, not an artificially lousy choice like Shine. I'll take Xing circa 1998 over that.
I agree with eliminating the high anchor. It's pretty clear from recent tests that its value approaches 5 at these bitrates.
Also, I'm for the exclusion of LAME 3.97 if it bumps an otherwise worthy contender. Far more relevant than possible LAME regression is its performance compared to other encoders in general. I doubt minor issues between point releases would make or break comparisons between fundamentally different software. LAME 3.97 vs. 3.98 is important, but I think this test would be too crowded for meaningful comparisons.
I would agree with foregoing the high anchor. You could have two low anchors, one worse than the other.
I just noticed that the command line encoder is 4.01.00 not 4.0.1. So perhaps FhG means 4.1.0.
I'd suggest that we disqualify Fraunhofer because of misleading version numbering.
Yes, it's 4.1.0 I just checked. This is the version that adds VBR (with VBRI headers) and original file length support.
Gabriel suggested 3.98 so I am going to include that instead of 3.97. Gabriel, do you recommend -V 5 --vb-new or some other setting for ~128 kbps?
iTunes and Helix are also featured for sure. Should I use both in VBR mode?
Fraunhofer is also an encoder I would like to include - same question now: CBR or VBR?
Is Gogo still used or did people switch to Helix / FhG / iTunes for quick encoding?
ff123 and other pros, what do you think about dropping the high anchor and using the five contenders LAME, FhG, iTunes, Helix and Gogo at 128 kbps?
As for the low anchor, I don't see any point in using Blade. I could imagine using Shine because it's very simple, Radium because it was very popular years ago or l3enc because of its age.
I just reread the complete last year's thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=47313). The codecs and settings were decided (almost) and we were talking about the sample choices on the last two pages.
We discussed about the need of a high anchor on the page three. The conclusion was to not include one.
The selected low anchor was FhG l3enc v.1, but I was concerned about its possibly too high quality.
Here are a few selected quotes from that thread:
I have had an impression that Gogo should be used with ABR at a bitrate like ~130 kbps. It is based on LAME 3.88 and VBR wasn't very matured at that time. Even LAME 3.9x ABR has been used at lower bitrates for example by Guruboolez until the most recent versions. I mentioned earlier that I have used command lines like this: "-a -b 192 -m j -q 2". For this test the command line could be like "-a -b 135 -m j" and the quality option 0, 1 or 2. (I have used -q 2 without any testing just because LAME 3.88 -h was mapped with the -q 2 option and it produced a good balance between the quality and speed).
Gabriel, were you recommending ABR over VBR in the LAME 3.88 days?
Fur such bitrates, definitively ABR. Note: it's likely that abr will undersize its target bitrate.
For Helix I would actually recommend using Real Player 10.5. I just tried it (what an awkward experience) and it seems to have only two 128 kbps MP3 options. CBR and VBR. Once again the 128 kbps VBR bitrates seem to be within the test target (~ about 135 kbps).
One would like to assume that Real Networks has tweaked their flagship program to produce best possible MP3 quality. This may not be practically true, but it would be an acceptable and a better reasoning for the used Helix setting than anything else I have seen in this thread so far.
... Perhaps it would be good to just use the latest WMP and the options MS has decided to make available. After all they should know what is best for their MP3 encoder.
The test idea would be then like what the big guys offer vs. LAME & Gogo at about 130 kbps. The big guys are naturally MS, Apple and Real. (I don't know how big share Real has anymore, but they were one of the first and Real Player is still very well-known.)
I am testing Gogo because it's pretty fast and I want to see how it competes against LAME and other fast encoders like iTunes, Helix and FhG.
Discussion about encoder decision is really closed now. All additional codec requests or questions regarding why codec A was used over B are going to be ignored. Current list of encoders is:
LAME 3.97: -V 5 --vbr-new
iTunes 7.0.1.8: 128 kbps, VBR, highest quality, automatic sampling rate, automatic channel mode, "Stereo (joint)" stereo mode, intelligent, filter frequencies below 10 Hz
RealPlayer 10.5: 128 kbps, VBR
Gogo 3.13: have to check which ABR settings are suitable. Should I manually specify JS coding? What about the quality - should I even use that switch?
FhG: Windows Media Player 11, 128 kbps CBR
Low Anchor: l3enc 1.0, -br 128000. Should I foce JS coding here, too?
I don't think much has changed since then. Perhaps we could reconsider the FhG version, but for me it would still make sense to include the WMP11 version, which is the version that most Windows users already have. If FhG 4.1.0 really is better in stereo mode it could be considered. It may also be regressed. I really would like to see some personal test results with ABX logs. It would also be useful to compare the v. 3.4 and 4.1.0 encoding speeds before making the choice. After all we were supposed to compare the faster options against LAME.
Also, wondering if Gabriel changed his mind regarding VBR vs. ABR in 3.98. Using RealPlayer for creating the Helix MP3s is reasonable - saves us from testing the various options given by the CLI.
Does anyone know how quick Microsoft usually is with updating the MP3 libraries?
Fraunhofer is also an encoder I would like to include - same question now: CBR or VBR?
There seems no mode to produce something around 128kbps using VBR, only much lower (mode 5) and much higher (mode 4).
Also, wondering if Gabriel changed his mind regarding VBR vs. ABR in 3.98.
Why would Gabriel had changed his mind between the versions 3.97 and 3.98 ? Isn't -V5 (--vbr-new) de facto standard that is known to be able to compete with the newer encoders. (If you mean the quoted reply, it was for 3.
88.)
I just updated my WMP11, iTunes and Real Player to the latest versions.
WMP11 had no recent MP3 codec updates. It uses the version 3.4 as I said earlier.
The current iTunes version is 7.4.0.28 and the installer is 13 MB bigger than the 7.2 installer that I had previously. I have no idea if its MP3 encoding has changed anyhow.
Real Player is still the version 10.5, but the so called product version number has advanced from 6.0.12.1483 to 6.0.12.1662 since last year.
I am going to run my 25-track bitrate test set with these versions and also check the encoding speeds.
I really like to see GoGo included because it has not been tested against Helix & FhG. For GoGo I would suggest to use -a -b 135 and a q-option that would produce at least about three times faster speed when compared with LAME. The default without a q-switch could be possible. On my PC the default setting produces about 60x speed and -q 2 about 40x, but I have not tested LAME yet. (Joint stereo seems to be enabled automatically at this ABR bitrate so a stereo mode switch is not needed)
EDIT
I'll include the FhG IIS v. 4.1.0 command line encoder in my bitrate & speed test. Perhaps I am able to do some quick quality comparisons too (using the encoded files). I'll post the results during the coming weekend.
ff123 and other pros, what do you think about dropping the high anchor and using the five contenders LAME, FhG, iTunes, Helix and Gogo at 128 kbps?
Previous listening tests at 128 kbps clearely teached us that transparency is almost reached for several people. Including an high anchor is very important but in this situation it may be pointless. I would go for the one of the following configurations:
• 3 competitors + 2 anchors (low & high)
• 4 competitors + 1 anchor (low)
=> 5 encodings to focuse on (and for most people, 4 only - assuming that low anchor is immediatelly detected)
VBR vs ABR/CBR:
I wouldn't drop VBR. From my bitrate table (based on classical music only - thus with limited validity) we should get three competitors with comparable bitrate : LAME -V5 (~130 kbps); Fraunhofer 4.xx CLI (~130 kbps) and of course Helix which is by far the most flexible tool. I don't know anything about iTunes & gogo.
The remaining question is: is VBR the best choice for HELIX and Fraunhofer (and other encoders). There's only one way to get a valid answer: pre-listening tests. I seriously doubt that people would spend their free time to get a solid experience with these rather unused (here on HA) encoders. I tried once and quickly gave up: I don't see the point of spending hours and hours to analyse encoders that are immediately unusable (for my taste).
Using CBR/ABR for each competitors is another possibility but such configuration won't learn anything to the community (assuming that most HA members are following the recommendations/experience of other members and are using LAME -V5 to get small and HQ MP3 encodings). So I would definitely discard this option.
If I had to make this test alone I would go for:
- VBR only encoders
- it implies to discard all encoders that doesn't allow ~130 kbps VBR encodings and keep the other ones
- unless switches are specifically labelled as 'improve quality' I wouldn't tweak the commandline (there are some dubious ones for HELIX) unless some people could provide valid proof of their positive effects on most situations.
I don't see any reason why every encoder should be used in VBR mode or to exclude competitors with poor VBR quality.
IMO we should use any encoder with settings that are expected to be best from experience though certainly there is no scientific evaluation.
As for additional switches I agree that only when there's strong experience against it we should use other than the default settings.
From Gabriel we know already that gogo should be used with ABR.
From my experience with FhG I can say that VBR mode isn't very robust. So CBR should be used IMO.
With Lame Gabriel and robert can say what setting to use but certainly this will be -V5.
Helix should be used in VBR mode IMO.
I have no idea for the other encoders.
If I had to make this test alone I would go for:
- VBR only encoders
- it implies to discard all encoders that doesn't allow ~130 kbps VBR encodings and keep the other ones
- unless switches are specifically labelled as 'improve quality' I wouldn't tweak the commandline (there are some dubious ones for HELIX) unless some people could provide valid proof of their positive effects on most situations.
As I suggested, there would be no problem If the test idea was MS, Apple & Real vs. LAME & GoGo.
VBR can and should be used in the iTunes and Real Player options as the developers provide the option and the average bitrate seems to be suitable.
WMP11 has no VBR option so MS has made the choice. If the newer FhG CL encoder (or the same encoder through Nero's UI) is selected instead of WMP11 the CBR option may be the only possibility. It has been said that there is no suitable VBR quality setting. In this case Fraunhofer has made the choice.
GoGo is an exception because it is not maintained anymore and we don't have any developers' recommendations. We should use a setting that a sensible and experienced user would select when LAME -V5 (--vbr-new) is too slow for getting the files ready before they are needed. If GoGo is left out of the test we will never know if e.g. Helix would have been better or worse than GoGo. If GoGo is found to be worse than the others it would work as a good "second low anchor". It may also positively surprise with some of the samples.
I don't see any reason why every encoder should be used in VBR mode or to exclude competitors with poor VBR quality.
To avoid criticism and suspicion... it already happened in the past: if any encoder, set with ABR/CBR, doesn't come on first position, some people (call them zealot or not) are claiming that the whole test is biased. Here on HA and also on other website. It partially ruin the credibility of the whole test (and not necessary for a bad reason).
IMO we should use any encoder with settings that are expected to be best from experience though certainly there is no scientific evaluation.
The lack of "scientific" evaluation lead to the whole internet carnival of opinions. HA.org was born to fight against this whole mess. Listening tests share the same goal. I can't imagine any listening test based on fragile preconceptions. That's precisely why I'm so worried about this idea of starting a MP3 listening test. Unlike multiformat listening tests, we don't know anything solid about most competitors.
As for additional switches I agree that only when there's strong experience against it we should use other than the default settings.
I also said that we shouldn't use the default setting if some specifically "HQ" switches are existing but aren't used by default. It's the case for Fraunhofer 4.xx IIRC.
From Gabriel we know already that gogo should be used with ABR.
As far as I can read he said that LAME 3.88 was likely to be better with ABR. I can't understand the source code of gogo and I can't say if gogo is nothing more than LAME 3.88 with speed optimisation. In this case we would have a valid opinion about gogo. But if GOGO is nothing more than a faster LAME 3.88 what's the point of testing such antiquity?
From my experience with FhG I can say that VBR mode isn't very robust. So CBR should be used IMO.
Perfect. But are there any chance to see detailed results?
IMHO, this whole test has no point unless the tested LAME rivals are sigificantly faster than LAME. That's why I may change my opinion about using a high quality -q switch with GoGo if it reduces the encoding speed too much.
My very modest proof is this:
As a quick rehearsal I tried this sample
http://www.hydrogenaudio.org/forums/index....showtopic=47370 (http://www.hydrogenaudio.org/forums/index.php?showtopic=47370)
using the following encoders & settings:
LAME 3.97b2, -V5 --vbr-new
WMP10 / FhG, 128 kbps CBR default settings (I used the acmenc command line interface)
Helix (hmp3enc.exe 23.7.2005), -X2 -U2 -V60 -HF2 -F17000
iTunes 6.0.5.20, 128 kbps, VBR, Quality: Highest, Joint Stereo, Smart enabled, Filter below 10 Hz enabled
iTunes 6.0.5.20, 128 kbps, CBR, Joint Stereo, Smart enabled, Filter below 10 Hz enabled
Gogo 3.13, -a -b 133 -m j -q 2
The first few distorded rock guitar chords sent all other encoders except LAME straight to the Hell.
I ABXed them all 8/8, but LAME was difficult and the others were quick and easy.
In general, LAME was quite good, not annoying at all, FhG was barely usable, the others were bad.
I don't know if the encoder versions and settings I used were optimal, but if the actual test samples behave even partially like this I think we may have a good chance to find a clear winner this time.
Personally, after this rehearsal, I would use only LAME at this bitrate. Even the other samples would be transparent with all encoders a standard rock quitar sound like this is too important for me.
I reuploaded the "AC_DC___Highway_to_Hell.wv" sample:
http://www.hydrogenaudio.org/forums/index....st&p=514963 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=47370&view=findpost&p=514963)
Fraunhofer is also an encoder I would like to include - same question now: CBR or VBR?
There seems no mode to produce something around 128kbps using VBR, only much lower (mode 5) and much higher (mode 4).
Correction, mode 4 seems to produce 132 kbps on a big set of files, so it seems usable.
it already happened in the past: if any encoder, set with ABR/CBR, doesn't come on first position, some people (call them zealot or not) are claiming that the whole test is biased. Here on HA and also on other website. It partially ruin the credibility of the whole test (and not necessary for a bad reason).
So should we do anything to avoid people to react strangely? IMO this is not a good target and we won't succeed anyway.
We should do the best according to what we know - well conscious about the imperfections.
You're absolutely right that we don't know a lot about the competitors, and everything is disputable, but that's no reason to me to worry about such a test.
To me any listening test has it's restricted meaning - but still is valuable and adds to experience.
halb27, as far as I heard so far, FhG 4 seems to be a new encoder so maybe the VBR performance increased compared to the 3 series.
So should we do anything to avoid people to react strangely? IMO this is not a good target and we won't succeed anyway.
No, indeed - we can't avoid it. But there's one question we must ask to ourselve: would such reaction be strange? Why should someone believe that CBR performs better than VBR? Why should someone accept that ABR is globally a better choice when this superiority is only attested for specific problems? I personally don't see any reason to do that. I (can) accept some advice from developers themselves but from unknown people I simply can't do it. Or to be more precise: I could trust some people's advices for a personal usage of an unfamiliar encoder but I wouldn't engage the credibility of a test and the work of 20 listeners on such fragile knowledge.
We should do the best according to what we know - well conscious about the imperfections.
OK, but if someone (me for exemple) will provide samples revealing that FHG VBR performs better than CBR? What would be our knowledge?
The only sure thing is lame -V5 cause we know it well. Helix, Fhg should be tested @ 130k vbr too, not CBR.. Gogo shouldn't be tested as its too old and not really faster than the others. Why do I have a gut feeling that the results will be again near perfect not showing much difference ?
halb27, as far as I heard so far, FhG 4 seems to be a new encoder so maybe the VBR performance increased compared to the 3 series.
OK, I did my test with an earlier version so my experience may not be valid any more.
According to my bitrate table Fhg VBR -m3 mode looks as the perfect VBR setting for such listening test: 128 kbps on 150 full classical music tracks (16 hours of music). For comparison, LAME 3.98b5 -V5 reaches 129 kbps on the same corpus.
As usual advantage of VBR coding, bitrate fluctuates and can reach high values on critical samples. Here are four examples, coming from my usual gallery (http://gurusamples.free.fr/), showing that Fhg VBR (-m3 -q0) outperforms CBR/ABR encodings (tested -br 134):
http://rapidshare.com/files/54002892/fhgsamples.zip (http://rapidshare.com/files/54002892/fhgsamples.zip)
You may notice it: individual bitrate is considerably higher than 130 kbps but that's exactly what we're expecting from VBR with such samples.
... OK, but if someone (me for exemple) will provide samples revealing that FHG VBR performs better than CBR? What would be our knowledge? ...
That would be a valuable piece in the puzzle of improving experience.
@halb27> I anticipated your wish (see previous message)
A small bitrate table (150 full tracks of music; genre = classical):
http://www.megaupload.com/?d=Z1VKHJ0U (http://www.megaupload.com/?d=Z1VKHJ0U)
In summary:
fraunhofer mp3encs v1.4 20070711
-m1..... 193 kbps
-m2..... 161 kbps
-m3..... 128 kbps
-m4..... 118 kbps
-m5..... 101 kbps
LAME 3.98 beta 5
-V4..... 160 kbps
-V5..... 129 kbps
-V6..... 112 kbps
EDIT: commandline used (foobar2000):
-if - -of %d -vbri -ofl -eof -sr 44100 -c 2 -br 0 -m 1 -q 0
-if - -of %d -vbri -ofl -eof -sr 44100 -c 2 -br 0 -m 2 -q 0
-if - -of %d -vbri -ofl -eof -sr 44100 -c 2 -br 0 -m 3 -q 0
-if - -of %d -vbri -ofl -eof -sr 44100 -c 2 -br 0 -m 4 -q 0
-if - -of %d -vbri -ofl -eof -sr 44100 -c 2 -br 0 -m 5 -q 0
-c2 = 2 channels
-br 0 = required to enable VBR
-m
n = VBR mode
-q 0 = enables fast quality coding mode [damned! I used the wrong switch!!! I had to redo the whole table]
Thanks for testing guruboolez!
In a quick test with my AC/DC sample a "FhG IIS 4.1.0 @ -m 3" file was easy to ABX. A lot easier than a LAME -V5 encoding. The electric guitar in the beginning of the sample is totally ruined. The bitrate was 144 kbps.
I am going to gather a few of my favorite samples and test FhG 3.4 CBR, FhG 4.1.0 CBR & VBR and GoGo ABR. However that is not going to happen before I have run the bitrate and encoding time tests on Saturday or Sunday.
Test -m 3 with some of my audio tracks (Pink Floyd, Led Zeppelin, Alphaville, Dire Straits...) and the bitrate is around 150 kbps.
menno also said that -m 4 seems to produce ~130 kbps.
fraunhofer mp3encs v1.4 20070711
-m3..... 128 kbps
-m4..... 118 kbps
I tested on more than 100 samples gathered from listening tests over the years:
-m3 produces 140 kbps
-m4 produces 130 kbps
Lame -V5 produces 136 kbps on this set
BTW, what does the "Original File Length" actually do? Does it have anything to do with gapless playback?
q 0 = enables fast quality coding mode
This is the default mode and should be preferred for this test.
Yes, was just about to ask whether or not we should force encoders to prefer speed over quality, but I saw that -q 0 is default anyways.
BTW, what does the "Original File Length" actually do? Does it have anything to do with gapless playback?
Yes, it is for gapless playback. FhG decided to reinvent the wheel.
Edit:
It would be great if Nyaochi or some other developer could create a front end that would save a LAME info header to FhG files, as I wrote in this reply:
The WMP11 encoder is currently this:
MPEG Layer-3 Codec Version 3.4.0
Fraunhofer IIS MPEG Layer-3 Codec (professional)
Copyright (C) 1996-2004 Fraunhofer IIS
The great thing about this codec is that it can used through Nyaochi's "acmenc" command line tool (after a small registry edit). Acmenc can include a LAME info tag style header which contains info for gapless playback. It is easy to configure foobar to encode through acmenc. I have used it as a faster option to LAME when I am in a hurry.
I wonder if Nyaochi could create a modified version which could be used with FhG's exe encoder (for creating a compatible gapless playback info header instead of FhG's quite useless newly introduced header).
This is the default mode and should be preferred for this test.
Why should speed be prefered over quality?
Testing the faster options vs. LAME has been the idea of this test since August 2006 (or even before).
Perhaps the new FhG version changes that, but could you first try if the encoder is any better at -q 1 than at -q 0.
Test -m 3 with some of my audio tracks (Pink Floyd, Led Zeppelin, Alphaville, Dire Straits...) and the bitrate is around 150 kbps.
menno also said that -m 4 seems to produce ~130 kbps.
I just tried with one “speed metal” album:
-m3 = 150 kbps
-m4 = 140 kbps
-m5 = 113 kbps (32 KHz)
I'd like to see a bitrate table including different musical genre (some members did it in the past) to see which -m mode would be the most adequate. -m4 sounds significantly worse than -m3 (crappy artifacts everywhere on metal).
Testing the faster options vs. LAME has been the idea of this test since August 2006 (or even before).
Yes, I remember that some people requested a “fast MP3 encoder listening test”, right?
But is it still the purpose here? Or are we interested to see if the new and rather untested MP3 encoders are valid challengers to LAME?
If the goal of this test is to check which fast MP3 implementation offers the best quality, then I don't see the point of including LAME (excepted maybe as high anchor). But of the goal is to evaluate what degree of quality all "modern" MP3 encoders are offering, then I would use them at their “high quality mode” - not their “fast” ones.
-m 3 seems to encode at 50x with foobar2000 (two parallel encoders due to HT CPU).
The actual idea of the test was to see if LAME, which is really slower than the other encoders, offers a much higher quality that would justify the speed "issue". LAME could serve as high anchor, but -V 5 --vbr-new might not be that much better than the rest thus doing a bad job as high anchor.
On rock / metal i get around 152k (m3) 140 (m4)
-m 3 seems to encode at 50x with foobar2000 (two parallel encoders due to HT CPU).
The actual idea of the test was to see if LAME, which is really slower than the other encoders, offers a much higher quality that would justify the speed "issue".
I missed this idea. Sorry.
But shouldn't the first logical competitor to LAME be LAME itself, at -q7 or -q9 setting (which are significantly improving the encoding speed)? If the purpose is to evaluate the tradeoff between quality and speed shouldn't we question first the choice of LAME developers before testing alternative encoders?
Here are some results with -m 3:
Vivaldi's Four Seasons: 130
Chris Rea (3 albums): 150
Coldplay (2 albums): 145
Deep Purple's Made in Japan Remastered: 150
Dire Straits (3 albums, 1 live with 2 CDs, 2 studio, 1 remastered version of a studio album): 150
I can let fb2k encode and see what else comes out after encoding my large Pink Floyd collection, Jethro Tull, etc., but I think it will stay at around 150 kbps.
Eminem's Curtain Call - The Hits: 145
-m 3 seems to encode at 50x with foobar2000 (two parallel encoders due to HT CPU).
The actual idea of the test was to see if LAME, which is really slower than the other encoders, offers a much higher quality that would justify the speed "issue".
I missed this idea. Sorry.
But shouldn't the first logical competitor to LAME be LAME itself, at -q7 or -q9 setting (which are significantly improving the encoding speed)? If the purpose is to evaluate the tradeoff between quality and speed shouldn't we question first the choice of LAME developers before testing alternative encoders?
I was thinking about testing the recommended LAME settings for 128 kbps (= -V 5 --vbr-new) against other fast encoders with their default setting to see if there's a big quality difference. Another possibility would be asking LAME devs for the fastest LAME setting and then comparing that to the fastest setting of the other encoders OR trying to figure out what high quality settings to use for the contenders and then checking the results of the test - if the other contenders that were set to prefer quality over speed are still worse than LAME, then their fast setting would be for sure, too. If they are better and the speed is similar to LAME's, LAME no longer is the king of MP3 encoders. If the quality is equal, look at the encoding speed. If the speed was also similar to LAME's, then we still have the "problem" that we don't know how well the fast settings perform.
Gogo shouldn't be tested as its too old and not really faster than the others. Why do I have a gut feeling that the results will be again near perfect not showing much difference ?
yeah, besides, gogo has already been tested.
... showing that Fhg VBR (-m3 -q0) outperforms CBR/ABR encodings (tested -br 134):
http://rapidshare.com/files/54002892/fhgsamples.zip (http://rapidshare.com/files/54002892/fhgsamples.zip)...
These are only the samples. I thought you provided a listening test.
8/8 everywhere (I didn't mentionned it, sorry).
I can restart the test if you really want ABX log files.
8/8 everywhere (I didn't mentionned it, sorry).
I can restart the test if you really want ABX log files.
No ABX log necessary, I just wanted to know the result.
Sample “A03” offers the most obvious difference IMHO.
I didn't try yet with VBR -m4 which seems much more suitable to a ~130 kbps listening test (with non-classical music samples).
What is fraunhofer mp3encs v1.4 20070711 please?
all4mp3's mp3sEncoder.exe V1.4 is dated 2007, July 5, and based on an encoder library V04.01.00 2007-05-18.
So this must be something else.
http://www.all4mp3.com/tools/sw_fhg_cl.html (http://www.all4mp3.com/tools/sw_fhg_cl.html)
check the filename
Thank you, but that's what my version originates from (the Win32 version). Guess you use another OS.
I checked all download links and all are providing “mp3sEnc(...)20070711.zip”
I see, you referred to the file name of the zip file, not the encoder itself.
exactly
1. I'm more interested in quality than speed, so, if there are other mp3 encoders being actively developed besides Lame (--I assume FhG is), and if unofficial but quick abx tests by known good listeners show they may plausibly be contenders, then I would like to see them included, or at least the one that is suspected of being the best rival. I also think mixing VBR & CBR can lead to some questions about the results (even in the 64kbps round), at least on tricky individual subtests where VBR produces a significantly bigger bitrate file than CBR, though I realize you keep it within reasonable bounds (10% or whatever). (Is any mp3 encoder not developed to take advantage of VBR likely to do as well as LAME in bitrates of 128kbps?)
2. Of course, if you are testing speed, then it would make sense to make sure all encoders are at the same level of speed (just as you would for bitrate)--produce what quality at the same bitrate in the same time. Else, you are comparing apples and oranges. What you are aiming for is quality per bit per second. Now you could correct for different times by normalizing the numbers after the test (ratios per time etc.), but that obviously is less meaningful (some encoders might do disproportionately bettter or worse at a different speed setting, so a linear ratio or normalization would be deceptive).
How about adding Lame 3.90.3 as well, just for kicks. And since I imagine there are quite a few of those mp3's laying around.
iTunes and Helix are also featured for sure. Should I use both in VBR mode?
I'd say yes.
Fraunhofer is also an encoder I would like to include - same question now: CBR or VBR?
Does FhG have a set of recommendations in any open documentation? According to this (http://www.iis.fraunhofer.de/EN/bf/amm/projects/mp3/index.jsp):
.. If you use a variable bitrate setting, choose the maximum quality setting available in the encoder interface.
That's not too terribly informative, but I don't see anything that leads me to believe that FhG advises against VBR, so I'd say VBR should be used.
How about adding Lame 3.90.3 as well, just for kicks. And since I imagine there are quite a few of those mp3's laying around.
Can't demand so much of the testers--I think they're trying to get the list down to four candidates, so 3.90.3 should not be included.
Yes, and besides, Lame 3.90.3 was most popularly used for VBR ~ 190-200 kbps (-alt-preset standard, equiv to -V2), not much at 128 kbps (for which no VBR mode was available, only CBR or ABR).
How about adding Lame 3.90.3 as well, just for kicks. And since I imagine there are quite a few of those mp3's laying around.
No thanks, definitely not an option.
Someone mentioned Radium/ProducerPro/l3codecp.acm v1.2.x before. I too would like to see this encoder included as it was so widely used. (heck, I still use it with virtualdub) I wan't to see how everyone else thinks it compares to the newer versions found in wmp10/11 (l3codecp.acm v3.4.x aka fastenc).
Personally I would drop GOGO since it is based on lame 3.88 (or something around there) and would much rather see lame 3.90.3/3.93.1 --alt-preset CBR 128 -h (only because you all don't like my crazy/custom settings ) to see how much lame really has/hasn't improved.
I think it would be better to have an FhG test, a Xing test, and a LAME test to see how much the encoders have changed over the years. Then compare the best with each other.
My $.02
I will definitely not include versions older than 3.97. 3.97 was an idea because it's the currently recommended encoder, but since Gabriel and others want to test 3.98, I prefer that over 3.97.
Any idea what was changed from Gogo 3.12 to 3.13?
I think before starting to discuss settings, we should decide what the test should be good for:
A. Compare recommended LAME quality settings to "defaults" of the competition (defaults in quotation marks because VBR will be probably enforced - what I mean is that no additional settings like optimization for speed or quality should be supplied)
B. Compare recommended LAME speed settings to fastest mode of the competition
C. Compare recommended LAME quality settings to best quality mode of the competition (but only if those settings have a note that they really increase quality, such as in FhG's case, so no funky Helix switches that nobody knows what they are actually good for)
I choose C. I'm interested in how competitors perform with quality-geared settings. Encoding speed should be mentioned but should also be a secondary consideration unless we can create a metric to gauge performance-per-time-unit as a fellow above suggested.
FhG MP3 Surround Encoder is an Next-Gen MP3 encoder & you must use the full version for any test...
I think before starting to discuss settings, we should decide what the test should be good for
C seems most preferable to me.
FhG MP3 Surround Encoder is an Next-Gen MP3 encoder & you must use the full version for any test...
I don't understand what you mean with "full version".
I think before starting to discuss settings, we should decide what the test should be good for:
FhG MP3 Surround Encoder is an Next-Gen MP3 encoder & you must use the full version for any test...
I'm not sure to understand what Silver Wave means by 'full version' but if 'full' means 'full quality' I would agree with this point.
The first questions I would ask:
• now we have bi-, quad- and soonly octo-core processors working at several GHz, is there any point of testing encoders, which are already very fast, with their fastest settings?
• shouldn't we first respect these new competitors and check their full potential - especially if we oppose them to a reference called LAME which will be set to offer its best - before evaluating their quality on their fastest mode?
In my opinion the true challenge of this “MP3 listening test” is to see what potential offer the latest “2007” Fraunhofer encoder, the flexible HELIX one, and maybe the last Apple's implementation (totally different from Fhg & Xing). I would consider them as true competitors to LAME, and not the poor ones we could handicap even further with 'fast'/'lower quality' settings.
It's a listening test: we don't have to presuppose that LAME's competitors are worse in order to justify our choice to lower their quality a bit more. It implies the following rule: if one encoder is set to reach its full potential (LAME) all others should be set with
the same principle. If our goal is to test ultra-fast MP3 implementations, then LAME 3.xx looks out of competition.
what are we interested for? To see if LAME's superiority is still valid? To see which x90 MP3 encoder offers the less distorted sound?
I know there's another question (already mentionned): is LAME relative “slowness” really justified by a quality boost? That's why some users are interested (for practical purpose) to "boost" speed to the max level. This question is perfectly valid and is truly interesting. But starting a listening test for the sole purpose to answer this question is problematic.
If LAME appears as better, what would be our gain? We would only “discover” that a x60 encoder sound worse than a x15 one:
what a lesson! And we would still ignore if Fraunhofer 4.xx in HQ mode is able to compete with LAME. Gain in this hypothesis =
nada.
... In my opinion the true challenge of this “MP3 listening test” is to see what potential offer the latest “2007” Fraunhofer encoder, the flexible HELIX one, and maybe the last Apple's implementation (totally different from Fhg & Xing). ...
I absolutely second that. Speed really shouldn't be an issue as any encoder can be considered fast enough.
Sure it's remarkable in case it should come out that a superfast encoder has better quality than a sufficiently fast one, but that's all.
As far as I remember in the first thread for the 128 kbps listening test there was a rough motivation to see how Lame competes with other encoders which happen to be faster. The speed thematics was there, but it was not an overwhelming one.
Do you guys mind if Real is used for encoding the Helix samples?
I prefer the CLI encoder because I did use it and had a pretty good impression qualitywise.
I also bought Real when I cared about Helix a lot but wasn't impressed and continued to use the CLI encoder.
I must admit however I don't remember the details, and it may be that it was just the GUI that disappointed me.
RealPLayer and the command line Helix encoder from Rarewares seem to use quite different VBR settings by default. Helix @ -V60 was 3.6 x faster than RealPlayer in my test.
I have run my bitrate and speed tests and just started to prepare my report. It may take an hour or so. I encoded my usual "25 complete tracks" test set 12 times using the discussed encoders and some possible setting options.
Another "speed" consideration is that encoding potential far outstrips ripping potential, and the disparity will only grow. Looking at Tom's Hardware CPU chart, it looks like just about any new machine purchased in the past three years or so is capable of encoding an entire CD with LAME in five minutes or less, though I don't remember which settings they use.
*Lame settings: for 3.98, I'd suggest "-V5" (vbr-new is the default vbr mode in 3.98), but I'm not sure if it would not be higher than 128kbps.
*High anchor: I agree that it could be dropped, as we are expecting some already high results
*Gogo: is anyone still using it?
*"Radium" crack: I think that the results would be very similar to Audioactive Production Studio
*Lame fast setting: well, I don't think that there is a need for any test to know that current Lame would produce quite lower quality than FhG if we want to compete with speed in mind.
Here are my results. For iTunes and RealPlayer I used the standard GUI programs. For the command line encoders I used Speeks' Batchenc frontend. The WMP11 mp3 codec was accessed by Nyaochi's Acmenc and Speek's Batchenc. The test bench was a 2.8 GHz P4. Before the speed tests I tried to disable all unnecessary background processes. The source files were in wave format on one HD and the target HD was separate. I measured the encoding times with a digital stopwatch. I didn't run several passes that would have been needed for lowering the error margin, but I think the results are accurate enough for this purpose. For measuring the bitrates I used Encspot Pro's "Complete Scan" feature (some other programs may not show correct bitrates with files that don't contain a Xing header).
The test file set consists of 25 selected complete tracks of various genres. The total duration is one hour, 31 minutes and 34 seconds.
The encoding speeds and average bitrates:
(http://i224.photobucket.com/albums/dd212/AB2K/speedtable.png)
The individual bitrates of the VBR files:
(http://i224.photobucket.com/albums/dd212/AB2K/vbrtable.png)
A chart of the VBR bitrates:
(http://i224.photobucket.com/albums/dd212/AB2K/vbrchart.png)
Some readers may also find the following table interesting (Album = the 25-track test set):
(http://i224.photobucket.com/albums/dd212/AB2K/peaks.png)
I uploaded the test data here (Excel file & Encspot Pro reports):
http://www.hydrogenaudio.org/forums/index....st&p=515301 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=47370&view=findpost&p=515301)
May I suggest that LAME ABR 128 be included in the test, aside from -V5 which is not a true 128 setting?
Thanks a lot for your test Alex. I am wondering if the highest iTunes setting (VBR quality 7) would increase bitrate too much. Have to do some testing myself.
May I suggest that LAME ABR 128 be included in the test, aside from -V5 which is not a true 128 setting?
No.
By the way, seeing that FhG 4.1 VBR is only 0.8x slower than Gogo ABR, I am wondering if I should test Gogo...
Last night I saw AlexB's sample thread in the uploads section.
Is this the place for sample proposals, or is it just to provide unknown new samples that can be tested?
Well, that is a rather old thread I started last year, but you can submit samples you think would be valuable for this test. Samples discussion will follow once we have the encoder and the settings set.
Thanks a lot for your test Alex. I am wondering if the highest iTunes setting (VBR quality 7) would increase bitrate too much. Have to do some testing myself.
I don't have any numbers in the quality options in the English version. Perhaps you have the German version and it is different. Mine has: Lowest, Low, Medium Low, Medium, Medium High, High and Highest.
By the way, seeing that FhG 4.1 VBR is only 0.8x slower than Gogo ABR, I am wondering if I should test Gogo...
Has anyone verified that FhG 4.1 @ -m 4 & -q 0 would be better than GoGo? (Obviously -m 3 produces too high bitrates for this test.)
Since this encoder discussion seems to not advance I would like to suggest that we discuss about each encoder separately. Perhaps then we could make final decisions. Since you mentioned the iTunes' quality settings we could start with iTunes. I tested some additional iTunes options:
(http://i224.photobucket.com/albums/dd212/AB2K/itunesbitrates.png)
Surprisingly the quality options have almost no effect to the encoding speeds. The very small encoding time differences may be caused by other things as well (I did only one pass). I wonder if these settings do anything, even though there was a very small bitrate increase.
The Smart Encoding Adjustments and Filter Frequencies Below 10 Hz options seem to slightly affect the speed so perhaps they actually do something.
In general I can't see any advantage of using iTunes instead of LAME. In my experience iTunes produces inferior quality and it seems now that it is not significantly faster. Also, it doesn't write a LAME info tag for gapless decoding so even if the quality would be comparable I would prefer LAME. Unless Apple has miraculously increased the quality of this encoder I wouldn't mind if iTunes was dropped from this test. If other forum members insist to have it tested I would suggest using the settings that probably produce the best quality. I think the setting names Apple has chosen suggest that the best quality would be achieved @ 128 kbps VBR, Quality: Highest, Stereo Mode: Joint Stereo, Smart Encoding Adjustments & Filter Frequencies Below 10 Hz: enabled.
Edit: I made a small mistake in the iTunes table. I uploaded a fixed file. Refresh your browser to see it.
Yes, I only have numbers, 3 of them also having a German description:
(http://www.maresweb.de/miscellaneous/itunes.png)
I think iTunes is an encoder that should be featured in this test and I would drop Gogo if an encoder really needs to be excluded.
Has anyone verified that FhG 4.1 @ -m 4 & -q 0 would be better than GoGo? (Obviously -m 3 produces too high bitrates for this test.)
Shouldn't we test with -q 1?
I think iTunes is an encoder that should be featured in this test
I've explained why I don't need its test results. Why would you like to see it tested? Just for completeness' sake? In any case this is not a biggie for me, but I think each choice should be supported by proper argumentation. *)
I really would like to see all iTunes's MP3 encoder fanboys (and girls) to show up with their ABX logs and settings suggestions...
I would drop Gogo if an encoder really needs to be excluded
So you don't want to discuss about each encoder separately? We have at least three other debates to solve: Helix vs. Real, FhG versions & settings and GoGo vs. the other fastest encoders (which may be an empty list if Real and FhG in the slow mode is chosen). If we could concentrate on one thing at a time perhaps we could make progress.
EDIT
*) Previously I thought that iTunes was faster. Now I have no personal reason for testing it.
By the way, seeing that FhG 4.1 VBR is only 0.8x slower than Gogo ABR, I am wondering if I should test Gogo...
Shouldn't we test with -q 1?
These two comments are contradictory. -q 1 is a lot slower than GoGo. We really would need to try a few samples and compare the FhG 4.1 -q 1, -q 0 and perhaps FhG 3.4 quality differences. But as I said I would like to discuss about FhG separately after we are through iTunes.
OK, you are right about the contradiction - I missed the fact that 71x was with -q 0, sorry.
So here's an important question: do we need to drop any of the 5 contenders LAME, FhG, iTunes, Gogo and Helix? If we add a low anchor to the configuration, there should be 6 encoders.
So here are my reasons for the contenders:
LAME: Recommended encoder here on HA and so far it is said to offer the best quality.
iTunes: Encoder is available for both Windows and OS X and seems to be popular outside HA. People with iPods who prefer to use MP3 over AAC usually choose iTunes to encode. Moreover, some artists who are offering songs in MP3 format seem to use iTunes as encoder (like Bruce Springsteen http://www.radionowheredownload.com/) (http://www.radionowheredownload.com/)).
FhG: New encoder version which is worth testing because it's also pretty fast.
Helix: Fast encoder.
Gogo: Fast encoder, but on the other hand, quite old and I am wondering if it offers any big advantage over Helix or FhG which at least are actively developed (or is Helix not developed any more?). FhG with -q 1 is much slower that Gogo, though.
Sounds like the speed thematics is still there as a major one - at least to some of us.
Anyway, I second taking each encoder separately one in a row and call for opinions whether it should be included and with which settings.
And as things began to start already to some extent with iTunes mp3:
I personally like to see it tested, with the settings: q highest, js, smart&filter enabled.
In case average bitrate thus turns out ot be significantly higher than that of the competitors, q medium should be chosen IMO.
iTunes: Encoder is available for both Windows and OS X and seems to be popular outside HA. People with iPods who prefer to use MP3 over AAC usually choose iTunes to encode. Moreover, some artists who are offering songs in MP3 format seem to use iTunes as encoder (like Bruce Springsteen http://www.radionowheredownload.com/) (http://www.radionowheredownload.com/)).
These are good points. Possibly the test can prove that it is worth of using LAME on a Mac too. It would also be good to be able to show proof to audio techinicians. If iTunes turns out to be better than I expect I would be happy to be proven wrong...
[OT] I have been listening to Bruce's Live 1975-1985 album (3CD Box) for an hour now... [/OT]
Before discussing about each encoder separately we should clarify the exact purpose of this test:
1/ evaluating several MP3 implementations at their best settings
2/ evaluating several MP3 implementations at their fastest settings
3/ evaluating several MP3 implementations set with different principles
4/ ???
Once the goal of the upcoming test totally clarified we should debate about the pertinence of each possible competitor.
I would go for the 1st one. The second one is in my opinion anachronic ; the third one would be biased (as it supposes that we can arbitrary decide that some implementations are only worth for their speed and don't have to be evaluated at their full quality potential).
I asked the same question a few posts ago and it seems that people tend to want 1. However, if the number of contenders is a problem, we need something in order to choose whether or not encoder A should be preferred over B. In such a situation, I think speed and popularity could be good criterions.
I go with 1) as well.
I'm also interested in the behavior of different settings, but as most people do I see that this is not easy to do and we better don't.
... if the number of contenders is a problem ...
5 contenders and 1 low anchor - is that too much?
I asked the same question a few posts ago and it seems that people tend to want 1. However, if the number of contenders is a problem, we need something in order to choose whether or not encoder A should be preferred over B. In such a situation, I think speed and popularity could be good criterions.
I agree: popularity seems a good criterion (though it's sometimes hard to evaluate).
I'm also interested in the behavior of different settings, but as most people do I see that this is not easy to do and we better don't.Do you mean testing different settings of the same encoder? It's interesting indeed, but this task belong to a dedicated listening test.
I thought it was obvious why I and some others are interested about the faster options. We already have found out that LAME -V5 produces reasonably fine quality in the multiformat test. For many of us the LAME Info tag is very important and we would like to see its support extented to all possible HW & SW players. IMO, the HA members would need very strong reasons to use something else than LAME.
Personally, I have my ripped CDs archived in a lossless format. I have one copy of the complete archive in a high bitrate lossy format for home hifi usage. For portables I would like to encode smaller files on the need basis.
Normally I can do that with LAME, but there are times when I am for example about to start a holiday trip in about two hours and I would like to encode 30 hours of new music to take with me. (I know, I should have done that earlier, but sometimes things just don't go as planned.)
If the decoding speed from the lossless source is 50x and LAME encodes at 15x speed it would take about 2 hours 40 minutes. If a faster encoder encodes at 75x speed the process would take one hour.
The question is, if the quality of the used fast encoder is acceptable.
Edit: typos...
Alex B> I'm sometimes in the same situation (the need to encode quickly some lossless files). I also noticed that the effective speed of fast encoders may be drastically ruined by fragmentation and disk access (amplified by multi-core processors).
I'm also quite sure that the whole process lossless library -> MP3 isn't the most popular way to get MP3 files (most people are probably ripping CD to MP3 and in this case the encoding speed is usually limited by the extraction time). And as you said, speed is only needed in some specific situation (before holidays, etc... otherwise overnight encoding with best/slow encoders is fine). So is this marginal (I suppose) behaviour enough to justify a public listening test? Or shouldn't we use Sebastian's opportunity to evaluate for once the real potential of most popular/attractive LAME competitors (which seems to be all considerably faster than LAME)?
Perhaps the encoder/settings selection doesn't have to be based on the speed or on the quality. Let me explain.
In my speed test I included LAME -V5 -g5. It wasn't significantly faster. In my experience LAME -q5 at medium bitrates is the lowest usable q setting. So for LAME I think it is better to use the standard -V5 setting.
As we have seen iTunes doesn't have any fast VBR settings, so it is best to select the highest quality setting.
The choice between Helix and RealPlayer may not be difficult after all. We have no knowledge about the internal options used in RealPlayer and it doesn't seem to have any speed advantage. The RealPlayer UI is awkward and the converter feature is not available in the free version. A new RealPlayer version is on the beta stage, but I have no knowledge about it (other than it looks different). I would select the more versatile Helix CL encoder and use a suitable -V setting without any additional untested quality switches. However, it would be good to check if Helix has any newer source code available. Do we have any voluntary developers who could create a new compile if needed?
IMHO, the FhG settings should be quickly pre-tested. The difference between -q 0 and -q 1 can be anything. We don't even know if -q1 is better at all or for example if -q 0 is below all usability -- and we will never know that if don't try it before choosing the setting. I can contribute.
That leaves us GoGo. I could include it in my pretest. I hope some others would be able to do the same.
I would like to suggest the following: If GoGo is clearly worse than FhG -q 0 and -q 1 then we could drop it from the test and include both FhG settings (in case both settings seem to be promising). If GoGo seems to be able to provide useful or even better quality than FhG we could continue this discussion.
Also, I think 5+1 would be fine.
Very reasonable and practicable procedure IMO.
I fully agree with Alex B's suggestions.
Might I ask as to why you seem to have stowed away your idea of last spring to organize a (yet unexplored) mid-bitrate multi-codec test next, after my inquiry about a 96k test:
http://www.hydrogenaudio.org/forums/index....53134&st=78 (http://www.hydrogenaudio.org/forums/index.php?showtopic=53134&st=78)
I can't help but wondering if it isn't a bit of a waste of time and resources to explore 128k MP3 (once again), probably the most extensively tested/known/used codec setting today, while the sub-100k range almost remains terra incognita in the field of public listening tests.
Edit: not to mention, as was coined in this thread already, that the prospect of getting all-tied 4.5ish results, seems hardly challenging or even interesting to me.
Polar,
This MP3 test has been discussed a lot. We did a long pretest discssion about a year ago and we decided to postpone this test in favor of lower bitrate multiformat tests (48 and 64 kbps). The last about ~130 kbps Mp3 test is outdated and now it is a very good time to do this test for the discussed reasons. (Check also the the last year's 14-page discussion).
Sebastian has said that the next test in his list is a 80 kbps multi-format test and I hope we can continue testing codecs in a faster cycle so perhaps an about 100 kbps test could follow soon after that. Then we would have tested all important bitrate classes between 48 and 128 kbps.
I think after the MP3 test we will have an AAC test with both LC and HE-AAC at 80 kbps. The winner of that test will be included in the multiformat 80 kbps test.
As a spectator and (very grateful) user of these tests, can I explain why I should like to see the iTunes MP3 encoder tested?
I use a Mac, and have just got an iPod. To navigate on an iPod, tags are very important. It's my understanding that iTunes uses a better database than Max to get its tag information, so I should prefer to use iTunes. I also believe that the script that allows one to use Lame in iTunes uses a now somewhat out-of-date version.
So I'd like to get some kind of handle on how much quality I might lose if I switched from Max+Lame to iTunes with native encoder (assuming that Lame is superior, by some degree to be determined). Therefore I'd like to see iTunes tested at best quality settings.
I could, of course, test it myself, but I don't want to become artifact-aware, and a test like this is more systematic than anything I'd set up myself.
I'm very grateful to the people who organise and take part in these tests. One of their virtues to ordinary punters like me is in discovering those things that it is NOT worth testing for oneself.
I remember in a previous test iTunes did quite badly, and I don't think they would have upgraded the codec since then - they've abandoned it since they switched to using AAC. I think it's worth to note though, the default mp3 bitrate iTunes is 160kbps (where every other program defaults to 128) so i'm sure they knew it wasn't great.
edit: I would really like to see 128k AAC as a high anchor in this test. I know we've already discussed having a high anchor but I think it would be useful to see exactly how much of an edge AAC has over MP3.
As I promised, I tried to do a "quick pretest" with a few samples using foobar's ABX module, but that didn't work. The codecs' quality didn't have immediately obvious differences and the results would have been almost arbitary depending on the used samples. So I felt kind of obligated to do a full-scale personal test.
I used the following codec settings:
FhGc = FIIS 3.4 CBR 128 kbps
FhGv = FIIS 4.1 VBR -br 0 -m 4 -vbri -ofl
FhGv1 = FIIS 4.1 VBR -br 0 -m 4 -q 1 -vbri -ofl
GoG = GoGo 3.13 ABR -a -b 137
GoG2 = GoGo 3.13 ABR -a -b 137 -q 2
Edit: I forgot to mention that I used FhG Mp3sdecoder v. 1.3 for decoding the samples. (Looks like the -ofl option worked. ABC-HR Java didn't adjust the starting position of the FhG VBR samples)
After seeing the results of my previous speed and bitrate test I upped the GoGo ABR bitrate setting from 135 to 137 so that it would better match the codec average. FhG VBR produced still a bit higher average bitrate with the now tested new samples (134 kbps vs. 131 kbps).
I tested 30 varied samples. All samples are created by myself and most of them are brand new. About half of them are from Vangelis' Alexander soundtrack album, which has a lot of interesting music (it's a mixture of classical, electronic and ethnic genres) and I wanted to try if that material would be useful for testing codecs.
The reference samples and my ABC-HR result files are available in this link:
http://rapidshare.com/files/55027379/alexb_test.zip (http://rapidshare.com/files/55027379/alexb_test.zip) (about 76 MB)
Here are the test results:
% Result file produced by chunky-0.8.4-beta
% Chunky -d C:\alexb_test\results -r sample -f C:\alexb_test\codecs.txt
%
% Sample Averages:
S.# FhGc FhGv FhGv1 GoG GoG2
#01 2.90 3.50 3.50 4.00 3.50
#02 3.90 3.30 3.30 2.40 3.30
#03 4.00 3.60 3.90 3.60 4.00
#04 4.00 4.00 4.00 4.00 4.00
#05 3.00 4.10 3.20 3.70 3.40
#06 4.00 3.60 3.30 4.50 2.90
#07 4.00 3.50 4.10 3.30 4.50
#08 4.00 3.80 4.40 4.40 4.20
#09 4.10 3.80 3.20 4.20 3.50
#10 4.30 4.30 3.70 4.50 3.40
#11 3.00 2.50 2.40 3.00 3.00
#12 3.70 4.10 4.00 4.20 3.50
#13 3.20 3.20 3.90 3.70 2.80
#14 3.40 2.80 2.20 3.50 2.50
#15 4.00 4.30 3.90 3.90 4.20
#16 3.20 3.70 3.90 3.70 3.00
#17 3.90 3.10 3.00 4.00 4.00
#18 3.80 4.10 3.90 4.10 3.80
#19 3.80 2.90 3.10 3.90 3.90
#20 3.10 3.10 3.70 3.90 1.90
#21 2.30 2.70 3.00 2.20 2.40
#22 3.90 3.90 3.90 3.90 4.30
#23 3.90 4.00 4.00 4.00 3.80
#24 4.20 4.20 4.20 3.90 4.20
#25 4.20 4.00 4.10 4.00 3.60
#26 3.90 3.70 4.00 3.70 3.70
#27 3.80 4.10 4.00 3.80 3.80
#28 3.50 3.60 3.60 4.00 3.20
#29 2.60 2.80 3.00 1.60 1.50
#30 3.70 3.70 3.50 3.70 3.30
%%% Codec averages:
%%% 3.64 3.60 3.60 3.71 3.44
Even though I did the test properly, I wouldn't trust the results too much.
Mostly I was able to differentiate the contenders without ABXing because of the lowpass. All encoders seem to use an about 15-16 kHz lowpass frequency at this bitrate/quality level. Beyond that the test was a tough call. The codecs were generally quite good and the slight lowpass didn't make the samples annoying (to me, it seems to be detectable only in a direct comparison with the reference).
I didn't want to spend several days with this test so I rated the samples rather quickly and it is possible that a new test with the same samples would produce slightly different results.
GoGo failed misarably on one killer sample, the sample number 29 (also FhG was bad with this sample, but better than GoGo). On the other hand FhG's VBR has obvious problems with some other samples.
I can make the following conclusions:
All encoders are tied and the higher quality options didn't make the encoders better.
Edit: Fixed a couple of typos and added the MP3 decoder info.
Thank you for your exhausting test.
At least it shows that
a) Gogo should be used without the -q2 switch
b) Gogo is competitive with FhG
Unfortunately it doesn't show much about how to use FhG.
Thank you for your exhausting test.
At least it shows that
a) Gogo should be used without the -q2 switch
b) Gogo is competitive with FhG
Unfortunately it doesn't show much about how to use FhG.
I would ignore any difference that is smaller than 0.5 in my test. As you know, when you hear small artifacts, which may not be the same (i.e. when each encoder produces more or less different artifacts in different positions), it is quite difficult to put the contenders in an annoyance order. On each listening time you may rate the encoders differently. If you spend more time with each test sample the opinion may became more consistent, but as I said, I had to do the test rather quickly.
With some samples I found clear winners and losers, but also within this subgroup the winners and losers varied from sample to another.
So I think my test can prove only that I didn't find any of the contenders to be clearly worse or better in this specific test.
I wonder why the quality switches didn't work as one would like to expect? Is it possible that when the encoder does a more comprehensive analysis it can also go wrong in some cases and a more straightforward faster approach can actually produce better results?
However, I too would select the faster setting for GoGo, because I would like to get more proof of its usability in the faster default mode. *)
Regarding FhG, I think the newer 4.1 version & VBR mode should be selected, because my test didn't show any clear regression when compared with the 3.4 version. This would also make the test a VBR-only test and the average bitrates of the test contenders would be closer to each other. According to my test it would be safe to select the faster default -q value, but I really would like to see results from other forum members. Perhaps my test samples didn't reveal everything.
Edit: *) In addition, before selecting a specific -q value for GoGo we probably should pretest -q 0, -q 1, -q 3 etc too. Otherwise we would be just quessing the best value. It is better to use the default setting similarly like with Helix. FhG 4.1 VBR is a bit different case because it has only two possible -q values, which can be pretested easier.
Thanks for your test. From my understanding FHG 4.1 vbr is ABR at M4 / M5 ?
From my understanding FHG 4.1 vbr is ABR at M4 / M5 ?
I wouldn't be worried about that. It looks like the encoders produce quite varied bitrates (including GoGo ABR):
(http://i224.photobucket.com/albums/dd212/AB2K/th_vbrtable.png) (http://i224.photobucket.com/albums/dd212/AB2K/vbrtable.png)
Click to enlarge. (http://i224.photobucket.com/albums/dd212/AB2K/vbrtable.png)
Does it really make sense to include GoGo? I think not:
1) Practically nobody uses it
2) Basically it's just an ancient version of Lame (3.92 I think) with some speed optimisations
Does it really make sense to include GoGo? I think not:
1) Practically nobody uses it
2) Basically it's just an ancient version of Lame (3.92 I think) with some speed optimisations
So you didn't want to contribute this discussion by suggesting which encoders should be tested and especially which FhG settings (or version) should be selected. Just bashing GoGo does not really advance this discussion.
1) How do you know that? What encoder people usually select when they need a faster MP3 encoder than e.g. LAME? (Naturally I mean only those people who know about the options. It is obvious that most users on this planet do not change the used encoder. They just wait longer or if that is not possible they don't get everything encoded in time.)
2) It is based on LAME 3.88 and much of its code is completely rewritten in assemply language. In my test it was about 5x faster than LAME. This what the encoder displays about its version:
GOGO-no-coda ver. 3.13 ( May. 20 2004 ) is a mp3 encoder based on lame 3.88,
which is distributed under LGPL on http://www.mp3dev.org/mp3/ (http://www.mp3dev.org/mp3/) .
See http://member.nifty.ne.jp/~pen/ (http://member.nifty.ne.jp/~pen/) , http://homepage1.nifty.com/herumi/gogo_e.html (http://homepage1.nifty.com/herumi/gogo_e.html) .
After my listening test I'm even more interested about GoGo. If it really is compentive against FhG it deserves to be tested in a public test. We are trying to find the best encoders that are available for encoding MP3 files just now. AFAIK, GoGo has no technical problems. The resulting MP3 files work fine. It doesn't matter for practical purposes if the development continues or not.
2) It is based on LAME 3.88 and much of its code is completely rewritten in assemply language. In my test it was about 5x faster than LAME. This what the encoder displays about its version:
GOGO-no-coda ver. 3.13 ( May. 20 2004 ) is a mp3 encoder based on lame 3.88,
which is distributed under LGPL on http://www.mp3dev.org/mp3/ (http://www.mp3dev.org/mp3/) .
See http://member.nifty.ne.jp/~pen/ (http://member.nifty.ne.jp/~pen/) , http://homepage1.nifty.com/herumi/gogo_e.html (http://homepage1.nifty.com/herumi/gogo_e.html) .
After my listening test I'm even more interested about GoGo. If it really is compentive against FhG it deserves to be tested in a public test. We are trying to find the best encoders that are available for encoding MP3 files just now. AFAIK, GoGo has no technical problems. The resulting MP3 files work fine. It doesn't matter for practical purposes if the development continues or not.
Well, GoGo was tested years ago in this (http://www.rjamorim.com/test/mp3-128/results.html) test and came out clearly worse than the current Lame version (3.95) then. So anyone concerned about quality will avoid it. And that was years ago. How many people really need their MP3s so fast that they are ready to sacrifice quality today with all the superfast computers around? Sorry, I really don't see an audience for such a codec anymore. Just look at the almost complete lack of discussion here on HA about using GoGo for serious encoding work.
If it's really based on Lame 3.88 it's even more ridiculous to include it. Just think of the quality tuning work of 6 1/2 years that went into Lame since then, including Dibrom's Presets and Gabriel's complete reworking of the -V n modes.
I think it's a waste of time to test GoGo if there are more interesting and more widely used codecs around that can be included in the test. As the test will be very hard for most participants due to the hight quality of the codecs, I think we should limit the number of codecs as far as possible.
I think after the MP3 test we will have an AAC test with both LC and HE-AAC at 80 kbps. The winner of that test will be included in the multiformat 80 kbps test.
I was hoping 96 or 128 AAC would be tested as these are more commonly used. Are you going about this systematically from lowest bitrates to high?
Well, GoGo was tested years ago in this (http://www.rjamorim.com/test/mp3-128/results.html) test and came out clearly worse than the current Lame version (3.95) then. So anyone concerned about quality will avoid it. And that was years ago. How many people really need their MP3s so fast that they are ready to sacrifice quality today with all the superfast computers around? Sorry, I really don't see an audience for such a codec anymore. Just look at the almost complete lack of discussion here on HA about using GoGo for serious encoding work.
If it's really based on Lame 3.88 it's even more ridiculous to include it. Just think of the quality tuning work of 6 1/2 years that went into Lame since then, including Dibrom's Presets and Gabriel's complete reworking of the -V n modes.
I think it's a waste of time to test GoGo if there are more interesting and more widely used codecs around that can be included in the test. As the test will be very hard for most participants due to the hight quality of the codecs, I think we should limit the number of codecs as far as possible.
I don't think any of the now tested other encoders can be competitive against LAME. In Roberto's test GoGo was tied with all other MP3 encoders except LAME:
(http://www.rjamorim.com/test/mp3-128/plot12.png)
I have a hunch that the situation has not changed much since then.
However, as you said that test is old and the encoders and suitable settings are going to be different now.
One of the reasons to do this test now was to check if the considerably faster encoders can produce usable quality when compared with LAME. The faster options seem to be FhG, Helix and GoGo. iTunes is not practically faster, but it should be included for other reasons as discussed in this thread.
The faster encoders are useful in many situations. One example is when you want to quickly create more or less temporary files for a small portable. Another application could be on-demand WAN or LAN distribution.
It would be nice to finally select the contenders after these 14- and 6-page discussions and start the test samples discussion. In Roberto's MP3 test and in my personal test the results varied clearly from sample to another. Actually, I think that a test like this could be easily manipulated to produce certain results by selecting certain test samples. The sample selection is going to be crucial.
@sebastian, can we also include lastest Coding Technologies AAC from Winamp. both LC an HE on 80kbps test?
Could we please discuss things about the AAC test when the right time has come?
Sorry for not showing up here lately, but I had a car accident and had to sort up some things. In the meanwhile I've got my car back from the workshop and hired a lawyer to deal with the insurance company of the guy who drove into the back of my car during a traffic jam.
So, where are we... We need to decide what encoders to feature and what settings to use. One (IMO) important question: how many contenders should we have?
Summary:
List of possible and pertinent (IMO) encoders:
• Fraunhofer surround
• GoGo
• Helix
• iTunes
• LAME
List of possible ambiguities:
• Fraunhofer surround : -q0 (fast & defaut) vs -q1 (slower but labelled as "high quality mode") // ABR vs VBR
• GoGo : -q2 or not
• Helix : no ambiguity (seems that nobody suggested ABR over VBR nor any magic commandline)
• iTunes : not discussed that much IIRC ; 128 VBR would maybe work (I mean bitrate-wise)
• LAME : no ambiguity (3.98b5 -V5)
List of possible goals of this test:
• best possible MP3
• best fastest MP3 encoder outside LAME
• best MP3 among "alive" encoders
• LAME vs major companies (iTunes, Fraunhofer, Real)
Correct me if I missed something.
___
I would personally go for a LAME + Fhg + iTunes + Helix match, set with VBR when possible and with settings we would trust to be the best ones (according to the community or the developers themselves).
So, since we have no high anchor, we could go for 5 contenders + 1 low anchor or 4 contenders + 1 low anchor. If 5 contenders are OK, we can test all 5 suggested MP3 encoders including Gogo. Otherwise, we have to decide which encoder we drop.
I think after the MP3 test we will have an AAC test with both LC and HE-AAC at 80 kbps. The winner of that test will be included in the multiformat 80 kbps test.
I was hoping 96 or 128 AAC would be tested as these are more commonly used. Are you going about this systematically from lowest bitrates to high?
No please, 80 kbps is excellent! With Vorbis, it is *the* threshold bitrate that I use for my portable. The quality is good enough and not the least bit irritating (unlike 64 kbps). I think this is an essential bitrate to test and I'm glad Sebastian is testing it. Sorry for the OT, just want to show my support for the future planned tests.
So, since we have no high anchor, we could go for 5 contenders + 1 low anchor or 4 contenders + 1 low anchor. If 5 contenders are OK, we can test all 5 suggested MP3 encoders including Gogo. Otherwise, we have to decide which encoder we drop.
As difficult as the last 128 kbps multi-format test proved to be, I think the fewer contenders the better, though this theoretically won't be as difficult since different implementations of the same (older) format are being tested here.
So, since we have no high anchor, we could go for 5 contenders + 1 low anchor or 4 contenders + 1 low anchor. If 5 contenders are OK, we can test all 5 suggested MP3 encoders including Gogo. Otherwise, we have to decide which encoder we drop.
I agree. The encoders are good enough, that the high anchor can be dropped.
Are 5 contenders too many?
Are 5 contenders too many?
don't think so (this time i would try to ABX again )
Are 5 contenders too many?
I would prefer 4 but I'm not against this 5+1 configuration if it helps us to unlock the situation a bit
Let's move on to the settings...
Is --vb-new default in LAME 3.98? Should we choose the highest quality setting in iTunes? FhG -q0 or -q1? Gogo ABR or VBR, any -q setting? Helix CLI encoder or Real, what setting?
• LAME 3.98b5: --vbr-new is not needed anymore (used by default)
• iTunes: I don't see any reason to not use the highest quality level.
• Fraunhofer: same than iTunes
• GoGo: I would trust AlexB's advice on GoGo (he has an experience I don't have)
• Real: As you wish (I would personally avoid to install Real and use the CLI tool instead). VBR seems to be the way to go.
___
I finished on my side a personnal MP3 listening test based on classical samples which features four encoders. I can give the average bitrate of 150 files (16 hours of music), unfortunately not representative of compressed and/or popular music:
• LAME -V5 = 128,73 kbps
• iTunes 128 VBR HQ = 131,54 kbps
• Helix -V65 = 129,12 kbps
• Fhg -m3 -q1 = 127,86 kbps
I'm lucky that VBR settings are so much comparable bitrate-wise. Fhg seems to be downgraded to -m4 with louder music to match the targeted bitrate range; Helix can easily be adjusted if needed; iTunes needs to be tested by someone else (if it's not already done).
For information: both iTunes & Fhg are giving pretty good (but not constant) results here.
• LAME 3.98b5: --vbr-new is not needed anymore (used by default)
• iTunes: I don't see any reason to not use the highest quality level.
• Fraunhofer: same than iTunes
• GoGo: I would trust AlexB's advice on GoGo (he has an experience I don't have)
• Real: As you wish (I would personally avoid to install Real and use the CLI tool instead). VBR seems to be the way to go.
In my ABC-HR test (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=57322&view=findpost&p=515992) of 30 various samples FHG VBR -m4 -q1 was surprisingly not better than the default faster mode.
It would be nice if you could verify that with your familiar classical samples. In addition, my test package contains some interesting new samples in case you would like to try them.
I finished on my side a personnal MP3 listening test based on classical samples which features four encoders. I can give the average bitrate of 150 files (16 hours of music), unfortunately not representative of compressed and/or popular music:
• LAME -V5 = 128,73 kbps
• iTunes 128 VBR HQ = 131,54 kbps
• Helix -V65 = 129,12 kbps
• Fhg -m3 -q1 = 127,86 kbps
I'm lucky that VBR settings are so much comparable bitrate-wise. Fhg seems to be downgraded to -m4 with louder music to match the targeted bitrate range; Helix can easily be adjusted if needed; iTunes needs to be tested by someone else (if it's not already done).
For information: both iTunes & Fhg are giving pretty good (but not constant) results here.
It seems that FhG needs to be used at -m4 if VBR mode is selected. -m3 produces too hig bitrates with anything else than classical.
"if it's not already done"
- I did (I copied my results here so you don't need to scroll back.):
Here are my results. For iTunes and RealPlayer I used the standard GUI programs. For the command line encoders I used Speeks' Batchenc frontend. The WMP11 mp3 codec was accessed by Nyaochi's Acmenc and Speek's Batchenc. The test bench was a 2.8 GHz P4. Before the speed tests I tried to disable all unnecessary background processes. The source files were in wave format on one HD and the target HD was separate. I measured the encoding times with a digital stopwatch. I didn't run several passes that would have been needed for lowering the error margin, but I think the results are accurate enough for this purpose. For measuring the bitrates I used Encspot Pro's "Complete Scan" feature (some other programs may not show correct bitrates with files that don't contain a Xing header).
The test file set consists of 25 selected complete tracks of various genres. The total duration is one hour, 31 minutes and 34 seconds.
The encoding speeds and average bitrates:
(http://i224.photobucket.com/albums/dd212/AB2K/speedtable.png)
The individual bitrates of the VBR files:
(http://i224.photobucket.com/albums/dd212/AB2K/vbrtable.png)
A chart of the VBR bitrates:
(http://i224.photobucket.com/albums/dd212/AB2K/vbrchart.png)
Some readers may also find the following table interesting (Album = the 25-track test set):
(http://i224.photobucket.com/albums/dd212/AB2K/peaks.png)
I uploaded the test data here (Excel file & Encspot Pro reports):
http://www.hydrogenaudio.org/forums/index....st&p=515301 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=47370&view=findpost&p=515301)
Thanks for reminding me some elements submitted some days ago
Apparently iTunes MP3 VBR (134 kbps) is suitable for this test.
Fraunhofer q0/q1 difference: I won't test it on my side (I tried and -bored- I quickly gave up last week). I strongly believe that we should use the switch called “high quality”; even if didn't improved anything according to your test, I didn't lowered the quality either. We should always test the HQ mode and let people use the fastest mode if they're ready to take some risks (even minimal ones) by maximizing encoding speed.
I agree that Fhg speed is pretty impressive with -q0 but I'd like to give the same weapons to Fhg's encoder (HQ mode).
Also, my test of iTunes settings:
(http://i224.photobucket.com/albums/dd212/AB2K/itunesbitrates.png)
Surprisingly the quality options have almost no effect to the encoding speeds. The very small encoding time differences may be caused by other things as well (I did only one pass). I wonder if these settings do anything, even though there was a very small bitrate increase.
The Smart Encoding Adjustments and Filter Frequencies Below 10 Hz options seem to slightly affect the speed so perhaps they actually do something.
... I think the setting names Apple has chosen suggest that the best quality would be achieved @ 128 kbps VBR, Quality: Highest, Stereo Mode: Joint Stereo, Smart Encoding Adjustments & Filter Frequencies Below 10 Hz: enabled.
... Fraunhofer q0/q1 difference: I won't test it on my side (I tried and -bored- I quickly gave up last week). I strongly believe that we should use the switch called “high quality”; even if didn't improved anything according to your test, I didn't lowered the quality either. We should always test the HQ mode and let people use the fastest mode if they're ready to take some risks (even minimal ones) by maximizing encoding speed.
I agree that Fhg speed is pretty impressive with -q0 but I'd like to give the same weapons to Fhg's encoder (HQ mode).
After my test I don't have any preference. Personally, if I would need to encode something really fast I wouldn't hesitate to use the default, faster mode. It didn't show strong regression with any of my test samples. Some samples were slightly worse and some were slightly better in the default mode.
In general, it would be nice to know why -q 1 obviously reduced quality with some of the samples.
Anyway, as I said before, the sample selection is going to be crucial in case we want to make this a fear test. For example, by picking up certain 18 or 20 samples of the 30 samples I tested I could change my overall test results.
Some samples were slightly worse and some were slightly better in the default mode.
I noticed it on my side too.
I'm interestin in test like Lame vs others encoders without any speed restrictions.
I feel we can do this with 2 low anchors: LC-AAC 80 to 112 kbps, AotuvB5 also 80 to 112 kbps. The value needs to be chosen, thats all. This can greately help in finding out to what extent the claims of aac and vorbis are true. WMA 10 pro, nero E-AAC 64 also look good.
the contenders should be Lame, Fraunhofer, Helix, iTunes and the rest mutually decided niche encoders which have proved a lot in the recent months
Sorry, anchors are MP3 too.
Sebastian. Is there any dead line of this discussion? Not hurry but there isn't much activity.
Well, what we have so far is:
LAME 3.98 -V5
FhG 4.1 mode 4 with the high quality flag set
iTunes 7.4.3.1 128 VBR, highest quality (7), smart & filter enabled
What I still don't know:
Gogo - what setting?
Helix - what encoder, what setting? I have no problem with installing RealPlayer on a VM in case we choose that. The CLI version offers more fine tuning but nobody (not even the developers - contacted them during the last discussion) can recommend anything special, so maybe Real chose the best setting already.
I second AlexBs setting:
- Gogo 3.13 -a -b 135
- Helix CLI -X2 -V60
Gogo is based on Lame 3.88, and Gabriel pointed out already that at that time Lame VBR mode was in a rather early state.
As for Helix I think most members here - if interested in using Helix at all - prefer a CLI version from within foobar or other tools usage than use the Real GUI.
We should't be too ambitious towards the meaning of such a listening test outside of HA.
Should we use any special -q switch for Gogo? Gabriel, do you have any recommendations?
Anyone disagrees with the suggestion for Helix?
Sorry, no real opinion about Gogo -q parameter (would have to dig through its source code to be able to answer).
I'm just reading this thread but haven't seen the actual test results (and couldn't find them with a quick search) bit figure that the test was completed awhile back since the last post to this thread was five months ago - can someone post a link to the test results? Thanks.
The test never started.
Now that things got back to normal on my end, I could finally organize the test if people are still interested. However, I could only do so at weekends since I come back at 7 PM from work.
So people are interested after all in an MP3 test... Problem is that we never agreed on what settings to use for Helix / Real for example.
Could use -X2 -V65 which was used on guruboolez last test and avgs to around 130kbps. But I tried it on a CVS 2005.08.09 build, which I found on rarewares. For some reason it sounds really bad on a couple tracks I tried it sounds like it was encoded on Blade or a old version of L3ENC.
The problem is that not even developers know what to recommend and choosing some "weird" settings without doing listening tests first can lead to unfairness.
The problem is that not even developers know what to recommend and choosing some "weird" settings without doing listening tests first can lead to unfairness.
Also a lack of documention on all the switches on the Helix encoder does not help at, ATM I found -HF -V150 to sound fine but not transparent, I just trying to look for a low-pass filter switch, to use on V65 and use a low-pass simaler to LAME -V 5 filter, to make it more fair.
Could we not just drop this implementation from the list of contenders?
Could we not just drop this implementation from the list of contenders?
I second that.
To be honest, I would personally use Real instead of dropping the encoder just because we cannot agree on parameters. Real uses Helix and doesn't offer any special settings that might mess things up more than they help.
I have uploaded a sample that Helix really chokes on at -V65.
Replica Sample (http://www.hydrogenaudio.org/forums/index.php?act=Attach&type=post&id=4346)
Ye gods!
LAME 3.97 beats that hollow at VBR -V4 with only an 8% increase in filesize!
Cheers, Slipstreem.
So, does anyone mind using Real to encode 128 kbps VBR?
So, does anyone mind using Real to encode 128 kbps VBR?
No one minds.
Well, I think we can move on to the samples then.
Any recommendations? Whole set of new samples? Use old samples? Mix of new samples and old samples? Feel free to upload new files!
Nice to see this test back on the menu.
I too have been very busy in my "real" life and have not had time for my audio hobbies (like HA).
To be honest, I would personally use Real instead of dropping the encoder just because we cannot agree on parameters. Real uses Helix and doesn't offer any special settings that might mess things up more than they help
I just installed the new Real Player 11 on my laptop and encoded my usual test file set @ 128 kbps VBR. The resulting average bitrate was 129.7 kbps. The encoding speed was on the slow side. The speed was about 14-15x on my laptop, but this value is not directly comparable with the 17.8x speed value I got previously for RP 10.5. I didn't try to test the audio quality, but I guess that the encoder has changed very little if not at all.
Unfortunately the Real Player encoder it is not practically any faster than LAME. Helix is much faster and would be more interesting to me. I guess LAME is still going to provide superior quality and the only reason to use some other encoder would be the speed advantage. It would be useful to know how big the quality sacrifice would be if Helix or some other fast encoder was used instead of LAME.
EDIT
Here is a link to my previous bitrate test results, which I posted in last September:
http://www.hydrogenaudio.org/forums/index....st&p=515303 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=57322&view=findpost&p=515303)
Isn't Real using Helix? Maybe Real uses some switches that should optimize the quality but have impact on the speed.
Are you already locked in on the encoders? I'm a big fan of WMAstd 2-pass vbr ~128kb.
Are you already locked in on the encoders? I'm a big fan of WMAstd 2-pass vbr ~128kb.
It is a
MP3 Listening Test.
Are you already locked in on the encoders? I'm a big fan of WMAstd 2-pass vbr ~128kb.
It is a MP3 Listening Test.
Wow, how embarrassing! Guess I should read the thread title more carefully..
Since LAME 3.98 has been released I am giving this thread a little bump.
Isn't Real using Helix? Maybe Real uses some switches that should optimize the quality but have impact on the speed.
I suppose it is based on the same source, though I have no way to know exactly.
In general, I would put emphasis on the "should" word you used. As my pretest with Frauenhofer and Gogo showed the faster options may not be any worse on average. Sometimes they surprisingly produce better quality than the slower "quality" options.
I guess that not much (if anything) has changed in RealPlayer 11's MP3 encoder. I have not compared the two versions by ABX testing, but the encoding speed behavior is very similar with the v.10.5.
I can repeat my pretest using Helix & Real if necessary, but I would first like to be sure that this test will take place. I would rather not do redundant ABX testing that goes beyond of what is useful for me personally.
It would be helpful if the Real developers could shed some light on this.
Edit: typo
As a reminder, I would like to ask anyone who seriously wants to take part in the test preparations also read the preceeding discussion here: http://www.hydrogenaudio.org/forums/index....showtopic=47313 (http://www.hydrogenaudio.org/forums/index.php?showtopic=47313)
I would definitely conduct the test if we finally decided which encoders to use. By the way, I once contacted one of the Helix developers and he wasn't able to recommend any setting.
I would definitely conduct the test if we finally decided which encoders to use. By the way, I once contacted one of the Helix developers and he wasn't able to recommend any setting.
I don't think the Helix setting would be a problem. We can just use the default VBR encoding mode as suggested earlier.
I think that for the majority of HA crowd only Helix would be a usable option in case it will appear to produce acceptable quality. There is no way to use the Real Player encoder outside the Real Player with other programs and frontends. Also, since it is on the slow side there is no reason to consider using it because of the encoding speed.
Just sent a mail to Karl asking if he knows what params to choose or at least if he can tell me who is responsible. If I don't get feedback until let's say next week Tuesday, I will either stick to whatever makes sense and was recommended on HA or simply use the least switches that produce 135 kbps (so probably the settings used by Alex B).
Just received a mail from Karl informing me that he forwarded my question to the audio team since - as I supposed - he is not responsible for the MP3 encoder.
I would recommend using the Helix CLI encoder and not RealPlayer, with the options suggested in this thread. I noticed the encoding speed when used in RealPlayer was dramatically slower than CLI, and that makes me suspicious as to what how (and perhaps even which) encoder the player is actually using. I did ask our audio codec team members about recommended Helix switches, but the original developer is no longer at Real.
OK, so let's assume the Helix problem is solved (it is actually: if I get feedback from the guys from the audio department I will use their suggestions, otherwise the settings mentioned in this thread)... This leaves us with the last problem: Gogo. The question is whether or not to use an additional -q parameter.
I think it would be good to repost my old test results here because they are now hidden somewhere in the middle of this thread.
The bitrate and speed tests:Here are my results. For iTunes and RealPlayer I used the standard GUI programs. For the command line encoders I used Speeks' Batchenc frontend. The WMP11 mp3 codec was accessed by Nyaochi's Acmenc and Speek's Batchenc. The test bench was a 2.8 GHz P4. Before the speed tests I tried to disable all unnecessary background processes. The source files were in wave format on one HD and the target HD was separate. I measured the encoding times with a digital stopwatch. I didn't run several passes that would have been needed for lowering the error margin, but I think the results are accurate enough for this purpose. For measuring the bitrates I used Encspot Pro's "Complete Scan" feature (some other programs may not show correct bitrates with files that don't contain a Xing header).
The test file set consists of 25 selected complete tracks of various genres. The total duration is one hour, 31 minutes and 34 seconds.
The encoding speeds and average bitrates:
(http://i224.photobucket.com/albums/dd212/AB2K/speedtable.png)
The individual bitrates of the VBR files:
(http://i224.photobucket.com/albums/dd212/AB2K/vbrtable.png)
A chart of the VBR bitrates:
(http://i224.photobucket.com/albums/dd212/AB2K/vbrchart.png)
Some readers may also find the following table interesting (Album = the 25-track test set):
(http://i224.photobucket.com/albums/dd212/AB2K/peaks.png)
I uploaded the test data here (Excel file & Encspot Pro reports):
http://www.hydrogenaudio.org/forums/index....st&p=515301 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=47370&view=findpost&p=515301)
My listening test:As I promised, I tried to do a "quick pretest" with a few samples using foobar's ABX module, but that didn't work. The codecs' quality didn't have immediately obvious differences and the results would have been almost arbitary depending on the used samples. So I felt kind of obligated to do a full-scale personal test.
I used the following codec settings:
FhGc = FIIS 3.4 CBR 128 kbps
FhGv = FIIS 4.1 VBR -br 0 -m 4 -vbri -ofl
FhGv1 = FIIS 4.1 VBR -br 0 -m 4 -q 1 -vbri -ofl
GoG = GoGo 3.13 ABR -a -b 137
GoG2 = GoGo 3.13 ABR -a -b 137 -q 2
Edit: I forgot to mention that I used FhG Mp3sdecoder v. 1.3 for decoding the samples. (Looks like the -ofl option worked. ABC-HR Java didn't adjust the starting position of the FhG VBR samples)
After seeing the results of my previous speed and bitrate test I upped the GoGo ABR bitrate setting from 135 to 137 so that it would better match the codec average. FhG VBR produced still a bit higher average bitrate with the now tested new samples (134 kbps vs. 131 kbps).
I tested 30 varied samples. All samples are created by myself and most of them are brand new. About half of them are from Vangelis' Alexander soundtrack album, which has a lot of interesting music (it's a mixture of classical, electronic and ethnic genres) and I wanted to try if that material would be useful for testing codecs.
The reference samples and my ABC-HR result files are available in this link:
http://rapidshare.com/files/55027379/alexb_test.zip (http://rapidshare.com/files/55027379/alexb_test.zip) (about 76 MB)
Here are the test results:
% Result file produced by chunky-0.8.4-beta
% Chunky -d C:\alexb_test\results -r sample -f C:\alexb_test\codecs.txt
%
% Sample Averages:
S.# FhGc FhGv FhGv1 GoG GoG2
#01 2.90 3.50 3.50 4.00 3.50
#02 3.90 3.30 3.30 2.40 3.30
#03 4.00 3.60 3.90 3.60 4.00
#04 4.00 4.00 4.00 4.00 4.00
#05 3.00 4.10 3.20 3.70 3.40
#06 4.00 3.60 3.30 4.50 2.90
#07 4.00 3.50 4.10 3.30 4.50
#08 4.00 3.80 4.40 4.40 4.20
#09 4.10 3.80 3.20 4.20 3.50
#10 4.30 4.30 3.70 4.50 3.40
#11 3.00 2.50 2.40 3.00 3.00
#12 3.70 4.10 4.00 4.20 3.50
#13 3.20 3.20 3.90 3.70 2.80
#14 3.40 2.80 2.20 3.50 2.50
#15 4.00 4.30 3.90 3.90 4.20
#16 3.20 3.70 3.90 3.70 3.00
#17 3.90 3.10 3.00 4.00 4.00
#18 3.80 4.10 3.90 4.10 3.80
#19 3.80 2.90 3.10 3.90 3.90
#20 3.10 3.10 3.70 3.90 1.90
#21 2.30 2.70 3.00 2.20 2.40
#22 3.90 3.90 3.90 3.90 4.30
#23 3.90 4.00 4.00 4.00 3.80
#24 4.20 4.20 4.20 3.90 4.20
#25 4.20 4.00 4.10 4.00 3.60
#26 3.90 3.70 4.00 3.70 3.70
#27 3.80 4.10 4.00 3.80 3.80
#28 3.50 3.60 3.60 4.00 3.20
#29 2.60 2.80 3.00 1.60 1.50
#30 3.70 3.70 3.50 3.70 3.30
%%% Codec averages:
%%% 3.64 3.60 3.60 3.71 3.44
Even though I did the test properly, I wouldn't trust the results too much.
Mostly I was able to differentiate the contenders without ABXing because of the lowpass. All encoders seem to use an about 15-16 kHz lowpass frequency at this bitrate/quality level. Beyond that the test was a tough call. The codecs were generally quite good and the slight lowpass didn't make the samples annoying (to me, it seems to be detectable only in a direct comparison with the reference).
I didn't want to spend several days with this test so I rated the samples rather quickly and it is possible that a new test with the same samples would produce slightly different results.
GoGo failed misarably on one killer sample, the sample number 29 (also FhG was bad with this sample, but better than GoGo). On the other hand FhG's VBR has obvious problems with some other samples.
I can make the following conclusions:
All encoders are tied and the higher quality options didn't make the encoders better.
Edit: Fixed a couple of typos and added the MP3 decoder info.
The test files are still available in the Rapidshare link. In addition, I created a couple of new pictures of the listening test results:
(http://i224.photobucket.com/albums/dd212/AB2K/128mp3test/d4fa400f.png)
(http://i224.photobucket.com/albums/dd212/AB2K/128mp3test/82c8268e.png)
OK, so let's assume the Helix problem is solved (it is actually: if I get feedback from the guys from the audio department I will use their suggestions, otherwise the settings mentioned in this thread)... This leaves us with the last problem: Gogo. The question is whether or not to use an additional -q parameter.
My conclusion after rechecking my results would be to use the faster default options with GoGo and FhG VBR.
Thinking about it right now, I am wondering if we should even include Gogo in the test... The problem is:
- CBR would tie Gogo's hands and is unfair
- VBR is not mature in Gogo's LAME "engine"
- ABR gives different results when encoding samples and when encoding the corresponding full track - we had this problem when it came to WMA's 2-pass VBR encoding
Seeing that FhG CBR 128 kbps without any quality switch is almost 80x, I would tend to use that instead of Gogo.
BTW, is Gogo 3.13 based on LAME 3.88 or LAME 3.9x? The website states the latter, but when you run the build from RW, it says that it's based on 3.88.
Thinking about it right now, I am wondering if we should even include Gogo in the test... The problem is:
- CBR would tie Gogo's hands and is unfair
- VBR is not mature in Gogo's LAME "engine"
- ABR gives different results when encoding samples and when encoding the corresponding full track - we had this problem when it came to WMA's 2-pass VBR encoding
ABR shouldn't be a problem. It is nothing like 2-pass encoding. AFAIK, the ABR "frame" is very short. Perhaps the LAME developer's could tell how exactly it works.
Seeing that FhG CBR 128 kbps without any quality switch is almost 80x, I would tend to use that instead of Gogo.
That's an interesting idea. Unfortunately I didn't include FhG 4.1 CBR in my pretest. It wasn't considered as a potential contender back then. As you can see FhG 3.4 CBR wasn't bad at all in my test. It provided the most constant quality.
BTW, is Gogo 3.13 based on LAME 3.88 or LAME 3.9x? The website states the latter, but when you run the build from RW, it says that it's based on 3.88.
My understanding is that it is based on 3.88. It was originally developed before LAME 3.90 was released and I guess the original codebase was not changed when newer versions were introduced.
My understanding of ABR is that the the encoder also "memorizes" what bitrate it used on previous frames and allocates bits for future frames taking this "memory" into consideration - sort of like the bit reservoir works, just that it allows larger fluctuations. The problem I see is that an encoded sample might have another bit allocation than if you were to encode the whole track and then cut out the part you want to test.
My understanding of ABR is that the the encoder also "memorizes" what bitrate it used on previous frames and allocates bits for future frames taking this "memory" into consideration - sort of like the bit reservoir works, just that it allows larger fluctuations. The problem I see is that an encoded sample might have another bit allocation than if you were to encode the whole track and then cut out the part you want to test.
I checked a few 44.1 kHz MP3 files and it appears that there's about 40 MP3 frames in one second of audio. I think this theoretical problem can practically affect only a few frames.
Also, I don't think CBR encoding is any different when bit reservoir is used. I searched and found this old discussion:
... ABR is one pass? How can it approximate a desired bitrate then? IMHO it needs to do a quick look on the file at least once.
... ABR is one pass. Well, here is how I think it works, anyone please confirm or correct me. In CBR there is something called bit reservoir. When a frame is to be encoded to say 128 kbps, the encoder may not match this exactly. In some frames there will be empty space, and in some the data just won't fit. So what the encoder can do when there is to much data, is to move this to a frame where there is empty space. How this works in detail I have no clue, but it is referred to as the bit reservoir.
The problem is that there is not always enough space in the bit reservoir when doing CBR. My guess is that in ABR this is skipped and the frame size is instead adapted to what the encoder wants.
Anyone wanna comfirm this, or correct me?
Gabriel didn't correct him:
ABR is the same thing as CBR with an unlimited bit reservoir (emulated by changing frame sizes)
After seeing the recent LAME 3.98 related posts I think we should test LAME 3.97 and LAME 3.98.
Perhaps:
LAME 3.97 VBR
LAME 3.98 VBR
FhG VBR
Helix VBR
iTunes VBR
The correct setting for LAME 3.98 may be something like -v 5.7. It produces 133.3 kbps on average with my test file set.
- ABR gives different results when encoding samples and when encoding the corresponding full track - we had this problem when it came to WMA's 2-pass VBR encoding
couldn't you encode the entire song, then cut out the sample portion?
I am particularly interested in ABR since that is what I use. My ipod seems to choke on VBR
couldn't you encode the entire song, then cut out the sample portion?
I am particularly interested in ABR since that is what I use. My ipod seems to choke on VBR
LAME ABR has not even been considered. Only GoGo ABR.
I would advice anyone who is interested in a specific encoder and setting combination that will not be included in the public test to first practice by participating the public test and after that do a new personal test using the same reference samples and the preferred encoder & setting. Most likely the results of that kind of personal test would also be valuable for others.
EDIT
I don't think cutting MP3 files would be accurate enough. It can be done only on the MP3 frame boundaries and there is no simple way to detect the correct passage. In addition, there's the above mentioned bit reservoir system causing more difficulties. Practically the files would need to be decoded and the tedious cutting work would need to be done with a wave editor. After that the samples would need to be included in a lossless or uncompressed format in the test package.
This would need to be done if a 2-pass encoder would be included in a listening test. As I said, LAME (or GoGo) ABR should not be a problem.
After seeing the recent LAME 3.98 related posts I think we should test LAME 3.97 and LAME 3.98.
I second that. This is a good time to compare LAME 3.98 and see how much it has improved upon LAME 3.97.
Do we need to test 3.97 vs. 3.98 in a public listening test? That was never considered in this test. What do LAME developers think?
I am not a Lame dev and I think that its a bad idea for a public test. Too difficult for most people - harder than a normal test 130 k test which is already hard enough for many.
As I said in a previous post - given the fact that Gogo is not developed any longer (at least quality-wise) and that VBR was not mature in 3.88 according to Gabriel if I recall correctly and therefore not recommended, I fancy the idea to test FhG CBR just to have a comparison between high quality encoders (LAME, FhG with Q switch and VBR, iTunes using best quality settings) and fast encoders (Helix VBR and FhG CBR without Q switch).
My understanding of ABR is that the the encoder also "memorizes" what bitrate it used on previous frames and allocates bits for future frames taking this "memory" into consideration - sort of like the bit reservoir works, just that it allows larger fluctuations. The problem I see is that an encoded sample might have another bit allocation than if you were to encode the whole track and then cut out the part you want to test.
IMHO, if you don't test the first second of encoded audio, that should be enough to let the encoder adapt itself correctly for Lame-based encoders. Within Lame (up to now), the ABR window is quite short.
Do we need to test 3.97 vs. 3.98 in a public listening test? That was never considered in this test. What do LAME developers think?
I'd say no. People might do some side tests about 3.98 vs 3.97 in order to help Lame's development, but I don't see the purpose within a multi-encoders listening test.
My understanding of ABR is that the the encoder also "memorizes" what bitrate it used on previous frames and allocates bits for future frames taking this "memory" into consideration - sort of like the bit reservoir works, just that it allows larger fluctuations. The problem I see is that an encoded sample might have another bit allocation than if you were to encode the whole track and then cut out the part you want to test.
IMHO, if you don't test the first second of encoded audio, that should be enough to let the encoder adapt itself correctly for Lame-based encoders. Within Lame (up to now), the ABR window is quite short.
Do we need to test 3.97 vs. 3.98 in a public listening test? That was never considered in this test. What do LAME developers think?
I'd say no. People might do some side tests about 3.98 vs 3.97 in order to help Lame's development, but I don't see the purpose within a multi-encoders listening test.
I'd say yes. LAME is one of the most important mp3 encoders ever made, so let's see if 3.98 is a step up from 3.97. This is probably one of the best opportunities to do so. It is a mp3 listening test after all ...
My understanding of ABR is that the the encoder also "memorizes" what bitrate it used on previous frames and allocates bits for future frames taking this "memory" into consideration - sort of like the bit reservoir works, just that it allows larger fluctuations. The problem I see is that an encoded sample might have another bit allocation than if you were to encode the whole track and then cut out the part you want to test.
IMHO, if you don't test the first second of encoded audio, that should be enough to let the encoder adapt itself correctly for Lame-based encoders. Within Lame (up to now), the ABR window is quite short.
I would like to add, that
all modes LAME has to offer use the same information:
- psymodel: uses info from two previous granues (one MPEG-1 frame, two MPEG-2 frames).
- quantization loops: use info from one previous frame
So, ABR doesn't do anything special compared to CBR or VBR here.
LAME's ABR doesn't even observe the actual average bitrate.
LAME's ABR really is working like its CBR, but:
- ABR bit allocation uses:
* base amount from target bitrate, base_bits
* bits from 320 kbps frame plus bitreservoir minus base_bits, depending on PE
- CBR bit allocation uses:
* base amount from target bitrate, base_bits
* bits from actual frame plus bitreservoir minus base_bits, depending on PE
Maybe I did the same mistake that some others have done when they ask to include a specific codec or setting. I asked to include an additional encoder (LAME 3.97) without carefully considering if that would serve the purpose of this test. I just saw a great oppurtunity to actually compare 3.97 and 3.98 in a public test. I would like to make sure that it is safe to use 3.98 instead of 3.97. /mnt's report might indicate that with some samples the quality may be regressed (especially because /mnt used LAME 3.98 with a setting that gives a bitrate advantage. LAME 3.98 -V5 appears to produce higher bitrates than 3.97 -V5 --vbr-new).
In general, I have not seen many personal reports of 3.98 at "about 130-135 kbps VBR" during the development cycle. However, as I already said, it should be easy to me and others to personally test LAME 3.97 using the samples from the public test.
As I said in a previous post - given the fact that Gogo is not developed any longer (at least quality-wise) and that VBR was not mature in 3.88 according to Gabriel if I recall correctly and therefore not recommended, I fancy the idea to test FhG CBR just to have a comparison between high quality encoders (LAME, FhG with Q switch and VBR, iTunes using best quality settings) and fast encoders (Helix VBR and FhG CBR without Q switch).
Do you have a reason to believe that FhG VBR -q 1 would provide better quality than FhG VBR -q 0 (aka default) ? My personal test didn't indicate anything like that, but naturally it is only a limited proof.
Personally I am not convinced that adding a CBR encoder would be a good idea, but I am not strictly against it. I have no opinion about FHG
v. 4.1 CBR because I have not tested it in the CBR mode.
After all, I would be good with only four VBR contenders and the low anchor. It would make the test easier.
Thanks for the information Gabriel and Robert. I decided to start a poll for the fifth contender having LAME 3.97 vs. FhG CBR (as very fast encoder) vs. Gogo ABR.
And here is the poll: http://www.hydrogenaudio.org/forums/index....showtopic=64446 (http://www.hydrogenaudio.org/forums/index.php?showtopic=64446)
Do you have a reason to believe that FhG VBR -q 1 would provide better quality than FhG VBR -q 0 (aka default) ?
Yes, it is labled as high quality switch so I have the right to assume that. I am doing it just to be on the safe side since I don't have the ability to contact the developers and ask for whatever setting they recommend.
Yes, it is labled as high quality switch so I have the right to assume that. I am doing it just to be on the safe side since I don't have the ability to contact the developers and ask for whatever setting they recommend.
Unfortunately other users have not posted ABX results. Personally I can only trust my ABX results (TOS 8). Actually, is it impossible to reach the FhG developers? Surely they should be interested in this matter.
From the LAME 3.98 "news" thread:
Do you guys recommend -V5.7 to be used in my test?
LAME 3.98 -V5 results 141.7 kbps with my test track set.
Unless Robert and Gabriel suggest something else I think we should proceed in the usual way:
1. Take the contenders that do not allow intermediate quality settings and find the average bitrate of all of these contenders.
2. Test and find the suitable settings for the other contenders. The settings that provide the closest bitrate match with the average bitrate from the "step 1" should be selected.
EDIT
A possible CBR encoder should be excluded from the step 1. Including it would lower the "step 1" average and give slight unfair advantage to the "step 1" VBR encoders.
I think also this is more on topic here:
Do you guys recommend -V5.7 to be used in my test?
I think NO coz there is no possibility to make all participating coders to produce equal resulting bit rates. So the use of real-life settings (resulting in acceptable bit rates) seems to be a reasonable compromise.
Hmm... what you mean by "real-life"?
The developers have obviously decided to let the average bitrate of LAME -V5 increase. I can't believe they would have not constantly tested the resulting bitrates. Someone could call this change cheating, but I think they have just redesigned the settings scale. Why should we not use the new scale and do what should be considered fair?
Hmm... what you mean by "real-life"?
The developers have obviously decided to let the average bitrate of LAME -V5 increase. I can't believe they would have not constantly tested the resulting bitrates. Someone could call this change cheating, but I think they have just redesigned the settings scale. Why should we not use the new scale and do what should be considered fair?
I meant that most people use integer values in practice ....
But right you are, - every listening test has its own fair policy. If this one already has some agreed procedure of choosing encoder settings, it should be kept, regardless of any "integer" values for sure.
... But right you are, - every listening test has its own fair policy. If this one already has some agreed procedure of choosing encoder settings, it should be kept, regardless of any "integer" values for sure.
I checked the three previous public listening tests:
Ogg Vorbis AoTuV 4.51 Beta
-q 4.25 and Nero HE-AAC Jul 20 2007
-q 0.24have been chosen in similar situations and the used procedure for finding the correct setting was like I suggested now.
If LAME is now supposed to be similarly "stepless" an intermediate setting can be used when it is closer to the test target.
EDIT
The test's "target bitrate" should probably be announced as 130-135 kbps or so.
Another option would be to pick LAME 3.98 -V5 as a reference and try to match the other encoder's settings with it. The test target would then probably be ~140 kbps.
Unfortunately other users have not posted ABX results. Personally I can only trust my ABX results (TOS 8). Actually, is it impossible to reach the FhG developers? Surely they should be interested in this matter.
I am very thankful for the time you invested in testing and although your conclusion is that there is no difference between the two settings, I am going to use the quality switch because of the aforementioned reasons. According to the Fraunhofer IIS page (http://www.iis.fraunhofer.de/EN/bf/amm/projects/mp3/index.jsp), FhG recommends using the maximum quality setting available in the encoder interface:
How do I achieve perfect audio quality using MP3?
[...]
2) [...]If you use a variable bitrate setting, choose the maximum quality setting available in the encoder interface.[...]
IMO the bitrate difference should not be bigger than +-2.5 kbps.
I found that the only scenarios that will work are those two:
135-140 kbps, if there should be iTunes VBR and Lame -V5:
1) Lame 3.98: -V5
137 kbps
2) FhG 1.4 (Encoder-Library V04.01.00): -br 0 -m 3 -q 1
138 kbps
3) iTunes 7.6.2.9: http://img77.imageshack.us/img77/9634/itunesmp3nf1.png (http://img77.imageshack.us/img77/9634/itunesmp3nf1.png)
135 kbps
4) Helix v5.1 2005.08.09: -V72 -X2 -SBT450 -TX0 -HF2
137 kbps
(lame 3.97)
127 (-V5 --vbr-new); 150 (-V4 --vbr-new)
those settings would disqualify lame 3.97
Or the ~128 kbps range:
1) Lame 3.98: -V5.8
128 kbps
2) FhG 1.4 (Encoder-Library V04.01.00): -br 0 -m 4 -q 1
128 kbps
3) iTunes 7.6.2.9: settings from above but CBR
128 kbps
hitting the 128 range with maximum VBR is not possible due to the lower limit at 128 kbps . Maximum VBR with 112 kbps doesn't work either (bitrate too low).
4) Helix v5.1 2005.08.09: -V62 -X2 -SBT450 -TX0 -HF2
128 kbps
5) Lame 3.97: -V5 --vbr-new
127 kbps
I'm not sure about the Helix settings at that low bitrate though.
As most people so far want to see Lame 3.97 as the 5th candidate yielding 128 kbps seems to be the way to go.
As always, I am going to build a bitrate table containing the average bitrate an encoder gave to all samples. If the difference between the encoder with the lowest average bitrate and the encoder with the highest average bitrate is greater than 10% of the target bitrate, I will think about either exchanging some of the samples or choosing different encoding parameters. The initial parameters will be chosen based on recommendations from developers and based on a large set of music tracks that should average to our target bitrate.
By the way, this means that in Raiden's example, the first scenario with LAME 3.97 -V5 --vbr-new would still be a valid solution. -V4 is also within the 10% range, but in this case, I would choose -V5 in order to compare the same quality setting between 3.97 and 3.98 and also because 127 kbps is closer to the average bitrate of the four oher encoders than 150 is.
Raiden,
What test tracks did you use and what tool did you use for checking the bitrates?
I would like to see lame 3.98 as the new recommended encoder. And to do so, tests has to be done. *vote for lame 3.97 AND 3.98*
Raiden,
What test tracks did you use and what tool did you use for checking the bitrates?
ooops... sorry! I should have put that info in the post above.
I haven't got many different genres on my PC (mostly electronic music) so this was just a quick test with a small 'representative' selection (just six albums, 375 minutes): Some electronic music/IDM, ambient, Jazz/NDW, Rock, and a radio play with lots of effects. I guess for modern, compressed music one has to add ~5 kbps to the values.
I've read the bitrate with foobar2000 (after fixing the VBR headers on the FhG files), averaged over time...
So no big conclusions, but there are a few:
- lame 3.98 MP3s are ~10 kbps bigger than their 3.97 counterparts at all quality levels, so direct comparison (i.e. 3.97 -V5 vs. 3.98 -V5) is not so good.
- FhG's VBR scale is not fine at all, whereas with Lame and Helix there is no problem. So FhG would likely be the 'bitrate reference'.
- iTunes VBR is very difficult. 112-VBR results in a bitrate of ~120 kbps, and 128-VBR in ~135 kbps, but I have not found an inbetween VBR setting for 128 kbps.
I'm very curious about bitrates from bigger testcases!
As I said already, if the target bitrate is 128 kbps (which it is), a difference of 12.8 (or let's round to 13) kbps is allowed between contenders (again, on average, not per sample - per sample the difference can be higher).
LAME developers, in case the bitrate difference between LAME 3.97 and LAME 3.98 is too big, is there anything I can do to increase the bitrate of LAME 3.97 without having a negative impact on audio quality?
Alex B, any chance you could post a bitrate table for your samples with LAME 3.97 -V5 --vbr-new to have a direct comparison?
LAME developers, in case the bitrate difference between LAME 3.97 and LAME 3.98 is too big, is there anything I can do to increase the bitrate of LAME 3.97 without having a negative impact on audio quality?
Or else, still include tried 'n' true 3.97 at -V5 --vbr-new, and play around with the new stepless quality settings on 3.98, to encode at - say - V5.5?
Problem is that all other encoders also produce bitrates near the 140 kbps range. The only encoder that (according to Raiden - I didn't had time to test myself that is why I asked Alex B if he can also test) encodes even "lower" than the target bitrate is LAME 3.97.
Personally, I wouldn't mind a bitrate difference of 10 kbps (which is even less than 10%) since I assume the difference to be very low between the two LAME versions and if it's not, it is most likely caused by other factors not only the bitrate.
Alex B, any chance you could post a bitrate table for your samples with LAME 3.97 -V5 --vbr-new to have a direct comparison?
I have already tested it. LAME 3.97 -V5 --vbr-new produces about 134 kbps with my sample set. This has not changed since I tested 3.97 beta 2 for the 128 kbps multiformat test. As you can see from my previous table also 3.98 beta 5 was quite similar. After that the scale has changed and now 3.98 is different. The obvious solution would be to adjust LAME 3.98 to exactly match 3.97 or the test average, but as you said, we have some tolerance and naturally we should not use a setting that would decrease the quality if the next minor setting "step" should produce clearly better quality.
I would like to ask Robert and Gabriel if there are some fundamental changes when, for instance, the setting is changed from -V5 to -V5.5 or -V5.7. In other words, should the integer steps still be preferred for some reason or is the -V scale now truly stepless? It would be good to have info about that before the encoding setting for LAME 3.98 is decided.
I am going run a few additional tests and post a new table soon. My test set has usually been quite good in representing the "various" genre aka an average of pop/rock/jazz/electronic genres even when it has been compared with big encoded libraries, but in addition, I am gathering a modest set of classical tracks and I'll publish those results in separate table.
Thanks a lot for your efforts and help! Problem with my music is that most of it is progressive rock. I will try to do some tests in the near future (today or tomorrow evening).
If LAME 3.98 appears to be the only encoder that stands out of the rest, that would hopefully be no problem since you can fine tune the -V parameter.
I would like to ask Robert and Gabriel if there are some fundamental changes when, for instance, the setting is changed from -V5 to -V5.5 or -V5.7. In other words, should the integer steps still be preferred for some reason or is the -V scale now truly stepless?
Only yes/no settings in the presets may change on integer steps. All other parameters--except sfb21mod--are linear interpolated between step n and n+1. There is no reason to prefer integer steps.
Small status update:
- Real did not conact me
- Gogo will be replaced by LAME 3.97 -V5 --vbr-new
- LAME 3.97 -V5 --vbr-new produces an average bitrate of 136 kbps according to foobar2000 based on my collection of 1083 files
Since all encoders are close to the 135 kbps rate, we should choose some settings for LAME 3.98 that should average to 135 kbps.
Small status update:
- Real did not conact me
- Gogo will be replaced by LAME 3.97 -V5 --vbr-new
- LAME 3.97 -V5 --vbr-new produces an average bitrate of 136 kbps according to foobar2000 based on my collection of 1083 files
Since all encoders are close to the 135 kbps rate, we should choose some settings for LAME 3.98 that should average to 135 kbps.
For my (much smaller ) test set: LAME 3.97 -V5 --vbr-new => 136.8 kbps. And LAME 3.98 produces the same bitrate somewhere at -V5.33.
And dependence between -V value and bitrate looks very linear within [5.2, 5.8] range.
Please suggest and / or upload samples here (http://www.hydrogenaudio.org/forums/index.php?showtopic=64673).
FhG released MP3S CLE V1.5, but the download link is dead, they will fix download link soon:
http://www.all4mp3.com/tools/sw_fhg_cl.html (http://www.all4mp3.com/tools/sw_fhg_cl.html)
Whats New in Version 1.5:
New features - Encoder
Minor changes and bugfixes.
Now supporting files with 24 bit resolution
Finally, download link is alive, download it here:
http://www.all4mp3.com/dev/download.aspx?n...15_20080530.zip (http://www.all4mp3.com/dev/download.aspx?nolog=1&file=mp3sEnc_Win32_FCRv15_20080530.zip)
what you think about lame 3.98 -V floatnumber multi-pass encode to get most close V to 128kbps size ("true abr")?
or this mode is not good for users and should be not tested (need more time for encode, etc)
Afaik there is no multipass switch in LAME.
Finally, download link is alive, download it here:
http://www.all4mp3.com/dev/download.aspx?n...15_20080530.zip (http://www.all4mp3.com/dev/download.aspx?nolog=1&file=mp3sEnc_Win32_FCRv15_20080530.zip)
No the link is broken, here it is: Fraunhofer IIS mp3surround command line utilities (http://www.all4mp3.com/tools/sw_fhg_cl.html).
Afaik there is no multipass switch in LAME.
Also, multipass encodes would have to be done on the whole songs, using it with short samples would produce skewed results. (I think this was discussed before wrt Nero or WMA.)
ok, case closed
Finally, download link is alive, download it here:
http://www.all4mp3.com/dev/download.aspx?n...15_20080530.zip (http://www.all4mp3.com/dev/download.aspx?nolog=1&file=mp3sEnc_Win32_FCRv15_20080530.zip)
No the link is broken, here it is: Fraunhofer IIS mp3surround command line utilities (http://www.all4mp3.com/tools/sw_fhg_cl.html).
Download link works fine:
http://www.all4mp3.com/dev/download.aspx?n...15_20080530.zip (http://www.all4mp3.com/dev/download.aspx?nolog=1&file=mp3sEnc_Win32_FCRv15_20080530.zip)
Check your connection.
Finally, download link is alive, download it here:
http://www.all4mp3.com/dev/download.aspx?n...15_20080530.zip (http://www.all4mp3.com/dev/download.aspx?nolog=1&file=mp3sEnc_Win32_FCRv15_20080530.zip)
No the link is broken, here it is: Fraunhofer IIS mp3surround command line utilities (http://www.all4mp3.com/tools/sw_fhg_cl.html).
Download link works fine:
http://www.all4mp3.com/dev/download.aspx?n...15_20080530.zip (http://www.all4mp3.com/dev/download.aspx?nolog=1&file=mp3sEnc_Win32_FCRv15_20080530.zip)
Check your connection.
I've checked once again link (http://www.all4mp3.com/dev/download_error.aspx). My connection is OK.
These are the links that work for me:
Win32 mp3 command line encoder (v1.5) (http://www.all4mp3.com/tools/download.aspx?eula=thm_eula.html&File=mp3sEnc_Win32_FCRv15_20080530.zip&Name=Win32%20mp3%20command%20line%20encoder)
Win64 mp3 command line encoder (v1.5) (http://www.all4mp3.com/tools/download.aspx?eula=thm_eula.html&File=mp3sEnc_Win64_FCRv15_20080530.zip&Name=Win64%20mp3%20command%20line%20encoder)
Mac OS X mp3 command line encoder (v1.5) (http://www.all4mp3.com/tools/download.aspx?eula=thm_eula.html&File=mp3sEnc_MacOSXuniv_FCRv15_20080530.zip&Name=MacOSX%20mp3%20command%20line%20encoder)
Linux32 mp3 command line encoder (v1.5) (http://www.all4mp3.com/tools/download.aspx?eula=thm_eula.html&File=mp3sEnc_Lin32_FCRv15_20080530.zip&Name=Linux32%20mp3%20command%20line%20encoder)
Linux64 mp3 command line encoder (v1.5) (http://www.all4mp3.com/tools/download.aspx?eula=thm_eula.html&File=mp3sEnc_Lin64_FCRv15_20080530.zip&Name=Linux64%20mp3%20command%20line%20encoder)
Sorry, but I can't help...
Yes, you're right.
I don't know what they doing.
Did you download that?
Also, what's new in this version?
I finally had time to continue my bitrate tests with classsical music as I promised earlier in this thread (a thing called summer got in the way...)
After browsing through my lossless classical library I picked 25 "reference" tracks that should be quite representative. I avoided the extremely low and high lossless bitrates and tried to select tracks that have quite varied qualities.
Apparently iTunes has changed radically since my last test. Back then the 128 kbps VBR setting was suitable, but the 7.7 version uses bitrates in a more relaxed way and the 128 kbps VBR setting produces higher bitrates than before. Fortunately the 112 kbps VBR setting appears to be suitable for our test.
In addition, I retested the "various" bitrates with the latest encoder versions when applicable.
Since I didn't have the old iTunes version installed I couldn't test the "classical" bitrates with it (which would have been unnecessary anyway).
FhG, iTunes and LAME 3.97 have only one suitable VBR setting for this test so I'd suggest to calculate the average bitrate of these encoders and adjust the Helix and LAME 3.98 settings to match this average. Helix -V60 and LAME -V5.7 appear to be pretty close to this average with my test tracks.
Here are the new results:
Summary
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/8402bb15.png)
Various - table and chart
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/c4166eec.png)
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/98f51e5b.png)
Classical - table and chart
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/b09a25fc.png)
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/1eb30cbd.png)
EDIT
I forgot to mention that if anyone wants to test FhG's bitrate behavior the bitrates must be measured correctly. Most programs don't show accurate bitrate values because FhG doesn't write Xing headers to VBR files.
I used EncSpot Pro's "full scan" option.
Thank you very much for your efforts Alex!
Mmm... there's a curious thing on those graphs, the performance of Itunes respect of the rest in a couple of files:
17 in the Various group, and in 5 and 19 in the Classical one. I guess this could point to flaws (killer samples) for it. Could you try abx'ing one of them?
(the 11 in the Various test is curious too, it shows an increase of bitrate where all the rest do the opposite)
Mmm... there's a curious thing on those graphs, the performance of Itunes respect of the rest in a couple of files:
17 in the Various group, and in 5 and 19 in the Classical one. I guess this could point to flaws (killer samples) for it. Could you try abx'ing one of them?
(the 11 in the Various test is curious too, it shows an increase of bitrate where all the rest do the opposite)
Yeah, displaying the measured values in a graph always shows interesting differences. We must remember that these values are from complete tracks and do not indicate how the encoders adjust bitrates on different parts of each track. Also, the bitrate is only one of the variables that a VBR encoder adjusts.
Since you are interested about the differences I'll check the tracks you mentioned and if they seem to be interesting for quality test purposes I'll post samples.
EDIT
I wonder what is wrong with the quotation system. The quote doesn't appear to show up correctly. I tried to fix it a couple of times.
It changes
[quote name='[JAZ]' date='Aug 20 2008, 21:14' post='583720']
to
[quote]' date='Aug 20 2008, 21:14' post='583720']
Moderation: Fixed your bracket problem. ;)
I wonder what is wrong with the quotation system. The quote doesn't appear to show up correctly. I tried to fix it a couple of times.
It doesn't like brackets in names because IB.Code tags are also enclosed in brackets. Bug is fixed in latest IPB 2.3.5.
type
"&-#-9-1-;" for a "["
and
"&-#-9-3-;" for a "]"
(but don't include the dashes)
It doesn't like brackets in names because IB.Code tags are also enclosed in brackets. Bug is fixed in latest IPB 2.3.5.
(iPods not dealing well with LAME VBR)
What do you mean by that?
Great Job Alex. I think it's "sad" getting
188 Kbps files while using iTunes
128 VBR !!!
It is a shame. I expressed concerns about not giving iTunes a fair shake last year, but if their 128 setting produces such high bitrates...
IIRC, with iTunes in VBR mode, the selected bitrate is the minimum bitrate.
Well, any news?
It was already one month and half since call for samples. What is situation around the test?
Thank you for fast reply and organization of test that never conducted.
Never mind.
Thank you for fast reply and organization of test that never conducted.
Never mind.
Even if I too deplore this thread and any public HA listening test was entirely becalmed, there's no need to get this bitter. Nobody stops you from organizing one of your own, in all its complexities, after all.
I have obtained one top secret information.....
Sebastian will most probably start test today
Hey guys, just a small status update:
1. First of all, it's not easy to organize listening tests and I have been pretty busy with private stuff lately.
2. A new LAME version was released on RareWares yesterday. It doesn't matter if it adds changes that affect audio quality because people usually only look at the version and then ask "is 3.98.x now better than 3.98?", "why didn't you test 3.98.x"? At least the known bugs should be fixed so I don't expect a new build to pop up soon so that I can take 3.98.2 for the test.
3. I am currently having problems with hosting the samples. Roberto was kind enough to host them for me in the past but I was unable to reach him and I cannot host them myself because of the copyright situation in Germany.
I don't think I will be able to make it today because I just got home from work, but I will really do my best to start the test this week. Thanks to everyone who submitted samples by the way.
Edit: IgorC, For some reason I wasn't notified that you replied to my thread that is why I didn't even notice. Re-subscribed now - hopefully it works.
I am currently having problems with hosting the samples. Roberto was kind enough to host them for me in the past but I was unable to reach him and I cannot host them myself because of the copyright situation in Germany.
I could do that for you. Straight FTP though, no torrents.
Thanks! What kind of connection do you have? Any traffic limits?
Thanks! What kind of connection do you have? Any traffic limits?
I happen to work for an ISP, so I have my own little private web/FTP/mail server hooked up to 2 redundant Gbit uplinks in a western European data centre. No traffic limits of any kind for a couple of tens of GBytes of sound files to become too much
So availability and performance shouldn't be any problem whatsoever.
Just finished encoding all samples - finally. One of the samples produces 214 kbps with LAME 3.98.2 -V5.7. LAME 3.97 -V5 --vbr-new reaches 198 kbps. iTunes on the other hand set to 112 kbps VBR highest setting reaches only 122 kbps.
Would you count this as a big problem?
YES!
Wait, just one sample is out of whack?
FhG produces also 212 kbps for this sample.
The other samples are also quite high. I will post a bitrate table... Just give me some time, it's after midnight and my brain and body only work at 50%.
LAME 3.97 LAME 3.98.2 iTunes FhG Helix Low Anchor
Sample 01 97 107 115 119 114 128
Sample 02 126 149 113 121 110 128
Sample 03 138 143 119 139 126 128
Sample 04 149 139 116 140 151 128
Sample 05 146 138 121 144 149 128
Sample 06 149 142 120 149 151 128
Sample 07 95 109 118 134 131 128
Sample 08 147 136 113 128 137 128
Sample 09 109 118 115 128 97 128
Sample 10 158 152 120 147 152 128
Sample 11 148 145 117 133 117 128
Sample 12 194 214 122 212 227 128
Sample 13 132 145 124 150 142 128
Sample 14 159 156 124 163 173 128
Seems that iTunes is only very restrictive. Wondering if I should simply set it to 128 kbps instead of 112 kbps.
Have you looked at a histogram?
Anyway, I don't want to repeat the discussion we had about this last year, but I haven't changed my opinion about it. iTunes should be configured at 128, especially since you don't expect it to do well anyhow. Might as well not leave any wiggle room like the previous test did.
Hmm, damn... Maybe they changed the MP3 encoder with iTunes 8. Does anyone have the old iTunes 7? I would install it on a VM and compare the bitrates it produces vs. iTunes 8. If iTunes 7 makes higher bitrates at 112 kbps, I will definitely go with 128 in iTunes 8. Otherwise, is it fair to adjust the encoder settings based on short samples only if otherwise 112 VBR produces an average of 128 kbps when fed with whole tracks?
Regarding the other encoders, if you "ignore" iTunes, I would say that the bitrate fluctuations of the other samples are OK (like for example Sample 02 where Helix allocated 110 kbps and LAME 3.98.2 149 kbps).
Found iTunes 7.7 myself. 112 kbps produces 122 kbps and 128 kbps produces 141 kbps for that specific sample.
The question is: should I choose 128 kbps for all iTunes samples even if 112 kbps is "nearer" to to the target bitrate of this test according to Alex B's findings (when encoding full tracks)?
Hold on a moment.
I am just encoding my various and classical sets with iTunes 8. Probably iTunes has changed again. Maybe the increased VBR bitrates in the latest iTunes 7 builds were not intentional after all. Before that previous change 128 kbps VBR produced a suitable average bitrate. I'll post the results in an hour or so.
P.S. Sorry, that I have not been available for a while. I have had busy times lately. Looks like you had to make the final sample selections without lots of help from others. Are you going to keep the list secret until the test is out?
I don't think bitrates have changed. With iTunes 7.7 (last 7 build) the bitrate is identical to iTunes 8. The problem is that iTunes seems to be very restrictive with the bitrate allocation. A quick look with EncSpot shows that 80% of the frames are encoded with the target bitrate.
You are right, the bitrate behavior has not changed. In fact, the resulting files are bit identical. I checked that with foobar's bitcompare tool. Maybe I misunderstood what you meant.
Here is an excerpt from my previous bitrate test report:
Apparently iTunes has changed radically since my last test. Back then the 128 kbps VBR setting was suitable, but the 7.7 version uses bitrates in a more relaxed way and the 128 kbps VBR setting produces higher bitrates than before. Fortunately the 112 kbps VBR setting appears to be suitable for our test.
Summary
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/8402bb15.png)
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/c4166eec.png)
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/b09a25fc.png)
IMHO, it would not be acceptable to use a setting that produces 161 kbps on average with various rock & pop music.
I don't think bitrates have changed. With iTunes 7.7 (last 7 build) the bitrate is identical to iTunes 8. The problem is that iTunes seems to be very restrictive with the bitrate allocation. A quick look with EncSpot shows that 80% of the frames are encoded with the target bitrate.
In my test the new iTunes 112 VBR produced bitrates from 114 to 168 kbps. AFAICS, its encoder is not very restrictive anymore (except that its minimum bitrate is strictly restricted to the defined setting.)
I wonder if iTunes is doing something funny when the encoded file has a short duration. Have you compared a sample and its complete source track after encoding?
Also, just to make sure we are using the same settings, here is what I have set for 112 kbps VBR:
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/it_mp3settings112vbr.png)
In my test the new iTunes 112 VBR produced bitrates from 114 to 168 kbps. AFAICS, its encoder is not very restrictive anymore (except that its minimum bitrate is strictly restricted to the defined setting.)
What I mean with restrictive is that when you have a look at the MP3 files produced by LAME or FhG with EncSpot, you will see that there is a quite even distribution of frames between let's say 64 kbps and 320 kbps. When you look at iTunes files, you see that about 80% of the frames are coded at 112 or 128 kbps (depending on what bitrate you set iTunes to encode to) and only 20% of the frames have other bitrates. That is why with my samples (most of them if you look at the table), iTunes has the lowest bitrate (you don't find a single sample with a bitrate higher than 130 kbps or lower than 112 kbps when the target bitrate is 112 kbps - the latter having the explanation that the bitrate you set in iTunes is also handled as min bitrate for VBR coding).
I wonder if iTunes is doing something funny when the encoded file has a short duration. Have you compared a sample and its complete source track after encoding?
I will encode one or two files when I am at home in a few hours.
Also, just to make sure we are using the same settings, here is what I have set for 112 kbps VBR:
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/it_mp3settings112vbr.png)
Yes, same settings here.
Now the big question is whether or not to leave everything as-is. In the end, all other coders (except for low anchor that is CBR) allocate high bitrates for the specific sample we're talking about here. That is why I would say that if iTunes thinks it can handle the file fine with 122 kbps and in fact it doesn't, then its psy / VBR model is to blame.
When you look at iTunes files, you see that about 80% of the frames are coded at 112 or 128 kbps (depending on what bitrate you set iTunes to encode to) and only 20% of the frames have other bitrates.
That is odd. I don't see it happening with my test tracks. Only the files with the lowest bitrate are like you said. The complex files show more bitrate variation. Though apparently 112 VBR has a 224 kbps max bitrate limit and 128 VBR 256 kbps max bitrate limit (I hadn't noticed the upper limit before)
Here's what EncSpot says about the three tracks that produced the highest bitrates @ 112 VBR:
[b]Kraftwerk[/b]
Bitrates:
----------------------------------------------------
112 |||||||||||||||||||||||||||||||||||||| 31.2%
128 |||||| 5.4%
160 |||||||||||||||||||||| 18.4%
192 |||||||||||||||| 12.9%
224 |||||||||||||||||||||||||||||||||||||||| 32.1%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 168
Mode : joint stereo
Frequency : 44100 Hz
Frames : 10045
ID3v2 Size : 2146
First Frame Pos : 2146
Length : 00:04:22
Max. Reservoir : 167
Av. Reservoir : 72
Emphasis : none
Scalefac : 9.9%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.1 ]--[ [url=http://www.guerillasoft.com]http://www.guerillasoft.com[/url] ]--
[b]Faithless[/b]
Bitrates:
----------------------------------------------------
112 |||||||||||||||||||||||||||||||||||||||| 25.7%
128 |||||||||||||||||| 11.6%
160 |||||||||||||||||||||||||||||||||||| 23.6%
192 ||||||||||||||||||||||||||||||| 20.5%
224 |||||||||||||||||||||||||||| 18.6%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 162
Mode : joint stereo
Frequency : 44100 Hz
Frames : 8588
ID3v2 Size : 2140
First Frame Pos : 2140
Length : 00:03:44
Max. Reservoir : 246
Av. Reservoir : 76
Emphasis : none
Scalefac : 5.7%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.1 ]--[ [url=http://www.guerillasoft.com]http://www.guerillasoft.com[/url] ]--
[b]Yello[/b]
Bitrates:
----------------------------------------------------
112 |||||||||||||||||||||||||||||||||||||||| 34.1%
128 |||||||||| 8.7%
160 ||||||||||||||||||| 17.0%
192 |||||||||||||||||| 15.4%
224 ||||||||||||||||||||||||||||| 24.8%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 161
Mode : joint stereo
Frequency : 44100 Hz
Frames : 7220
ID3v2 Size : 2131
First Frame Pos : 2131
Length : 00:03:08
Max. Reservoir : 345
Av. Reservoir : 76
Emphasis : none
Scalefac : 4.7%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.1 ]--[ [url=http://www.guerillasoft.com]http://www.guerillasoft.com[/url] ]--
...and before anyone complains: The files are encoded with iTunes. EncSpot's guess is incorrect.
Just did a quick test with some samples and there is no difference between full tracks and samples. If the sample had let's say 88% 112 kbps frames, the full track also had around 80% 112 kbps frames. Wondering why...
I created samples of the three high bitrate tracks in my previous post:
[b]Kraftwerk (sample)[/b]
Bitrates:
----------------------------------------------------
112 ||||||||||||||||||||||||||| 25.2%
128 ||||| 5.3%
160 |||||||||||||||||||| 18.9%
192 ||||||||||||||| 14.0%
224 |||||||||||||||||||||||||||||||||||||||| 36.5%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 174
Mode : joint stereo
Frequency : 44100 Hz
Frames : 1142
ID3v2 Size : 2149
First Frame Pos : 2149
Length : 00:00:29
Max. Reservoir : 204
Av. Reservoir : 78
Emphasis : none
Scalefac : 12.1%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.1 ]--[ [url=http://www.guerillasoft.com]http://www.guerillasoft.com[/url] ]--
[b]Faithless (sample)[/b]
Bitrates:
----------------------------------------------------
112 |||||||||||||||||||||||||||||||||||||||| 25.1%
128 |||||||||||||||||||||| 13.9%
160 ||||||||||||||||||||||||||||||||||||| 23.4%
192 ||||||||||||||||||||||||||||||||||||| 23.3%
224 |||||||||||||||||||||| 14.2%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 160
Mode : joint stereo
Frequency : 44100 Hz
Frames : 1143
ID3v2 Size : 2143
First Frame Pos : 2143
Length : 00:00:29
Max. Reservoir : 186
Av. Reservoir : 77
Emphasis : none
Scalefac : 5.1%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.1 ]--[ [url=http://www.guerillasoft.com]http://www.guerillasoft.com[/url] ]--
[b]Yello (sample)[/b]
Bitrates:
----------------------------------------------------
112 |||||||||||||||||||||||||||||||||||||||| 42.1%
128 ||||||||||||| 14.4%
160 ||||||||||||||||||||||| 25.0%
192 |||||||||| 11.3%
224 |||||| 7.2%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 143
Mode : joint stereo
Frequency : 44100 Hz
Frames : 1146
ID3v2 Size : 2134
First Frame Pos : 2134
Length : 00:00:29
Max. Reservoir : 186
Av. Reservoir : 71
Emphasis : none
Scalefac : 5.6%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.1 ]--[ [url=http://www.guerillasoft.com]http://www.guerillasoft.com[/url] ]--
I think they are fine when I compare about the same passages in the complete encoded tracks.
The big question is why your "hard to encode" samples do not produce higher bitrates if mine do.
You can download my samples from here: http://rapidshare.com/files/150063026/set_...ate_samples.rar (http://rapidshare.com/files/150063026/set_various_3_high_birate_samples.rar) (10.8 MB)
Could you please encode fatboy (http://www.hydrogenaudio.org/forums/index.php?act=Attach&type=post&id=501) with iTunes and tell me what the outcome is? I really hope it is 122 kbps - otherwise, there is something really wrong with my PC.
If that produces a low bitrate while other tracks you tested produce a high bitrate it means that iTunes' VBR implementation thinks that the 122 kbps should be sufficient and I see no problem then. If the sample sounds worse, than LAME which uses 200 kbps, then we can say hard luck because iTunes could have increased the bitrate if it felt the need for it.
Done.
[b]Fatboy_30sec[/b]
Bitrates:
----------------------------------------------------
112 ||||||||||| 17.8%
128 || 4.4%
160 ||||| 8.5%
192 ||||| 8.4%
224 |||||||||||||||||||||||||||||||||||||||| 60.9%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 191
Mode : joint stereo
Frequency : 44100 Hz
Frames : 1122
ID3v2 Size : 2123
First Frame Pos : 2123
Length : 00:00:29
Max. Reservoir : 428
Av. Reservoir : 65
Emphasis : none
Scalefac : 5.6%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.1 ]--[ [url=http://www.guerillasoft.com]http://www.guerillasoft.com[/url] ]--
The encoded file is here: http://rapidshare.com/files/150073489/fatboy_30sec.rar (http://rapidshare.com/files/150073489/fatboy_30sec.rar)
You must feel like or maybe ...
Ever called Apple's support?
WTF?! This is impossible... I decoded that file with fb2k to WAV. Then I added it to the iTunes library, configured the MP3 encoder, right clicked on the file and told it to convert the file to MP3. Then I took both fb2k and EncSpot 2.2 (that castrated Pro version) and checked the bitrate: 122 kbps.
Edit: The really funny thing is that the test with iTunes 7.7 which I performed today was with a different machine.
Naturally I could help if your iTunes really has a problem that can't be quickly solved. It would take only a few minutes to encode the sample set.
EDIT
What is your iTunes build version? I have 8.0.0.35.
1) I took fatboy sample (only 5 seconds long) from http://lame.sourceforge.net/download/samples/fatboy.wv; (http://lame.sourceforge.net/download/samples/fatboy.wv;)
2) encoded it with Itunes (112kbps VBR);
3) resulting file is 125kbps.
So now we have 122 kbps, 125 kbps and 191 kbps - awesome.
OK, you have the short version. Could you test with the 30 seconds version?
What is your iTunes build version? I have 8.0.0.35.
8.0.0.35 as well. I have a German version, BTW, but that shouldn't matter.
I take you use Windows not Mac OS? I'm on a German Vista Home Premium SP1. Wondering if my encoder is "broken" or yours since mine seems to always produce files that are very close to the target bitrate specified in the settings, while your is fluctuating a lot more. Also, 128 kbps produces nearly 128 kbps over my collection of files while your encoder reaches 160 kbps in average.
I have English Windows XP Pro.
I tested the shorter fatboy:
[b]Fatboy_5sec[/b]
Bitrates:
----------------------------------------------------
112 |||| 9.7%
128 2.1%
160 | 2.6%
192 2.1%
224 |||||||||||||||||||||||||||||||||||||||| 83.1%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 208
Mode : joint stereo
Frequency : 44100 Hz
Frames : 195
ID3v2 Size : 2122
First Frame Pos : 2122
Length : 00:00:05.0
Max. Reservoir : 345
Av. Reservoir : 59
Emphasis : none
Scalefac : 4.1%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.1 ]--[ [url=http://www.guerillasoft.com]http://www.guerillasoft.com[/url] ]--
I checked also the 30s sample by calculating the bitrate from the file size and duration (I removed the ID3 tags with Mp3tag before checking the file size):
(701348 bytes x 8 bits per byte / 29.245 sec) x 1000 = 191.854 kbps
This is really weird. I will encode at work tomorrow where I have an XP and Vista PC. Maybe the OS is to blame.
BTW, were iTunes 7 MP3s recognized as being encoded by FhG in EncSpot? I hope Apple didn't throw away their encoder in favor of FhG.
Edit: And BTW, you are testing with 112 kbps, correct?
What is your iTunes build version? I have 8.0.0.35.
Wondering if my encoder is "broken" or yours since mine seems to always produce files that are very close to the target bitrate specified in the settings, while your is fluctuating a lot more. Also, 128 kbps produces nearly 128 kbps over my collection of files while your encoder reaches 160 kbps in average.
Please encode the three samples I uploaded to Rapishare so that we can compare the results.
EDIT
I think we need help from others. It would help if a few other HA members could encode these samples. The correcty settings can be checked from the screenshot I posted earlier.
I have now only one PC with iTunes (a laptop). My HTPC is "under construction" and it may take a few days before it is usable again. Otherwise I could verify the results on a different PC and software setup.
Faithless:
Bitrates:
----------------------------------------------------
112 |||||||||||||||||||||||||||||||||||||||| 77.1%
128 ||||| 11.5%
160 |||| 8.9%
192 1.8%
224 0.5%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 120
Mode : joint stereo
Frequency : 44100 Hz
Frames : 1143
ID3v2 Size : 2139
First Frame Pos : 2139
Length : 00:00:29
Max. Reservoir : 115
Av. Reservoir : 56
Emphasis : none
Scalefac : 3.7%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.2 ]--[ http://www.guerillasoft.com ]--
Kraftwerk:
Bitrates:
----------------------------------------------------
112 |||||||||||||||||||||||||||||||||||||||| 75.6%
128 | 3.4%
160 |||||||||| 19.7%
192 0.4%
224 0.8%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 123
Mode : joint stereo
Frequency : 44100 Hz
Frames : 1142
ID3v2 Size : 2145
First Frame Pos : 2145
Length : 00:00:29
Max. Reservoir : 156
Av. Reservoir : 48
Emphasis : none
Scalefac : 2.8%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.2 ]--[ http://www.guerillasoft.com ]--
Yello:
Bitrates:
----------------------------------------------------
112 |||||||||||||||||||||||||||||||||||||||| 76.6%
128 |||| 9.0%
160 ||||| 11.4%
192 1.8%
224 1.0%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 121
Mode : joint stereo
Frequency : 44100 Hz
Frames : 1146
ID3v2 Size : 2130
First Frame Pos : 2130
Length : 00:00:29
Max. Reservoir : 164
Av. Reservoir : 58
Emphasis : none
Scalefac : 3.4%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.2 ]--[ http://www.guerillasoft.com ]--
BTW, were iTunes 7 MP3s recognized as being encoded by FhG in EncSpot? I hope Apple didn't throw away their encoder in favor of FhG.
Encspot has always done that.
Edit: And BTW, you are testing with 112 kbps, correct?
I have double and triple-checked the settings.
Also, my iTunes v.7.7.1.11 VBR112 and VBR128 results from August are identical with the new v.8 results.
Though I used this same PC back then. (I didn't want to install the bloated iTunes 7.7 on my other PC.)
This PC has a quite new OS installation. I installed a new hard drive and a vanilla XP Pro SP2 in April. The first iTunes version I installed was 7.6.2.9 (I have the installer still on the HD).
Seriously, this is very strange. I hope other encoders (LAME, FhG, Xing) aren't affected by this.
I will test tomorrow morning under Vista and XP.
This is a wild guess, but could it be possible that Apple has not changed the encoder after all? What if they have just loosened some hidden parameters, which might be stored in the registry. If they did a crappy job maybe a newer installer does not correctly overwrite the old parameters. It might be handling them as user settings which are carried over.
Have you had old iTunes versions installed on your OS? When I tested 7.4.2.9 about a year ago the bitrates were a lot lower.
I had 7.4 as first version IIRC, then 7.7 and then updated to 8 when it came out a few days ago.
I tested fatboy_30sec and I got the same result as Sebastian (121kbps). Here is a result of bitrate fluctuation analysis:
iTunes: (larger image (http://i410.photobucket.com/albums/pp188/naoyan2008/itunes.png))
(http://i410.photobucket.com/albums/pp188/naoyan2008/itunes_s.png)
iTunes(Alex): (larger image (http://i410.photobucket.com/albums/pp188/naoyan2008/alex.png))
(http://i410.photobucket.com/albums/pp188/naoyan2008/alex_s.png)
LAME: (larger image (http://i410.photobucket.com/albums/pp188/naoyan2008/lame.png))
(http://i410.photobucket.com/albums/pp188/naoyan2008/lame_s.png)
The result of fatboy_30sec.mp3 by Alex seems to be close to the result by true VBR encoder like LAME. I'm very interested in how Alex obtained this mp3 file with iTunes...
thanks for help
Awesome! I just installed iTunes 8 on a fresh Vista Home Premium SP 1 and a fresh Windows XP Professional SP 3. Results for fatboy (30 seconds):
Vista: 122 kbps
XP: 191 kbps
So, I guess I will have to contact Apple support...
That's really strange, but at least you found something. I was beginning to suspect that my SW configuration has something terribly wrong.
I wonder what kind of file a Mac iTunes would encode. Could anyone test this on a Mac and post a sample with details about the CPU and OS/SW versions?
EDIT
I converted the Fatboy 30 sec FLAC sample to Apple Lossless. It is available here:
http://rapidshare.com/files/150190105/fatboy_30sec_alac.zip (http://rapidshare.com/files/150190105/fatboy_30sec_alac.zip)
I've tested fatboy_30sec on various environments with my Mac. Here are results:
OSX 10.5, Core 2 Duo, iTunes 8.0: 121 kbps
OSX 10.4, Core 2 Duo, iTunes 7.7.1: 121 kbps
OSX 10.4, PPC G5, iTunes 8.0: 122 kbps
OSX 10.5, PPC G4, iTunes 7.7.1: 192 kbps
OSX 10.5, PPC G4, iTunes 7.6: 192 kbps
OSX 10.4, PPC G4, iTunes 7.2: 192 kbps
Strange result. Different encoding algorithms are used according to CPU???
-
A friend visited our office today. He had an old iTunes version installed on his Celeron M / XP Home SP2 laptop.
I encoded the fatboy sample with his iTunes 6.0.5.20. The version is well over two years old.
[b]Fatboy_30sec_iTunes_v.6.0.5.20[/b]
Bitrates:
----------------------------------------------------
112 ||||||||||| 17.1%
128 || 3.7%
160 ||||| 8.6%
192 ||||| 8.6%
224 |||||||||||||||||||||||||||||||||||||||| 61.9%
----------------------------------------------------
Type : mpeg 1 layer III
Bitrate : 192
Mode : joint stereo
Frequency : 44100 Hz
Frames : 1122
ID3v2 Size : 2102
First Frame Pos : 2102
Length : 00:00:29
Max. Reservoir : 428
Av. Reservoir : 68
Emphasis : none
Scalefac : 4.9%
Bad Last Frame : no
Encoder : FhG (fastenc)
Lame Header : No
--[ EncSpot 2.1 ]--[ [url=http://www.guerillasoft.com]http://www.guerillasoft.com[/url] ]--
The file is not identical with my iTunes 8 encoded file, but the bitrate behavior is similar.
Also this is a surprising result because I have believed that the older iTunes versions always created very restricted VBR files.
WinXP SP2 clean install, Fatboy 5-sec:
Itunes 6.0.4.2 (QT 7.1): 207 kbps
Itunes 7.7.0.43 (QT 7.50.61.0): 207 kbps
Curiouser and curiouser...
WinXP SP2 clean install, Fatboy 5-sec:
Itunes 6.0.4.2 (QT 7.1): 207 kbps
Itunes 7.7.0.43 (QT 7.50.61.0): 207 kbps
I got about the same bitrate when I encoded the short sample with iTunes 8. EncSpot displayed 208 kbps.
EDIT
What OS and iTunes version did you use when you got 125 kbps?
EncSpot displayed 208 kbps.
Yes. But iTunes itself and foobar2000 report 207kbps.
What OS and iTunes version did you use when you got 125 kbps?
WinXP SP3 with a lot of other stuff, Itunes 7.7.0.43.
BTW: WinXP SP2 clean install, then upgrade to SP3, then install Itunes 7.7.0.43 => 207 kbps. I installed also WMP11 and Aug'2008 DirectX but it didn't change the bitrate.
1 kbps more or less is not important, I'd say (could be caused by rounding). I find it very, very strange that your previous test was also carried out under XP.
What do you think about this:
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/it8qsettings.png)
I used the seven different quality settings (iTunes 8 / XP). The screenshot is from EncSpot.
My previous experience about a year ago was different:
... Since you mentioned the iTunes' quality settings we could start with iTunes. I tested some additional iTunes options:
(http://i224.photobucket.com/albums/dd212/AB2K/itunesbitrates.png)
Surprisingly the quality options have almost no effect to the encoding speeds. The very small encoding time differences may be caused by other things as well (I did only one pass). I wonder if these settings do anything, even though there was a very small bitrate increase.
The Smart Encoding Adjustments and Filter Frequencies Below 10 Hz options seem to slightly affect the speed so perhaps they actually do something.
I'd guess that for some reason iTunes does not always use the set options correctly.
It's unfortunate that no one said anything when I posted my iTunes 7.4.0.28 results a year ago. Though probably not many of us actively use iTunes for MP3 encoding or have experimented with its MP3 settings.
If the Apple developers would have followed what happened here they could have reacted.
I GET IT!
Dualcore processor => 125 kbps.
Dualcore processor with 1 core disabled => 207 kbps.
I just added "/numproc=1" switch to boot.ini and rebooted. (Setting affinity through Task manager won't help).
But how come I had different results while testing on the same machine - an FSC Amilo Xi 2550 notebook with a dual core CPU with XP and Vista? I am really confused.
Edit: Found the problem. XP was really set to /numproc=1 /noguiboot.
Tested the new 8.0.1 version and I get the same results on my Vista machine.
Oh, is there a new version of iTunes?
Have you already tried the effect of the quality settings on XP and Vista? It would be useful to know if those settings are ineffective when the encoder appears to be creating only ABR like MP3 files.
I only have a Vista machine at home and today is Unification Day in Germany so I am not at work. I can try on a VM with Vista and XP, though (which I am going to do now).
Edit: And yeah, I turned on my PC today and the Apple Updater popped up.
I only have a Vista machine at home and today is Unification Day in Germany so I am not at work. I can try on a VM with Vista and XP, though (which I am going to do now).
I'd guess the settings have a clear effect whenever the files appear to be real VBR, like in my case.
I can't test the other case because none of the PCs I have access at the moment produce those ABR like results.
BTW, do you have a multicore machine or a singlecore one?
Single.
My "HTPC" would have HyperThreading, but it is having strange problems and I have disassembled it for testing the individual components. I may need to replace the motherboard. It is the machine that produced the lower iTunes VBR bitrates a year ago (on XP).
It seems iTunes encoder has some problem on multi core machines. How does it fit with nao's result on a MAC, is the G5 a multi core machine too?
I don't have iTunes, does it feature some switch/check-box to en-/dis-able multi core usage?
So, I just did some tests with fatboy on my dualcore machine running Vista. The lowest VBR quality setting produces 118 kbps, the mid one 120 kbps and the highest one 122 kbps. The two options Smart Encoding and Filter 10 Hz also have an impact, but a very small one (bitrate difference was only 1 kbps).
I am afraid iTunes does not have an option to enable or disable multicore usage.
While at it - Alex B, could you please encode fatboy to 128 kbps VBR AAC using iTunes? Wondering if the AAC encoder has the same problem. Please tell me what bitrate fb2k reports.
How does it fit with nao's result on a MAC, is the G5 a multi core machine too?
My G5 machine has a dual-core CPU. Only G4 machine has a single-core CPU in my test. So, probably the insight that iTunes mp3 encoder has a problem on multi-CPU environment is correct. Unfortunately, there is no option in iTunes to turn-off multithreading.
OK, that convinced me. I filed a bug report to Apple. Hopefully they will reply.
BTW, regarding the AAC. fb2k reports 131 kbps and iTunes 128 kbps.
I will have to check why I got different results at work.
So, I just did some tests with fatboy on my dualcore machine running Vista. The lowest VBR quality setting produces 118 kbps, the mid one 120 kbps and the highest one 122 kbps. The two options Smart Encoding and Filter 10 Hz also have an impact, but a very small one (bitrate difference was only 1 kbps).
So the quality settings have some very slight effect. Apparently the selected setting is memorized and used, but it does not work properly (I suppose we can assume that the setting is designed to be able to adjust the amount of bitrate fluctuation.)
While at it - Alex B, could you please encode fatboy to 128 kbps VBR AAC using iTunes? Wondering if the AAC encoder has the same problem. Please tell me what bitrate fb2k reports.
From foobar's file properties window:
File Name : fatboy_30sec.m4a
File Size : 478KB (489 618 bytes)
Duration : 0:29.207 (1288011 samples)
Sample Rate : 44100 Hz
Channels : 2
Bitrate : 131 kbps
Codec : AAC
Codec Profile : AAC LC
Encoding : lossy
Tool : iTunes 8.0.0.35, QuickTime 7.5.5
iTunes AAC encoder does not support multi-threaded encoding, so probably it is not affected by the problem.
Sebastian, please upload the 122 kbps "VBR 112" sample. I'd like to compare the files in a listening test.
Actually a "dual core" VBR 128 encoded sample would be better. It would have been the selected setting if the bitrates were tested only on machines that trigger this problem.
Here it is: http://www.hydrogenaudio.org/forums/index....showtopic=66253 (http://www.hydrogenaudio.org/forums/index.php?showtopic=66253)
Did you notice the following text just below encoder settings:
Details
56 kbps (mono)/112 kbps (stereo), VBR (Highest quality), joint stereo, optimized for MMX/SSE, using MP.
for dualcore and the same text without "using MP" - for singlecore?
I encoded one album (~1 hour long) with MP3 CBR 160 kbps:
1 core: 3m 20s (= 200s)
2 cores: 1m 57s (= 117s)
The files have different comment tag, but identical audio content. With VBR, this isn't so:
VBR 112 kbps:
1 core: 136 kbps 3m 37s (= 217s)
2 cores: 118 kbps 2m 12s (= 132s)
So multiprocessing support is really good but gives incorrect result with VBR enabled.
A listening test didn't bring up any further surprises.
ABC/HR for Java, Version 0.5b, 03 October 2008
Testname: fatboy - itunes
Tester: Alex B
1R = C:\bitratetest\dualcore_vbr128.wav
2L = C:\bitratetest\singlecore_vbr112.wav
---------------------------------------
General Comments:
---------------------------------------
1R File: C:\bitratetest\dualcore_vbr128.wav
1R Rating: 1.4
1R Comment:
---------------------------------------
2L File: C:\bitratetest\singlecore_vbr112.wav
2L Rating: 2.0
2L Comment:
---------------------------------------
ABX Results:
C:\bitratetest\dualcore_vbr128.wav vs C:\bitratetest\singlecore_vbr112.wav
8 out of 8, pval = 0.0030
---- Detailed ABX results ----
C:\bitratetest\dualcore_vbr128.wav vs C:\bitratetest\singlecore_vbr112.wav
Playback Range: 03.623 to 05.910
6:17:58 PM p 1/1 pval = 0.5
6:18:15 PM p 2/2 pval = 0.25
6:18:37 PM p 3/3 pval = 0.125
6:18:59 PM p 4/4 pval = 0.062
6:19:37 PM p 5/5 pval = 0.031
6:19:55 PM p 6/6 pval = 0.015
6:20:44 PM p 7/7 pval = 0.0070
6:21:01 PM p 8/8 pval = 0.0030
As expected, both samples were seriously distorted. I did the ABX test only for checking that I can reliably distinguish the MP3 samples from each other.
Both were bad, but the single core VBR112 version was clearly better.
Does anybody else think we should consider dropping iTunes as a contender as a result of all this? In my opinion, we shouldn't have buggy encoders in the test.
Does anybody else think we should consider dropping iTunes as a contender as a result of all this? In my opinion, we shouldn't have buggy encoders in the test.
But Apple can fix this issue easily just by disabling multiprocessor support when VBR mode is on .
Does anybody else think we should consider dropping iTunes as a contender as a result of all this? In my opinion, we shouldn't have buggy encoders in the test.
That's exactly the reason I wanted to compare the two files. I wanted to know if the "single core" version produces expected (i.e. better) quality when the encoded file has a higher bitrate and thus the single core version could be considered as normal.
I doubt Apple will fix anything quickly so we have only two practical options: either we can drop iTunes completely or we can test samples that are created with a single core machine. We can test them now and Sebastian can decide later what to do. In the past Nero's AAC encoder has been disqualified from the final results because it wasn't working correctly. However, its results were published separately because that was considered to be fair for the high quality Nero encoder and because the fixes Nero did after the test wouldn't have changed the test results considerably.
EDIT
It might be good to test a few other samples before making the decision. The fatboy sample is a real killer so maybe a couple of easier samples should be tested.
OK, that convinced me. I filed a bug report to Apple. Hopefully they will reply.
BTW, regarding the AAC. fb2k reports 131 kbps and iTunes 128 kbps.
Yes, this is common practice for iTunes and has been for sometime now with both their mp3 and AAC encoders. It wasn't until recently where Apple changed what iTunes would display for iTunes encoded VBR AAC files. It would display the overall average bitrate and that only confused people. They would encode a 128kbps VBR AAC files and iTunes would show a bitrate of 130kbps+. Then Apple changed things again so that iTunes would display the setting used. For a 128kbps VBR AAC file, it would display 128kbps (VBR) for the bitrate thus confusing less people (I guess). It is a shame as I would rather have iTunes display the overall average bitrate of songs (it does this for Lame encoded mp3 files) and then put (VBR) in parenthesis (again, it does this for Lame and FhG encoded mp3 files).
It is true that Apple can fix display/encoding issues with iTunes rather quickly but they don't. Apple just released a new version of iTunes and it doesn't look like they are going to update it anytime soon (at least the mp3 encoding part). The iTunes mp3 encoder receives updates even less frequently than either iTunes or its AAC encoder. So don't hold your breath for an mp3 encoder update even if it is a reported bug. Everyone on hydrogenaudio could report this bug and it still wouldn't speed Apple up. Alex B is right though, some more tests should be conducted before ruling out iTunes.
I also agree that other samples should be tested for differences. It could provide some sense of the bug's scope. But I'm still sceptical about including iTunes, even after just this one result. IIRC, the bug in the old NeroAAC contender was only discovered after the test had begun. So we had the results of that contender already. Since we know of this bug before the test, it doesn't make sense to me to ask people to listen and rate it.
I installed several iTunes versions and tested them with fatboy (5-sec long) sample. Encoding settings are the same: 112kbps (VBR, max quality).
1 core 2 cores
iTunes 4.9.0.17 135 137
iTunes 5.0.1.4 194 119
iTunes 6.0.4.2 194 119
iTunes 7.0.2.16 207 125
iTunes 7.7.0.43 207 126
iTunes 5.0.1.4 was released at september'2005. No-one noticed this bug before?
I guess that is because a small community uses the iTunes mp3 encoder, all others look towards Lame and the various programs using the FhG encoder (Windows Media Player, the Zune PC software, etc.).
Thank you for your findings lvqcl.
Do you guys think I should wait x days for Apple to respond to my mail or should I continue with the test right away? I have a singlecore Celeron notebook, so it wouldn't be a problem to encode the samples and start the test.
It's one thing that you have a single core machine to make the encodes, but many people don't have a single core computer. So, in the event that iTunes makes a strong showing in the results, and therefore people are more comfortable using it, they won't get the same performance from *their* encodes. The iTunes contender samples would not be representative of real world use.
Of course, it may be that Apple fixes the problem, but I wouldn't hold my breath.
Of course, I would include a big "WARNING" or something like that on the listening test page.
I still think it should be disqualified. These tests are time consuming and require a lot of unbroken concentration. I was planning on participating but I don't think I will if this broken encoder is included.
Seriously, what is the point in having a rank on a contender when its samples can only be produced on single core machines?
I could also do the same thing I did with the Nero AAC encoder after Guru noticed the bug with the bitrate allocation, although that was a slightly different story.
It would be good if Apple would acknowledge that the encoder has a problem with multi-core machines and that the encoder behaves as intended on a single-core machine. Then we could assume that Apple will eventually fix the problem and the test results would be usable for all iTunes users.
BTW, has anyone tried what happens when the machine has more than two cores?
As Sebastian have said, in general it would be very interesting to have up-to-date test results of the iTunes MP3 encoder. It is probably the dominant MP3 encoder on Macs and many professional content creators work on Macs.
EDIT
After finding out that iTunes isn't faster than LAME I was against testing it because I didn't see the usefulness of the results in a wider perspective beyond my personal needs, but I have changed my mind.
I could test on a 8 core machine... (4 real, 4 synthetical cores)
Just did a quick test and encoded at 112 and 128 kbps. The results are just like with two cores. 112 kbps file comes out at 122 kbps, 128 kbps file comes out at 141 kbps.
I am trying to get in touch with someone from Apple directly. If I don't get any feedback soon or if the fix is supposed to come in a version that will be released too late (let's say in a month or so), I guess the best thing I can do is to 1. display a big warning on the presentation page and 2. display the iTunes ranking detached from the rest like I did for Nero. As Alex B pointed out, iTunes is a really popular MP3 encoder at least on Macs and I would like to avoid disqualifying it entirely.
Could the mysterious Apple developer mentioned in the following quote help contacting the right person(s) at Apple?
It's one person (that I know of) that is member of this forum. But I suspect he's not alone working on the AAC encoder...
... I bullied him already :B
... He said he wouldn't mind including it in the encoder, but it doesn't depend only on him. It also depends on the QuickTime division, the iTunes division, the iPod division... For that reason, it'll probably only happen when a decision comes from above.
I have no idea who this person could be.
I have no idea who this person could be.
I do...
You have to be a HA oldtimer to know that But he posted pretty publically on this forum in the past.
Does anyone have an idea of what Apple could have done wrong to cause an unlikely problem like this?
Could Apple actually have included two separate MP3 encoder components and made a mistake when configuring the internal options?
If we would need to "reverse engineer" the situation, what would a developer use for intentionally making the encoder behave differently on different machines?
Some digging: http://developer.apple.com/quicktime/whatsnew.htm (http://developer.apple.com/quicktime/whatsnew.htm)
- "Multiprocessor (MP) Support" was added to QuickTime version 5
- "VBR Sound Compression Support" was added in QuickTime version 6
It's bad, that there seems to be no way to control the MP feature in QuickTime.
Alex B
I think the answer is quite simple. If you want to use full power of a multicore CPU, then you need to split the computing into more threads. This means you have to significantly change your algorithm. I heard before that you will get different results with multithreaded versions of libavcodec, x264 or XviD codecs.
Multicore CPU is IMHO a very bad idea and direction. But CPU manufacturers had probably no other cheap solution on how to further increase power of their processors. If you want to use all cores at the same time in your encoder, you have to write it differently (much more complicated) compared to single thread encoder.
I hope this explains why you get different results on singlecore and multicore CPUs.
Multicore CPU is IMHO a very bad idea and direction. But CPU manufacturers had probably no other cheap solution on how to further increase power of their processors. If you want to use all cores at the same time in your encoder, you have to write it differently (much more complicated) compared to single thread encoder.
Multicore is not a bad idea. People normally have more than one program running. So each core can take care of each program.
Does anyone have an idea of what Apple could have done wrong to cause an unlikely problem like this?
Well, somewhere the split information processed on each core must be merged. You can have an issue there.
By the way, are CBR results binary identical on single and multi core machines? Do they have a problem there too?
[...]
I encoded one album (~1 hour long) with MP3 CBR 160 kbps:
1 core: 3m 20s (= 200s)
2 cores: 1m 57s (= 117s)
The files have different comment tag, but identical audio content. With VBR, this isn't so:
[...]
Just wanted to let you know that someone from Apple is investigating this.
Some digging: http://developer.apple.com/quicktime/whatsnew.htm (http://developer.apple.com/quicktime/whatsnew.htm)
- "Multiprocessor (MP) Support" was added to QuickTime version 5
- "VBR Sound Compression Support" was added in QuickTime version 6
It's bad, that there seems to be no way to control the MP feature in QuickTime.
Let me quote myself:
1 core 2 cores
iTunes 4.9.0.17 135 137
iTunes 5.0.1.4 194 119
iTunes 4.9.0.17 uses QuickTime 6.5.2 and iTunes 5.0.1.4 uses QT 7.0.2. So I doubt that this particular bug was introduced in QT 5 or 6.
I also tried to install iTunes 4.9 and then upgrade QT to 7.5. But this didn't affect MP3 coder at all: mp3 files created before and after upgrade were bit-identical. (AAC encoding was changed, however: when encoding to AAC, iTunes crashes )
Probably iTunes mp3 encoder doesn't depend on QuickTime, because QT API has no interface to produce mp3.
Good news! I just received a mail from one of the persons in charge and the bug was indeed identified and should be fixed in the next iTunes version. Thanks to the Apple developers for reacting so quickly and also thanks to the HA members who supported in identifying the problem.
Edit: BTW, it seems that in some cases, CBR encoding can be affected by this bug as well.
Did that person say anything about which behavior is correct? I wouldn't be surprised if they would consider the single-core version broken and limit the amount of the bitrate variation.
BTW, I'd like to see bitrates of all 14 test samples when they encoded with iTunes on single-core CPU (just out of curiosity)
Did that person say anything about which behavior is correct? I wouldn't be surprised if they would consider the single-core version broken and limit the amount of the bitrate variation.
Single core behavior is correct.
While we wait the test to begin it might be useful to revisit the comments that were posted in the 64kbps multiformat test's announce thread: http://www.hydrogenaudio.org/forums/index....showtopic=56397 (http://www.hydrogenaudio.org/forums/index.php?showtopic=56397)
In that thread I made some suggestions about how the test presentation and instructions could be developed further: http://www.hydrogenaudio.org/forums/index....st&p=509971 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=56397&view=findpost&p=509971)
EDIT
In addition, the comments in the post-test thread make a good read:
http://www.hydrogenaudio.org/forums/index....showtopic=56851 (http://www.hydrogenaudio.org/forums/index.php?showtopic=56851)
I am afraid that a major "overhaul" of the listening test page and the instruction is not going to be ready in time for this test. I am planning to change the listening test pages and make them somewhat more attractive, but 1. I am not really talented in designing so I have to see who can help me and 2. this is going to take some time so it will be only available for the next teast at earliest. I hope to have less ranked references in this test. Most of the people who perform listening tests are HA readers anyways so I was actually assuming that they know perform a "correct" test.
I didn't want to put more pressure on you.
I merely wanted to keep this discussion alive and posted the links for newcomers so that they can get a better impression of the complete testing process.
Small status update: I am trying to get in touch with schnofler for some changes in ABC/HR. Everything is pretty much done except for the ABC/HR configuration files and is ready to be uploaded.
Here is the updated bitrate table that also contains the encoding speeds (except for iTunes that has to be tested once the fixed version is out).
I didn't want to redo all tests on the single-core machine and comparing iTunes' single-core results to the dual-core results of the other encoders is not fair (at least if any of the other encoders makes use of multi-threading).
Sample 1 Sample 2 Sample 3 Sample 4 Sample 5 Sample 6 Sample 7 Sample 8 Sample 9 Sample 10 Sample 11 Sample 12 Sample 13 Sample 14 Max Min Avg Speed 1)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
LAME 3.97 | 97 126 138 149 146 150 95 147 109 158 149 194 133 159 194 95 139 18x
LAME 3.98.2 | 107 149 143 140 138 143 109 136 118 152 145 214 146 156 214 107 143 27x
FhG | 119 121 139 140 144 149 134 128 129 147 134 212 151 163 212 119 144 45x
iTunes | 118 117 139 145 133 125 141 120 126 148 151 192 159 158 192 117 141 N/A 2)
Helix | 114 110 126 151 149 151 131 138 97 152 117 228 143 173 228 97 141 90x
Low Anchor | 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 1,63x
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Max | 119 149 143 151 149 151 141 147 129 158 151 228 159 173
Min | 97 110 126 140 133 125 95 120 97 147 117 192 133 156
Avg | 111 125 137 145 142 144 122 134 116 151 139 208 146 162 142
Max - Min | 22 39 17 11 16 26 46 27 32 11 34 36 26 17
1) These results are only an approximate value and were obtained on a dual-core Intel E6550 with 2 GB RAM running on Windows Vista SP1 32-bit
2) Encoding speed will be measured with iTunes once the bug-fixed version was released. The buggy version encoded at 18x but produced lower bitrates.
Edit 1: IMPORTANT: Please notice that the low anchor wasn't taken into consideration for building the min, max, delta and average bitrates.
Edit 2: Sorry for breaking the layout - I used a codebox so I don't exactly know what is going wrong here.
Sorry to flutter the dovecotes perhaps, but wouldn't it be fairer to rename the test into 140k then, in stead of 128k?
Sorry to flutter the dovecotes perhaps, but wouldn't it be fairer to rename the test into 140k then, in stead of 128k?
Short test samples do not represent the average bitrate behavior of complete tracks of various genres in a media library. In my tests the overall average was 127.8 kbps. I think it is close enough.
Since this thread is huge and things can easily get lost in it I'll replicate my test results here (once again):
I finally had time to continue my bitrate tests with classsical music as I promised earlier in this thread (a thing called summer got in the way...)
After browsing through my lossless classical library I picked 25 "reference" tracks that should be quite representative. I avoided the extremely low and high lossless bitrates and tried to select tracks that have quite varied qualities.
Apparently iTunes has changed radically since my last test. Back then the 128 kbps VBR setting was suitable, but the 7.7 version uses bitrates in a more relaxed way and the 128 kbps VBR setting produces higher bitrates than before. Fortunately the 112 kbps VBR setting appears to be suitable for our test.
EDIT: As explained earlier in this thread, the fundamental difference in the bitrate behavior was found out to be caused by a bug in iTunes. However, the "iTunes 7.7" bitrates in my test are now assumed to be correct. Since my test Apple has released iTunes 8.0, but apparently its MP3 encoder has not changed. iTunes 7.7 and 8.0 create identical MP3 files.
In addition, I retested the "various" bitrates with the latest encoder versions when applicable.
Since I didn't have the old iTunes version installed I couldn't test the "classical" bitrates with it (which would have been unnecessary anyway).
FhG, iTunes and LAME 3.97 have only one suitable VBR setting for this test so I'd suggest to calculate the average bitrate of these encoders and adjust the Helix and LAME 3.98 settings to match this average. Helix -V60 and LAME -V5.7 appear to be pretty close to this average with my test tracks.
Here are the new results:
Summary
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/8402bb15.png)
Various - table and chart
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/c4166eec.png)
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/98f51e5b.png)
Classical - table and chart
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/b09a25fc.png)
(http://i238.photobucket.com/albums/ff132/alexb2k/HA/1eb30cbd.png)
EDIT
I forgot to mention that if anyone wants to test FhG's bitrate behavior the bitrates must be measured correctly. Most programs don't show accurate bitrate values because FhG doesn't write Xing headers to VBR files.
I used EncSpot Pro's "full scan" option.
Alex, I think this is the 4th time you quoted yourself.
Is it? And they still don't learn...
Maybe I should put the test results presentation in my signature.
(Don't take this personally, Polar. It's just that the "test samples vs. complete library" bitrate difference issue has been discussed over and over again in the listening test threads.)
http://www.listening-tests.info/mp3-128-1/ (http://www.listening-tests.info/mp3-128-1/)
Please report any major bugs. I will make the public announcement tomorrow (actually today since it's after midnight already ).
BTW, many thanks to rjamorim and Polar for hosting the samples!
Just a grammar/punctuation nitpick:
The purpose of this test is to find out which popular MP3 VBR encoder outputs the best quality on bitrates around 128 kbps.
(removed comma and added MP3 )
From the readme:
Linux users are asked to use Wine with "wine wcmd /c DecodeXX.bat" from the "bin" directory.
Apparently "wcmd" was renamed to "cmd" in wine 0.9.21 which was released 2 years ago.
Is there any description of the samples available? E.g. if I want to test a specific genre I'd have to download all samples. Also, if one likes a track, it might be interesting what's the artist/title.
Thanks for the wmcd feedback. I will edit the readmes once I get home this evening.
As for the sample names - they will be published only after the test. With past tests I had the problem, that a lot of people focused on several samples for which I then had lots of results while other samples remained untested.
If a lot of people complain about this, I might publish them, though.