You can download it from here (http://dl.getdropbox.com/u/2681777/Lame3.99.5z/lame-3.99.5z.zip).
What’s the functional extension?
It offers VBR quality settings -V3+ to -V0+ and -V0+eco (economic version of -V0+).
What are -Vn+ and -V0+eco good for?
They improve pre-echo behavior.
Beyond that, they combine the quality advantages of VBR (regarding pre-echo) with the quality advantages of CBR/ABR (with respect to ringing and other tonal issues).
Lame users can be classified into three categories:
a) Users who don’t care about rare quality issues and/or care much about small file size.
The common way for these users to work with Lame is to use -V5, -V4, or similar.
b) Users who don’t like to have obvious and especially ugly issues in their music even when they’re rare but who care about file size as well.
The common way for these users to work with Lame is to use -V2, -V3, or similar.
CBR 192 or similar, or ABR in this bitrate range, is an alternative (but seldom used).
c) Users who want overall transparency or at least a quality which comes close to it, and who don’t care much about file size.
The common way for these users to work with Lame is to use -V0, -V1, or similar as a VBR method, or to use CBR 320 or 256. Using very high bitrate ABR is an alternative (seldom used).
For users of group b) and c) -Vn+/-V0+eco offers significant quality advantages:
We have two major issue classes with most of the lossy codecs:
- temporal smearing (pre-echo) issues
- ringing (tremolo) and other tonal issues.
Let’s look at the worst samples I know for these classes:
- eig (extremely strong temporal smearing)
- lead-voice (extreme ringing, for instance at sec. 0..2)
With samples like these users of group b) can’t be very happy when using -V2 or -V3, because the ringing issues are very obvious and ugly. The temporal smearing of eig is pretty obvious as well, especially around sec. 3. Using CBR/ABR 192 or similar is a good procedure to fight the ringing, but temporal smearing is much worse than with VBR, it’s real ugly.
Things don’t really change when using slightly increased quality settings.
For users of group c) it’s exactly the same thing, with quality requirements and quality received just both on a higher level.
So the traditional way of doing things isn’t totally satisfactory.
Users of group b) can use -V3+ or -V2+ (recommended) and get much better results in the overall view.
Users of group c) can use -V1+ or -V0+eco (recommended) or -V0+ (recommended for the paranoid like me) and get transparency or close-to-transparency. Sure it’s impossible to prove transparency for the universe of music, but it’s true for the samples mentioned. And as these are very outstanding samples within their problem classes and because of the technical details of -Vn+ described below it’s plausible that the approach works rather universally.
How is it done?
-Vn+ uses -Vn internally (-V0+eco uses -V0), but the accuracy demands for short blocks are increased. Short blocks are used when the encoder takes care of good temporal resolution. Audio data bitrate is kept rather high also with long blocks which are normally used.
These audio data requirements are helpful for any kind of problem, they are not restricted to ringing or pre-echo issues.
Moreover a strategy is used which is targeting at providing close to maximum possible audio data space for short blocks.
What’s the price to pay?
Compared to -Vn the increased accuracy demands of -Vn+ raise average bitrate. As -Vn+ is targeting at significant quality improvements compared to -V2 for real bad samples, we need an average bitrate around 200 kbps at least.
-V3+ and -V2+ are designed for users of group b) above, and as such take care of average bitrate not to be much higher than 200 kbps. For my test set of various pop music average bitrate is 205 kbps for -V3+, and 217 kbps for -V2+.
For users of group c) I allow for the full quality resp. average bitrate range mp3 can offer.
-V1+ takes an average bitrate of 257 kbps for my test set, -V0+ takes 317 kbps.
-V0+eco (economic version of -V0+) takes 266 kbps. So -V0+eco comes nearly for free as -V0 takes 260 kbps for my test set.
Unlike versions I published before, mp3packer isn’t really needed any more to squeeze the unused bits out of the mp3 file (with the exception of fractional settings like -V0.5+ between -V1+ and -V0+).
mp3packer brings average bitrate down by only 1 kbps maximum for -Vn+ between -V3+ and -V2+, by 1 to 2 kbps for -Vn+ between -V2+ and -V1+, and by 2 kbps for -V0+ and -V0+eco. So I think we can forget about mp3packer with these settings.
Options
--adbr_short x
sets the minimum audio data bitrate for short blocks to x [kbps] when using -Vn+ or -V0+eco, with x in the range 150..450.
Defaults are 360,370,420,440,440 kbps for -V3+,-V2+,-V1+,-V0+eco,-V0+ resp.
--adbr_long x
sets the minimum audio data bitrate for long blocks to x [kbps] when using -Vn+ or -V0+eco, with x in the range 110..310.
Defaults are 160,170,215,220,290 kbps for -V3+,-V2+,-V1+,-V0+eco,-V0+ resp.
--frameAnalysis
prints detailed information for each frame (L/R or M/S representation, blocktype of both granules, available audio data bits, audio data bits used, etc.). Works for both -Vn and -Vn+.
I think you forgot to mention the lowpass values that are lowered compared to the defaults.
I like the fact that you made an effort to support the range -V 3 - 0 with this version, as the "y" version of this was useless to me. I will try this for the occasions where I hear a good recording but still decide to go lossy (and not lossyWav).
I do admit I am surprised by the bitrates for short blocks, I assumed they were the same as for normal blocks.
edit: short blocks
Thank you. I have used the extension since last Dec and appreciate the contribution. If I may, a couple questions.
First, for those who use -V0+, is there any significant difference between this version ("z") and the prior version ("y").
Second, what is the functional difference between -V0+ and -V0+eco? From the default values of the parameters for minimum bit rates I can see that the minimum bit rate for long blocks is considerably lower for the eco level (220 vs. 290), but are there other differences as well?
Again, Thanks.
.... Solidornot
The main difference between -V0+eco and -V0 is the minimum audio data bitrate for long blocks as you found out.
There's a bit (not much) of internal details however which I didn't report about. In 3.99.5z (not in 3.99.5y) minimum audio data bitrate for long blocks depends a bit on the energy in the frame. The defaults I gave with the --adbr_long description are valid for a frame of rather moderate loudness. For quiet music the minimum audio data requirements are lowered. For louder music the requirements are increased but only up to a certain limit. With -V0+eco this limit for a further audio data bitrate increase is very restricted (~15 kbps IIRC - I can't look it up at the moment as I am on vacation). With -Vn+ this limit is higher (though this doesn't really apply to -V0+ with its close to 320 kbps minimum audio data bitrate).
3.99.5y -V0 corresponds to 3.99.5z -V0+eco. It's not identical but I can't imagine that there's a noticeable difference. No need to re-encode, but maybe you prefer the new possibilities for new encodings.
The main difference between -V0+eco and -V0 is the minimum audio data bitrate for long blocks as you found out.
......
3.99.5y -V0 corresponds to 3.99.5z -V0+eco. It's not identical but I can't imagine that there's a noticeable difference. No need to re-encode, but maybe you prefer the new possibilities for new encodings.
All very clear. Much thanks.
...and for future use (not existing encodes), the 'z' version, since it no longer needs mp3packer, is much more usable.
3.99.5z doesn't work with dBpoweramp R14.3.
I'm not using the command line. I just replaced LAME 3.99.5 app.
Maybe just the VC10 runtime library is missing which I packaged together with 3.99.5z.
Worked ok for me on DBPowerAmp...replaced the original lame.exe and copied the included VC10 Library to the folder as well, added -V0+eco to the Additional CLI. No problems so far..
THank you
The version that's packed with 9.99.5z is "Microsoft Visual C++ 2010 x86 10.0.30319.1"
I have the updated versions installed x64 & x86 10.0.40219.
I am running Win7 64bit.
Lame3.99.5z is a 32 bit exe (which works with 64 bit Windows7) and is supposed to need a 32 bit VC runtime library. OK, sounds like you have it installed. Can you try the older version instead of 10.0.40219 to see whether or not this is the problem?
lame3995z.exe + msvcr100.dll 10.0.40219.325: works here.
That's weird. The regular LAME 3.99.5 works just fine.
Unfortunately, I wasn't able to get 3.99.5z to work with Exact Audio Copy, while 3.99.5 does. (3.99.5z returns a Missing DLL error.) Perhaps EAC uses a version of LAME that was compiled differently than the "standard" LAME? Any suggestions?
In which folder is the msvcr100.dll?
I took it as an occasion to install EAC1.0b3 on my Windows7 system. 3.99.5z works fine. I used
-V0+eco --noreplaygain --silent --pad-id3v2 --ta "%artist%" --tt "%title%" --tg "%genre%" --tl "%albumtitle%" --ty "%year%" --tn "%tracknr%" %source% %dest%
as the command line option for the user-defined encoder lame3995z.exe.
I have the msvcr100.dll in Windows\System32. Did you run vcredist_x86.exe provided in the zipfile?
I have the msvcr100.dll in Windows\System32. Did you run vcredist_x86.exe provided in the zipfile?
Hmm, that's probably my mistake. I'm using a different version of that distributable, so while I have msvcr100.dll, it's not in the system32 folder. Thanks.
I have the msvcr100.dll in Windows\System32. Did you run vcredist_x86.exe provided in the zipfile?
Hmm, that's probably my mistake. I'm using a different version of that distributable, so while I have msvcr100.dll, it's not in the system32 folder. Thanks.
I finally figured it out. I had the 64-bit C++ redistributable installed, and so didn't install the 32-bit version that came with 3.99.5z. Lesson learned!
Is it already possible to use wildcards with LAME?
the last time I checked LAME, it wasn't possible to do this:
lame -V0 *.wav
or
lame --decode *.mp3
Maybe it's possible now, but I don't know where I can download the latest LAME version.
Maybe it's possible now, but I don't know where I can download the latest LAME version.
The latest (non-variant) LAME version is available from http://sourceforge.net/projects/lame/files/lame/3.99/ (http://sourceforge.net/projects/lame/files/lame/3.99/) .
halb27, can you help me understand: What is the advantage of using VBR -V0+ over CBR 320, when the resulting file sizes are virtually the same? Does -V0+ use more accurate short frames than the standard CBR mode?
EDIT: Also, what are the short frame and long frame bitrate targets for the standard -V0?
EDIT 2: From a quality perspective, is there any problem with using -V0+ but setting the --adbr flags to the minimum allowed? Theoretically, wouldn't that allow LAME to use the full range of available bitrates for a short or long frame, and decide to use the smallest one that allows for high quality transparency?
Is it already possible to use wildcards with LAME?
the last time I checked LAME, it wasn't possible to do this:
lame -V0 *.wav
LAME doesn't support wildcards, but you can use a FOR command to perform batch operations from the command line, e.g.
FOR %1 IN (*.wav) DO lame.exe -V 0 %1 %1.mp3
would encode all WAV files in the cwd into MP3 files with the same filenames.
halb27, can you help me understand: What is the advantage of using VBR -V0+ over CBR 320, when the resulting file sizes are virtually the same? Does -V0+ use more accurate short frames than the standard CBR mode?
EDIT: Also, what are the short frame and long frame bitrate targets for the standard -V0?
EDIT 2: From a quality perspective, is there any problem with using -V0+ but setting the --adbr flags to the minimum allowed? Theoretically, wouldn't that allow LAME to use the full range of available bitrates for a short or long frame, and decide to use the smallest one that allows for high quality transparency?
a) The audio data creation process of CBR is different from that of VBR. VBR audio data can be very different even if bitrate is similar. Yes, -V0+ provides the more accurate encoding for frames with short blocks in them. CBR 320 doesn't do a very good job at impulse-like spots in the music (compare for instance eig using CBR320 against -V0+). Moreover there is some suspicion that CBR is flawed a bit.
b) There is no frame bitrate target at all with standard -V0.
c) There is not much sense in setting the --adbr flags to the minimum values allowed. This would get you more or less standard -Vn behavior. You can't totally rely on the VBR mechanism to provide top quality (though not much is left, and from pre-tests I expect robert to get us significantly closer to perfection with Lame 3.100). In case we could perfectly rely on VBR there'd be no room for the functional extension. The essence of the functional extension is to bring a user-selectable amount of brute-force safety to VBR the way most people expect it from using CBR.
However in practical terms using -V0+eco comes close to your idea as average bitrate is very close to that of standard -V0, and you get a significant amount of better quality needed on occasion more or less for free.
A bunch of stuff
Ah! I think I finally understand now. Thanks.
Essentially, the various VBRs better utilize the available bitrate (up to 320kbps) than equivalent CBRs/ABRs, at least at the higher quality end. 3.99.5z, meanwhile, adds an option to strictly enforce the minimum size of long frames and (more importantly) short frames, while the standard 3.99.5 does not have such a restriction, and so - depending upon the algorithm and the quality setting selected - may choose to use frame sizes that are too low, leading to preecho and other problems.
One last question, and I think I'm done. -V0+ doesn't actually make changes to the psychoacoustic model or any of the 3.99.5 algorithms, does it? It simply forces the higher minimum frame sizes and raises the lowpass filter, forcing the existing algorithms to "work harder" to achieve the required accuracy?
On a different note - this discussion kind of makes me wonder why someone hasn't just built a "best possible VBR-quality accuracy under a 320kbps constant frame size" model. In other words, use the VBR approach, but continue to recalculate each frame, increasing the accuracy requirement each time, until a constant 320kbps is achieved. The bit reservoir could be utilized to provide additional accuracy to adjacent frames which are more complex. This would mean that some frames would be more accurate than others, but theoretically, such an approach would yield the most accurate possible ISO-compliant MP3s. (I suspect, BTW, that -V0+ comes close to this.)
EDIT: I'm sincerely hoping this isn't a question simply borne of my ignorance of LAME
Yes, you understand it very well. -Vn+ resp. -V0+eco doesn't change the psychoacoustic parts of Lame. As you said it forces the existing algorithms to work harder (to achieve minimum bitrate requirements, seperately for long and short blocks), and it takes some care to provide close to maximum possible audio data space (important for short blocks) which includes restricting extremely high frequencies to 17.5 kHz.
As for your CBR idea: robert told me that he plans to use the audio data creation process which is used for VBR for CBR as well. So probably you will get that. ATM you can use -V0+ and repackage this into the CBR framework by using mp3packer in case you really need CBR. Audio data won't change of course, so from pure audio considerations there's no use in it.
It simply forces the higher minimum frame sizes and raises the lowpass filter, forcing the existing algorithms to "work harder" [..]
[..] which includes restricting extremely high frequencies to 17.5 kHz.
This means it doesn't raise but lowers the --lowpass, compare to standard LAME.
So while we're asking questions about the working: does raising the lowpass a bit again (like --lowpass 18600), does that hurt the quality or does it only raise the bit rate.
(sorry, I know you spend a lot of effort in tuning to what you believe is optimal, but curious minds like to know).
...So while we're asking questions about the working: does raising the lowpass a bit again (like --lowpass 18600), does that hurt the quality or does it only raise the bit rate.
Depending on the actual situation for encoding a frame a higher lowpass frequency can raise the bitrate a little bit, or reduce the quality a little bit, but most of the time the difference will be next to nothing. Things won't change essentially.
I personally can't imagine that 17.5 kHz lowpassed music can be an audible issue to somebody, but if you feel like it's too low don't worry and use a higher lowpass.
Halb, you mentioned that LAME 3.100 is in development right now - is Robert making that version available somewhere for testers? Based on the changes you've mentioned in that version, I wouldn't mind using it instead of 3.99.5 for ripping my old CDs, even if it is still being tested.
Regarding the "CBR idea" - it sounds like Robert is indeed developing that very idea. Let me try explaining it a different way to make sure. Essentially, I'm looking for an encoder that always provides the maximum possible accuracy, given the 320kbps restriction. Or, to put it a different way, a VBR whose accuracy requirement is continuously adjusted for each frame independently, until an accuracy requirement is found for each frame which fills every frame with 320kbps of data (or achieves 100% accuracy, whichever comes first). Theoretically, this would provide the best quality possible in a standard MP3.
To give a simple example, let's say that 5 long frames in a row would require 270, 280, 270, 290, 300kbps to achieve the -V0 quality (say, 95% accuracy). Reencoding those frames for 96% accuracy would require 290, 310, 290, 320, 320kbps. The last two frames would need to stay at 96% accuracy as they have capped, but the first three could be increased further - say, to 97% or even 98%. Short frames, the use of the bit reservoir, and dynamic adjustments to the lowpass/highpass filters on a frame-by-frame basis, would further complicate this.
Does that make sense?
I recognize that such an encoder would likely be very slow, but I would argue it'd be worth it. I'd even propose a new name for this method - something like DBR (dynamic bit rate).
What you describe is pretty much what 3.99.5z does when you use -V0+. I don't know about the details of what robert has in mind, I'm just looking forward to it. He gave me two test versions to do specific tests that's why I know a bit about 3.100 development, and treatment of tonal issues is really very promising.
What you describe is pretty much what 3.99.5z does when you use -V0+.
Interesting. I have one last question then. Why not just set 310/450--the highest possible settings--as the minimum long/short frame sizes, and run in -V0+ mode? Wouldn't this force the encoder to essentially always use 320 frames, instead of averaging 317 or so (which is what -V0+ is hitting for me right now)? What would the encoder do if these settings required exceeding of the 320kbps overall limit? And, would this ensure a superior rip versus CBR/ABR due to the Variable psychoacoustic model being used?
Halb, you mentioned that LAME 3.100 is in development right now - is Robert making that version available somewhere for testers? Based on the changes you've mentioned in that version, I wouldn't mind using it instead of 3.99.5 for ripping my old CDs, even if it is still being tested.
Ripping and encoding with a lossy encoder that is still under development is a bad idea, in my opinion. If you can't wait then rip to lossless and reencode with whichever version of the lossy encoder is available now, then re-reencode when the stable version is ready.
@BFG:
The more you go to the limits the more often accuracy demands cannot be fulfilled due to lacking data space and due to the priority of keeping bit reservoir large for the sake of short blocks. In this respect minimum audio data bitrate of 290 kbps for long blocks is already pretty high (and overkill in 99.99..% of situations). What exact values to choose for -V0+ is more a matter of taste than a science. If you prefer higher values: go ahead but don't expect a noticeable different bahavior.
As a side note: don't think in terms of frame bitrate, audio data bitrate is what counts. For audio data there's no 320 kbps limit, but there's always a limit of available audio data space (which -Vn+ tries to maximize, but it can't work miracles depending on the encoding situation).
@BFG:
The more you go to the limits the more often accuracy demands cannot be fulfilled due to lacking data space and due to the priority of keeping bit reservoir large for the sake of short blocks. In this respect minimum audio data bitrate of 290 kbps for long blocks is already pretty high (and overkill in 99.99..% of situations). What exact values to choose for -V0+ is more a matter of taste than a science. If you prefer higher values: go ahead but don't expect a noticeable different bahavior.
As a side note: don't think in terms of frame bitrate, audio data bitrate is what counts. For audio data there's no 320 kbps limit, but there's always a limit of available audio data space (which -Vn+ tries to maximize, but it can't work miracles depending on the encoding situation).
Thanks again! I'll probably give the 310/450 limits a try, just to see what happens...but as you mentioned I doubt it'll change much, since even at 290/440 (i.e. -V0+ defaults) the encoder was probably hitting the limits most of the time anyway.
And I did have "frame bitrate" and "data bitrate" confused...I appreciate the explanation.
Well, I'm testing some rather extreme settings tonight to see what I get. The goal is to cram as much meaningful audio data as possible into the 320kbps audio stream, favoring short frames over long frames and lowrange/midrange frequencies over highrange, but not arbitrarily excluding any portion of the spectrum or filling any portion of the MP3 with filler data like would happen with a CBR320. I'm not sure if I'm achieving that.
Settings used are: -v -q0 -V0+ --lowpass -1 --highpass -1 --adbr_short 450 --adbr_long 310 --replaygain-accurate
As you can imagine, encoding is SLOOOOOWWWWW...on my machine, about 2x speed versus 4x for standard -V0+ and 18x for standard -V0.
So, here's my questions:
(1) Would you agree that these settings allow for the most transparent MP3 encoding possible with the current LAME psychoacoustic model, without wasting any space? Or, am I making a mistake by (for example) disabling the lowpass/highpass filters?
(2) In your opinion, is it worth reencoding one's entire library with these settings, or with -V0+, when it's already encoded at standard -V0?
(3) In your opinion, are these settings superior in any meaningful way to standard -V0+?
I'll be interested in the responses. I suspect, in the end, that the only way to answer these will be for me to do some ABX tests.
...I suspect, in the end, that the only way to answer these will be for me to do some ABX tests.
Absolutely. I very welcome this. As you can see from my signature, I personally prefer another solution.
Absolutely. I very welcome this. As you can see from my signature, I personally prefer another solution.
Well, so far I've been unable to tell the difference between this approach and -V0, let alone between this and -V0+. But then again, I've only got laptop speakers and a modest stereo to try it on right now. Anyone else willing to ABX the settings I published two posts above?
(I need to find some more complex songs to throw at LAME, or a hifi system, apparently.)
For the worst tonal and pre-echo problems I know: try lead-voice and eig provided in the zip file. A laptop usually is fine as long as it's not too noisy, but use headphones or earphones and turn up volume if the computer isn't totally quiet. You'll find a difference -V0+(eco) vs. -V0 for sure.
Thanks for all the info on this, halb27 (not to mention providing the z variant itself). After some further testing, I have settled on:
-v -q0 -V0+ --lowpass -1 --adbr_short 450 --adbr_long 110 --replaygain-accurate
This appears to use the standard -V0 approach on long frames (which I find to be satisfactory) but require maximum possible accuracy on short frames. It's also quite a bit faster on my machine - about 9-10x - and only bumps the filesize about 8kbps above standard -V0. This also seems to handle the samples you mentioned quite well (though I would appreciate a second opinion!)
One last question: how does -V0+ handle the minimum requirements for switch (i.e. long-->short or short-->long) frames?
Granules of blocktype 'start' and 'stop' are essentially treated like long blocks, but I require them to use some extra bits.
I did a short test (I am about to leave home) with your setting for lead-voice and trumpet_myPrince, and yes, it sounded fine. I didn't expect that from --adbr_long 110.
I will investigate this further.
I did a short test (I am about to leave home) with your setting for lead-voice and trumpet_myPrince, and yes, it sounded fine. I didn't expect that from --adbr_long 110.
I will investigate this further.
I'm glad to know I'm not the only one surprised by this!
In some ways, however, it makes sense; -V0 (in my opinion) does fine with long blocks; it's only the short blocks that cause trouble. And, reducing the minimum frame size for long blocks leaves more room in the bit reservoir for high-quality short blocks.
As a future feature request, I'd ask that it be possible to specify the minimum size for start/stop (switch) blocks as well.
A preliminary remark: you can drop the '-v -q0' without changing the output.
Though I knew lead-voice has a high percentage of short blocks I did not think that quality gets in when taking special care just of these when using -V0.
However it's not all there is. Take for instance herding_calls (http://dl.getdropbox.com/u/2681777/problems/herding_calls.flac) and listen to sec. 1...4.
There are only long blocks, and -V0 (as well as your setting) is not transparent. It takes some time however to hear the issue, and I need to find the right volume (not too quiet, not too loud). So it's best to listen to the problem at very low quality levels first to hear what the issue is. -V0+eco isn't transparent either, but -V0+eco --adbr_long 240 is (to me).
I agree though that also -V0 resp. your setting leads to a result which I'd call close to transparent. In the end it's a matter of taste what exact parameters to use.
At any rate I agree with you that the essential thing to improve upon -V0 is to take special care of short blocks.
I was asked to provide more parameters. So I have been thinking about a new version for which there are more questions.
The extension has always been about avoiding inaccurately encoded frames, providing close to maximum possible audio data space, and minimum target bitrate for long and short blocks. The priority of these things have changed however since I started, and right now we see that at least for -V0+(eco) the minimum target requirements for long blocks have lost significance.
Default lowpass
The default lowpass of 17.5 kHz originates from the time when avoiding inaccurately encoded frames had priority 1. Questions about the lowpass have shown me that the default lowpass is not liked. And right: lowpassing is nothing that the functional extension has as a goal, so I'd like to omit it and fall back to Lame's orginal behavior in this respect. Any objections?
Additional parameters
The problem with additional parameters is that they require a certain understanding of the mechanism used. I had a hard time already deciding upon --adbr_long and --adbr_short though these don't require too deep an understanding. The problem is that harder stuff confuses people who don't want to dig that deep. As a compromise I can provide these options, but I don't describe them here on HA. Instead I provide an extra documentation file for these with the functional extension itself. Any objections?
More priority to short block behavior
We've seen that the most important thing about the functional extension is short block behavior. Long block behavior isn't that important. So I'd like to reflect that:
a) For -V3+ to -V1+ as well as for -V0+eco I'd like to use a minimum long block bitrate default of --adbr_long 160 which is the default of current -V3+.
b) For -V0+ and -V0+eco I'd like to strenghten the criteria for providing close to maximum possible audio data space. They are quite strong already now, but a bit more can be done. I'll try to find a compromise that doesn't have a serious impact on encoding speed, especially not for -V0+eco.
Any objections?
Streamlining -Vn+ parameter
There's not much use for -V1+ as bitrate is close to that of -V0+eco. Same goes for -V3+ the bitrate of which is close to that of -V2+.
So I'd like to drop -V3+ and -V1+, or, more generally speaking, the concept of providing a continuous quality scale of -V3+ to -V0+ which covers the bitrate range of ~200...320 kbps.
Instead the '+' in '-Vn+' should primarily address increased accuracy of short blocks for the provided -Vn+ compared to -Vn. As a consequence 3.99.5z's -V0+eco should be named -V0+. 3.99.5z's -V0+ additionally takes care of extreme quality for long blocks and should be named -V0++ because of this. Kind of falling back to a prior naming scheme, but consequent IMO. Any objections?
Hi, halb27.
V3+ ends up with the same average bitrate as normal V1.5 on large number of samples.
I have tried V3+ on eig sample and I think it will be interesting to make more detailed tests in some moment as there was a substantial improvement.
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname:
1L = D:\Audio\Test samples\eig\LAME 3.99.5z V3.wav
2R = D:\Audio\Test samples\eig\LAME 3.99.5 V1.5.wav
---------------------------------------
General Comments:
---------------------------------------
1L File: D:\Audio\Test samples\eig\LAME 3.99.5z V3.wav
1L Rating: 4.0
1L Comment:
---------------------------------------
2R File: D:\Audio\Test samples\eig\LAME 3.99.5 V1.5.wav
2R Rating: 3.5
2R Comment:
---------------------------------------
ABX Results:
D:\Audio\Test samples\eig\LAME 3.99.5z V3.wav vs D:\Audio\Test samples\eig\LAME 3.99.5 V1.5.wav
5 out of 5, pval = 0.031
Any chance for bitrates in range of -V5 ... -V2 in a future?
Any chance for bitrates in range of -V5 ... -V2 in a future?
Hi halb27
I hope to get time for more testing maybe next month, and in the 3.99.5y variant that accidentally included -V5+ I did find it useful for ABXing, so if implementation isn't difficult I'd agree with IgorC's request (at least for testing, even if it requires an over-ride switch to allow access to that mode). If not, I might just keep the un-fixed y version for -V5+ tests.
I understand your rationale for limiting the -Vn range to the high-quality end, though I can see uses for lower quality -Vn+ for fixing individual problem tracks. I also understand IgorC's idea of comparing settings that have similar typical bitrates to see which is better.
I'm OK with having to force a lowpass if I want the old behaviour (though I guess the reduced lowpass you use currently could be part of 'eco' mode for any n in a more generic -Vn+eco and we could even add -Y to the commandline if desired when n <3)
As an aside for testing and diagnosis of problem samples, I presume that the old mp3x frame analyzer (see official lame site regarding tuning) that was once in lame (around 3.90.2 era) isn't simple to bundle around 3.99.5z as I haven't seen a compile of mp3x for years. I think the old one would at least show the output, just not in comparison to the input WAV, which I seem to recall being a feature (though it's a long time and memory is fallible)
OK, I can provide -Vn+ for -V5+ to -V0+. With this I think it's best to stay with 3.99.5z's naming scheme -Vn+/-V0+eco.
So far I can see no severe suggestions against dropping 3.99.5z's specific lowpass, so as long as none comes up I'd like to drop it. We can get any individually prefered behavior by using switches like --lowpass, -Y, --adbr-xxxx.
@Dynamic: I don't know mp3x and don't want to do work with it. Did you try 3.99.5z's --frameAnalyzer option? What are you missing?
I haven't tried --frameAnalysis but I was thinking more about digging into problem samples to try to detect certain problems like highly tonal signals during short blocks and coming up with a detection algorithm that could be rolled back into general LAME VBR codebase, which means that the original PCM, the MP3 decoder output and the FFT analysis spectrum would be useful pointers - and just the sort of things mp3x provided, IIRC.
I had the impression that mp3x was once very closely linked to LAME encoder up until around 3.90 time (especially before Dibrom's --alt-preset standard work when early investigations were tuning it from poor performance against the then best-in-class like FhG). But I suspect mp3x now is more than just a compile-time switch in the lame codebase, and would probably need an unjustifiable amount of work to get it running. Maybe if I find time to look into this, I'll set up a compiler, possibly under Linux until I can build a current standard LAME compile or the 3.99.5z version myself, then I can break out at short blocks and dump out the info I'm interested in to plot in LibreOffice Calc (or MS Excel) and try tweaking the code to trigger extra high bitrate only for problem samples and perhaps generate some samples artificially to ABX the correct threshold in the absence of other variables. I might even search for a very old mp3x version and see if provides enough info to germinate some ideas before I put in the effort of getting myself set up to compile the current LAME code frequently.
I'd be hopeful that it must be algorithmically determinable, and thereby of cracking a few of the problem samples like trumpet and Angels Fall First by specific analysis used only when short-blocks are active, and of doing so without increasing LAME VBR bitrate on general currently-transparent music. It clearly won't help herding calls, which only uses long blocks. Presumably that either needs a different short-block detection threshold (unlikely from the sound of it) or needs more bits in the long blocks for some frequency resolution reason that the psymodel is missing.
This work might be some time away, as I have too much going on in life for now, but I'm quite keen to give it a go in some downtime.
You're welcome to do this work. As far as I am concerned your thoughts are nothing for me. Anything with a real impact on psychoacoustics is beyond my scope. My thing is introducing a certain amount of brute force safety into -Vn while deliberately (though also necessarily because of my restricted knowledge) staying pretty much on the surface of mp3 encoding. For this --frameAnalysis is quite appropriate for looking at the encoder's behavior IMO.
I haven't tried --frameAnalysis but I was thinking more about digging into problem samples to try to detect certain problems like highly tonal signals during short blocks and coming up with a detection algorithm that could be rolled back into general LAME VBR codebase, which means that the original PCM, the MP3 decoder output and the FFT analysis spectrum would be useful pointers - and just the sort of things mp3x provided, IIRC.
I had the impression that mp3x was once very closely linked to LAME encoder up until around 3.90 time (especially before Dibrom's --alt-preset standard work when early investigations were tuning it from poor performance against the then best-in-class like FhG). But I suspect mp3x now is more than just a compile-time switch in the lame codebase, and would probably need an unjustifiable amount of work to get it running. Maybe if I find time to look into this, I'll set up a compiler, possibly under Linux until I can build a current standard LAME compile or the 3.99.5z version myself, then I can break out at short blocks and dump out the info I'm interested in to plot in LibreOffice Calc (or MS Excel) and try tweaking the code to trigger extra high bitrate only for problem samples and perhaps generate some samples artificially to ABX the correct threshold in the absence of other variables. I might even search for a very old mp3x version and see if provides enough info to germinate some ideas before I put in the effort of getting myself set up to compile the current LAME code frequently.
I'd be hopeful that it must be algorithmically determinable, and thereby of cracking a few of the problem samples like trumpet and Angels Fall First by specific analysis used only when short-blocks are active, and of doing so without increasing LAME VBR bitrate on general currently-transparent music. It clearly won't help herding calls, which only uses long blocks. Presumably that either needs a different short-block detection threshold (unlikely from the sound of it) or needs more bits in the long blocks for some frequency resolution reason that the psymodel is missing.
This work might be some time away, as I have too much going on in life for now, but I'm quite keen to give it a go in some downtime.
You should give current LAME 3.100 alpha 2--a cvs snapshot--a try. mp3x still does work, though you'll need to compile it yourself, or ask john33 to do it for you.
http://lame.cvs.sourceforge.net/viewvc/lame/lame/?view=tar (http://lame.cvs.sourceforge.net/viewvc/lame/lame/?view=tar)
http://lame.cvs.sourceforge.net/viewvc/lam...l?revision=HEAD (http://lame.cvs.sourceforge.net/viewvc/lame/lame/doc/html/history.html?revision=HEAD)
I was asked to provide more parameters. So I have been thinking about a new version for which there are more questions.
All of your suggestions for the next version make good sense to me, with one exception.
I would argue that users of the highest possible mode (-V0+ or -V0++, whatever you decide upon), are not concerned about encoding speed. So, I would recommend that that setting be tuned to maximum possible quality, with no concern whatsoever for the encoding speed. -V0+eco, meanwhile, could attempt to deliver nearly the same quality but at a much faster encoding speed and slightly smaller file size.
On a different topic -- do you find that
-V0+ -adbr_long 160 creates a transparent "herding calls"? I have tried it, and it does sound transparent to me. But I'd like a second opinion. So, right now I'm using
-V0+ -q0 -adbr_long 160 -adbr_short 450.
Thirdly, I did request earlier an -adbr_switch option. I would like to set the minimum size of these just like you allow us to do with _long and _short. But, it may be best to let the minimum size of switch frames be calculated on the basis of long and short minima?
Finally, Robert--I'm curious if you plan to make any further adjustments to 3.100 based on the discussion in this thread? I see from the prerelease notes that you've made adjustments to handle "lead voice", which is great. I'd also LOVE to see:
- further adjustments to the psychoacoustic model to better handle "herding calls" and a few other tough samples in VBR
- a higher accuracy requirement for short and switch frames, at least in the higher VBRs
- a -Vmax (or whatever you'd like to call it) switch that packs the most accuracy possible into a 320kbps constant audio rate, using the VBR model. That is, have LAME continually search for a "best fit" solution until it always uses up all available space for every frame, taking into account bit reservoir needs etc. If I understand LAME correctly, this would be different from CBR 320 in that CBR can often "waste" space that was unneeded. In fact, this might be a good way to replace the CBR model that some people have complained about.
- a freeformat VBR (ok, you're right, silly idea )
Yes, I will take care of best quality for -V0+, and encoding speed is of very minor concern here.
Also, I will provide all the audio data bitrate parameters including adjustments for the start resp. stop blocktype.
As 3.99.5z -V0+eco herding_call isn't transparent to me -V0+ --adbr_long 160 won't be either.
But you should keep things in relation. Were you able to ABX herding_calls? Sounds like: no. And to me too it's close to transparent, also with standard Lame's -V0. Though quality improvement is always welcome we can't expect transparency for any track and any listener. Being close to transparency in a rather universal way is the best we can expect (besides transparency for most tracks).
lead-voice is another story. Lame's VBR quality is inadequately bad here at the moment, and major improvement is very welcome. Any other tonal problem sample I know is very good to perfect when encoded with current Lame at -V0.
I have tried one more sample. This time a tonal one, "Fugue Premikres notes"
The sample is here (Sample09).
http://www.hydrogenaudio.org/forums/index....st&p=808142 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=96800&view=findpost&p=808142)
3.99.5z did better again and even with a bit lower bitrate.
ABC-HR
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname:
1R = D:\Audio\Test samples\09 Fugue Prem. Notes\LAME 3.99.5 V 1.5.wav
2L = D:\Audio\Test samples\09 Fugue Prem. Notes\LAME 3.99.5z V3+.wav
---------------------------------------
General Comments:
---------------------------------------
1R File: D:\Audio\Test samples\09 Fugue Prem. Notes\LAME 3.99.5 V 1.5.wav
1R Rating: 3.8
1R Comment:
---------------------------------------
2L File: D:\Audio\Test samples\09 Fugue Prem. Notes\LAME 3.99.5z V3+.wav
2L Rating: 4.5
2L Comment:
---------------------------------------
ABX Results:
And I needed foobar's ABXY (not ABX).
foo_abx 1.3.4 report
foobar2000 v1.1.15
2012/10/28 22:13:16
File A: D:\Audio\Test samples\09 Fugue Prem. Notes\LAME 3.99.5z V3+.mp3
File B: D:\Audio\Test samples\09 Fugue Prem. Notes\LAME 3.99.5 V 1.5.mp3
22:13:16 : Test started.
22:13:52 : 01/01 50.0%
22:14:12 : 02/02 25.0%
22:14:34 : 03/03 12.5%
22:14:45 : 04/04 6.3%
22:15:08 : 04/05 18.8%
22:15:31 : 05/06 10.9%
22:15:48 : 06/07 6.3%
22:16:10 : 07/08 3.5%
22:16:12 : Test finished.
----------
Total: 7/8 (3.5%)
Untill now my findings are only positive with 17.5kHz bandwidth, at least for V3+. I don't hear LAME 3.99.5 preserving high frequencies any decently. There are more artifacts than details.
Thanks for your tests, IgorC.
If I skipped the default lowpass we would have -V3's default lowpass of --lowpass 18200 when using -V3+. Would you mind repeating the test using -V3+ --lowpass 18200?
halb27, would you say lead-voice is still problematic to your ears with lame3.100 alpha which is supposed to have improved it (see first red text item) (http://lame.cvs.sourceforge.net/viewvc/lame/lame/doc/html/history.html?revision=HEAD)?
It sounds like Robert may already have doing something along the lines of what I'm envisaging, which should apply to lead_voice, and I hope to find a bit of quiet time today to test 3.100.a1 on my favourite samples.
BTW, thanks Robert for the info about mp3x. If I get enough down time to take it seriously, I'll probably ask john33 if he can compile it for me on the latest alpha.
With the test version robert gave me lead-voice was close to transparent, a very good result. Guess it's the same thing with the alpha version. We can expect fine behavior also in this respect with 3.100.
Here (http://dl.getdropbox.com/u/2681777/Lame3.99.5c/lame-3.99.5c.zip) comes a preliminary new version to play with.
It provides -V5+ to -V0+, and lowpass behavior of -Vn+ is identical to that of -Vn.
It also provides -V0+eco, but in this case a lowpass of 18.2 kHz is used. Without a lowpass a high amount of unused bits in the file would be introduced.
There are several individual options which are described in an extra documentation file in the context of what the functional extension machinery is doing.
Compared to 3.99.5z there is more emphasis on short block behavior.
Enjoy!
Thanks for your tests, IgorC.
If I skipped the default lowpass we would have -V3's default lowpass of --lowpass 18200 when using -V3+. Would you mind repeating the test using -V3+ --lowpass 18200?
I can't tell the difference between these two (17.5 vs 18.2 kHz).
So I guess both 17.5k and 18.2k are fine at least for these two samples.
And just to clear, my two previous tests were between 17.5 vs 19 kHz, respectively for V3+ vs normal V1.5 because these two settings yield the same bitrate on average.
Thank You for 'c' version.
Thanks for your test, IgorC.
I think I've found a solution for the --lowpass question: keep -Vn's lowpass behavior with -Vn+, but provide a -Vn+eco variant for any -Vn+ level which uses a somewhat lower lowpass and on the other hand provides (with the exception of -V0+eco) a slightly better short block behavior.
After all, using a very high lowpass or no lowpass at all is more for the theoretically oriented folks who've read that frequency range should go up to 20 kHz or so. There's also no reason to restrict an eco mode to -V0+. Will go to work with it.
Thanks for posting the "C" version, halb27. I haven't gotten to use it much yet, but so far I like what I'm hearing.
I have one quick question, to help me compare "C" to "Z": what were the effective settings for the new options (--adbr_startshort, --adbr_long_max, --adbr_trans_plus, --adbr_tolerance, --adbr_snrinc_max) for -V0+ and -V0+eco in the Z version?
Recent changes in the strategy of the functional extension weren't too lucky:
I implemented the universal -Vn+eco, but with the exception of -V1+eco I did not get a better listening experience compared to -Vn+, nor could I save bitrate, nor could I get seriously better internal parameters while keeping the bitrate. So a universal -Vn+eco may sound nice but has no real meaning.
As I was doing intensive listening tests I also found a lowpass is necessary for eig. Without it the result even with -V0+ or -V0+eco is audibly worse than when using a lowpass.
So I will also give away the nice idea of staying with -Vn's lowpass behavior when using -Vn+. I'll default to --lowpass 18200 whenever Lame uses a higher lowpass or no lowpass at all.
While I will consequently prefer short block behavior over long block behavior I went too far with thinking that long block behavior can be covered by -Vn behavior to a large extent. It`s true as long as we're happy with requiring the somewhat vague 'close to transparency' not to be very strong. But for -V0+eco to me the result isn't really satisfying for 'trumpet_myPrince'. The meaning of -V0+eco has turned a bit into using -V0 internally while keeping bitrate around 260 kbps and introducing only a small amount of unused bits. That's rather techinical and has turned the --adbr_long value low with the consequence that quality isn't totally adequate IMO. I'll concentrate on quality again, and for the lower quality levels of -Vn+ I am very happy with the quality compared to the average bitrate, but in the 260 kbps range I'm not. With the surprisingly good results I got for -V1+eco I will try to reach a better quality in the 260 kbps range based on -V1. -V1 gives more space for improvements due to a higher --adbr_long. If successful this would kick off -V0+eco for the sake of a better -V1+.
@BFG:
3.99.5z's default parameter values were 290/220 (-V0+/-V0+eco) for --adbr_long, 310/253 for --adbr_long_max, 440/440 for --adbr_short, 310/235 for --adbr_startshort, 10/10 for --adbr_trans_plus, 30/30 for --adbr_snrinc_max. The tolerance mechanism was changed in order to make the tolerance available in a clear way as a parameter. In terms of --adbr_tolerance 3.99.5z's tolerance value was more like 5 for -V0+/-V0+eco. 3.99.5c cannot perfectly be compared with 3.99.5z since introducing more parameters made me clean up the code as I had to change it anyway. This has a certain impact also on functionality. Especially I give short block behavior a consequent preference now over long block behavior when it comes to keeping bit reservoir large. The machinery has changed a bit, but I think there's only a very low chance that the changes are audible.
As for audible effects and the outline of further development: see above.
Sorry, in my last post '310/253 for --adbr_long_max' should be '310/235 for --adbr_long_max', and '310/235 for --adbr_startshort' in my last post should be '380/380 for --adbr_startshort'.
Here (http://dl.getdropbox.com/u/2681777/Lame3.99.5d/lame-3.99.5d.zip) comes a new version.
a) The default lowpass is 18200 now in case Lame uses a higher lowpass or no lowpass at all.
b) Emphasis is even more on short block behavior. Even -V5+'s short block behavior is close to what's possible.
c) There's no -V0+eco any more, just -V5+ to -V0+ which cover the average bitrate range ~180 kbps ... ~320 kbps.
-V1+ takes the role of -V0+eco.
With the exception of -V0+ and the lowest -Vn+, average bitrate is raised in order to more homogeneously cover the range 180...320 kbps.
I've done a lot of listening tests and I'm happy with what I've heard.
Enjoy!
Here (http://dl.getdropbox.com/u/2681777/Lame3.99.5d/lame-3.99.5d.zip) comes a new version.
Thanks for the "D" version! Unfortunately, the adbr commandline options seem to do nothing in this version.
For example, I tried setting
-V0+ --adbr_long 160 --adbr_startshort 260 --adbr_trans_plus 10, and the standard -V0+ settings seemed to be used instead. Only when I changed the -V switch did the output appear to change.
Sorry. Glad you tested so quickly.
I've corrected it as well as some minor bugs in the documentation.
The link above (http://dl.getdropbox.com/u/2681777/Lame3.99.5d/lame-3.99.5d.zip) has the corrected version.
I did a minor change: default to the 'universal' --adbr_tolerance 5 also for -V0+. Link updated.
But apart from error corrections I stop changes now for this preliminary new version.
I have enjoyed reading this very technical thread even though I am not in any way a developer of codecs. I have a very tangential question that I pose purely as a matter of curiosity.
While reading about these efforts to more effectively resolve "pre-echo", which reduces the transparency of LAME encoded lossy files, by focusing on how Short Blocks are used by the encoder, I found myself thinking through the explanations for why this problem occurs.
Several posters indicated that the encoder (all MP3 encoders?) use Short Blocks for encoding areas of dynamic change in music - sharp attacks, instant volume surges, percussive sounds, etc., and that the LAME encoder does not necessarily do this as efficiently or cleanly as it could, which thereby causes the "pre-echo" and reduces the perceived transparency of the resultant lossy file. Please correct me if my understanding is not correct.
My question is whether the way a musical selection is mastered affects the encoder's performance in this regard. Would a rock musical selection mastered in the "modern" way, e.g. more compressed, possibly brickwalled, with dynamic range of about 6dbs or so, be "easier" for LAME to encode because of the fact that the dynamic changes within that selection have been truncated relative to a more dynamic selection, such as a classical symphony recording (which may have a dynamic range in the 16-18db area)?
If so, is this the reason why most pop, rock, and country music is mastered in this way nowadays, e.g. because it will sound "better" when ripped to a lossy format? (Don't know if AAC has similar issues)
Thanks!
Throwing away part of the dynamic range for the sake of a higher loudness has nothing to do with transient, impulse-like spots in the music lossy codecs can have problems with.
Sorry. Glad you tested so quickly.
I've corrected it as well as some minor bugs in the documentation.
The link above (http://dl.getdropbox.com/u/2681777/Lame3.99.5d/lame-3.99.5d.zip) has the corrected version.
Thanks for the quick response! The new version does indeed recognize the adbr switches, and I am testing it now.
Out of curiosity, why did you decide not to default all long frames to a much lower size (160kbps or similar) as you originally stated? Was it because of the difficulty with the herding calls sample?
trumpet_myPrince triggered this (see post #59).
Can you wait a bit please with testing? I've just found that there can be improvement with treating an out of data space situation. It won't change things essentially but I'd like to implement this improvement into a further new version. (I know it's a bit too many versions ATM, sorry, but cleaning up the code has made me give special attention to all the special situations, unfortunately not at the same time. I know it's bad, but holding improvement back would be worse).
Here comes another version (http://dl.getdropbox.com/u/2681777/Lame3.99.5e/lame-3.99.5e.zip). Sorry for the flood.
There was a bug in version d which allowed encoding a frame with bitrate >> targetbitrate in a situation where mp3's channel or granule size restrictions were hurt. Now bitrate is reduced which can lead to a regular situation without violating restrictions.
Another criterion for lowering bitrate or not in an out-of-channel- or out-of-granule-space situation is made more natural with version e. With version d bitrate was lowered as long as there was also an out-of-frame-space situation. This often lead to an SNR increase of 0, that is -Vn behavior. Now I reduce bitrate only when there is an out-of-frame-space-situation after reducing assigned bits to the limits of the channels resp. granules concerned thus roughly simulating Lame's machinery to deal with out-of-space situations.
You can see the improved behavior over version d as well as z when looking at the --frameAnalysis print for harp40_1 (supplied in the zip file) for frame #99, #393, #489.
Hopefully this is all there is.
What do you think about the quality/average bitrate distribution for -V5+ to -V0+?
A related question is:
Is it better to achieve an average bitrate of say ~220 kbps by using -V3+ as the basis as it is ATM (which means a rather advanced --adbr_long value can be used), or is it better to use -V2 as the basis (and a smaller --adbr_long value)? Answers are related to where to put the most attractive average bitrate for -Vn+, while looking at all the -Vn+.
Except for -V0+ I personally have no real bias. For a start I tried to give a relatively homogenous distribution in the overall bitrate range, but that's not all there is.
halb27,
If You don't mind here are a few questions to make an idea what to expect.
Which versions have sense to test and which are already obsolete (if any)? z,e,d,c,?
What are your general plan and thoughts about your functional extention of 3.99.5? Is it any near to be complete or You are in the middle of experiments?
I would like to perform a serie of a tests with limited numbers of samples while there are new versions each weak or so and report the results here if You wish. Once the long term version will be released it will interesting for me to make a detailed test on a large number of samples.
Hallo IgorC,
You're absolutely right. I've been too involved in development recently, and providing new preliminary versions again and again is very unlucky, especially for those very welcome users like you who are willing to do real tests. I absolutely feel with you. So please forgive me.
And due to experience ATM I can't say 'version e is it'. So I will do the following: do some more experiments but don't publish any result. After I've cooled down, that is: haven't found anything else that can be improved within a week or so after the last changes, I'll come back with the final result of current development.
Great.
As for version 'e', V5+eco is twice faster than V5+. Does it mean that eco preset trades speed at cost of quality? If it does then is there any setting to enable the highest quality for eco?
I like to listen an encoder at low bitrate to familiarize with artifacts and then go up to transparent rate. So V5+eco (130 kbps) is a reasonable start and then gradually go up to V0+.
There is no -V5+eco. I gave up the -Vn+eco idea (see above).
Unfortunately when using Lame (original as well as my variants) you can always append non-numeric stuff to the quality level which is ignored. '-V5abc' does exactly the same thing as does '-V5'. ATM I just look up the last letter and see if it's a '+' which triggers -Vn+ mode. After having realized I'm in -Vn+ mode I remove the '+' and let Lame do anything else for finding out about quality level. So -V5+eco is -V5.
Please wait until I'm finished with current work. I'm working on preventing unused bits. This will have a major influence.
A short outline:
Whenever possible I hold a reserve of bits so high that in case the next frame has a short block, it can be encoded with the number of bits which corresponds to --adbr_short.
Frame size is chosen thus that targetbits + bits reserved fit into the current frame. Frame size cannot be choosen arbitrarily however, but in steps according to the frame bitrates allowed (usually 32 kbps stepsize [that is 836 bits stepsize for 44.1 kHz sampled music], but 64 kbps [1672 bits] stepsize when going from 256 kbps to a 320 kbps frame). Depending on the chosen frame size the number of bits reserved can exceed the maximum allowed number of bits which is 4088. This way up to 835 resp. 1671 bits can be unusable because they cannot be reached by the next frame. For bitrates way below 256 kbps and values for --adbr_short way below 460 the percentage of unused bits is low so the problem is negligible. Other than that it is very real, that's why quality levels like -V0.5+ blow up bitrate significantly because of unused bits.
By intelligently reducing targetbits, bitreservoir requirements, or turn the otherwise unused bits into used ones, all according to what's best in the actual situation, I am about to solve the issue. Though not really needed it has a positive effect for the lower quality levels too.
The maybe most important thing however is that when doing so -Vn+ is very usable also for n between 1 and 0. I do want to have a very good setting that yields an average bitrate of ~270 kbps. That's why -V0+eco formerly, or -V1+ with version e. With the avoidance of unused bits it can be -V0.7+ or so instead.
Which allows to put the parameters of -Vn+ so that average bitrate is closer to that of -Vn than it is in version e (with the exception of -V0+). While not too small --adbr_long values help a lot on tonal problems it's not all there is. A lower -Vn and a higher --adbr_long is not necessarily better than a lower --adbr_long and a higher -Vn which yields the same average bitrate. harp40_1 is a sample that shows that -V2 is a good basis for achieving good quality, -V3 is more problematic. I will go into this deeper.
Sp please wait a bit yet.
BTW, thanks Robert for the info about mp3x. If I get enough down time to take it seriously, I'll probably ask john33 if he can compile it for me on the latest alpha.
John was so kind and compiled current LAME 3.100 alpha 2, but it looks like you'll have to compile the frame analyzer mp3x yourself.
Thanks for the pointer, Robert.
I've grabbed 3.100 alpha 2 from rarewares.org (http://www.rarewares.org/files/mp3/) / LAME bundles (http://www.rarewares.org/mp3-lame-bundle.php) so at least I can see how the latest version handles certain problem samples before I plough ahead.
This thread has me curious: Robert, have you considered incorporating some of halb27's "minimal bitrate" approaches to combat some of the issues discussed in this thread?
I suppose a superior approach would be to tweak the psychoacoustic model itself so that it better handles such samples. But I can imagine how difficult it must be to do so without degrading the model's performance elsewhere.
BFG, I can't speak for Robert, but my own thoughts are along the psymodel lines.
I think that:
a fairly large proportion of cases where Lame3.99.5 -Vn has problems that halb27's -Vn+ version fixes consist of sharp transient (highly localized in the time-domain but spread out in the frequency domain) simultaneous with a tonal signal (highly-localized in the frequency domain but spread out in the time domain).
The time-frequency product tradeoff type characteristic (localized in one means spread-out in the other) is analagous to Heisenberg's Uncertainty Principle (Δt.Δf ~ constant).
The mathematics of transforms such as MDCT (or FT) means that:
if you have a long block, you have a lot of frequency bins, each of which is fairly narrow in bandwidth, allowing fairly precise reproduction of long-duration tonal signals (localized peaks in the frequency domain) even with relatively imprecise values* stored for each frequency bin (the imprecision implies lower bit-depth and hence lower bitrate). As these tonal signals are spread out in the time domain, any time-domain variation is slow enough not to need precise representation.
*these frequency-domain values are complex numbers, basically implying that they carry information about both amplitude and phase. Values from neighbouring bins actually interfere when transformed into the time domain, allowing reproduction of frequencies more precisely defined than the bin-width itself.
Stlll in a long block, if you have an event that is localized in time, however, such as a transient, you can reproduce it, but it requires much greater precision for the values of each frequency bin to sum together in the time domain with correct phase to reproduce the time-localization to prevent it from being smeared out like a soft noise over a longer time (which produces pre-echo and post-echo, though post-echo is more readily masked). Such precision (or bit-depth) over so many frequency bins requires a high bitrate.
An alternative is to detect these time-localized transients and split the time into, say three short blocks. There are now fewer frequency bins in each short block (each having greater bandwidth) but there's less smearing of time (the maximum smearing being the duration of the whole short block), and sufficient time-localization can be achieved with a modest precision of the values for each frequency bin, thus a modest bit-depth and bit-rate (at the expense of frequency-smearing). As time-localized signals are frequency-unlocalized (broad spectrum, noiselike) that's often not a problem.
If there is a tonal (frequency-localized but time-smeared) signal to be represented within the short block that we don't think will be masked by the loud transient, its frequency can be reproduced more accurately only by increasing the precision of the values for each of the frequency bins (because the summation of interfering components of neighbouring broad bandwidth frequency bins when we convert back to the time-domain will then accurately preserve the frequency and phase of the tonal signal. This greater precision, as before, requires greater bit-depth to represent the values in the transform-domain and thus higher bitrate.
It's this latter case that -Vn+ seems to solve, but it doesn't currently detect that there actually IS an important tonal component that isn't masked by the transient (pre-masking and post-masking), it just assumes that there might be, so to be on the safe side, employ a much higher bitrate (much higher precision of bin values) during all short blocks.
For any encoder, with enough processing time, it should be possible to derive an extra measurement on the analysis FFT in the psymodel, but only do the check once short blocks have been triggered (and only test the check on short blocks). That check would look for tonal signals (frequency-localized) during these short blocks, and probably during the switching windows too (long->short and short->long), to determine whether any of them might not be masked entirely by the transient and whether they require higher precision in the transform-domain quantization (and thus higher bitrate) to maintain their frequency precision despite the wide bin-width. It might be possible to determine a suitable mathematical function to determine the required quantization precision from listening tests on tone+transient signals of varying relative amplitudes (and varying tone frequency ranges) and to build in enough margin of safety to account for practical limitations arising from window functions and the like, or failing that to simply determine a threshold of 'tonality' that triggers the encoder to turn up the precision to the maximum for the affected short blocks. Either way would mainly solve the problem cases without boosting bitrate for many general unproblematic short blocks, which is the efficient approach normally adopted in LAME VBR tuning.
Robert has improved the lead-voice problem sample in the latest 3.100 alpha, which I'd have put into this category, so I'll do some keen listening tests to see what might be fixed. Having taken a quick look at the diffs for the latest psymodel.c seem to include a good deal of stuff relating to tonality measures, so I'm hopeful that a lot of the problem samples are going to be hard to ABX using 3.100 alpha2 when I get time to try.
There remain some problems that don't fit this tonal+transient during short-block description, so halb27's -Vn+ modes will still have mileage while the psymodel hasn't fixed them.
BFG, I can't speak for Robert, but my own thoughts are along the psymodel lines.
I think that:
Thanks for the explanation Dynamic; I'll need to read through it a couple more times to ensure I fully understand it.
I have a lot of interest in this area, but not much understanding!
In the meantime, I have a (perhaps silly) question: would adding a twopass system to LAME help the tonal or sharp attack problems in any way?
That is, if LAME already knew what the data in future frames looked like, would it be able to more accurately encode the current frames and/or anticipate tonal or sharp attack problems?
Yes, a lowpass can help.
Yes, a lowpass can help.
Twopass, not lowpass...that is, fully analyzing the file once, and THEN going back and compressing it.
I don't know if it's a viable option for LAME but I have seen it work wonders for DivX and other lossy video codecs.
Sorry.
A twopass system could help a bit to better manage potential out of data space situations. I reduce the accuracy requirements a bit for a frame with granules of type start and short because there is a chance that the next frame will have a short block too. If I knew that I could do better. But it's of minor concern IMO because I think it's best to provide a large data space for the first frame in a sequence of short blocks. From the second frame we can't have much more than the data space of a 320 kbps frame. As the accuracy reduction for a startshort frame isn't severe (amd can be personolized downto 0) I don't think it's worth while going into the pain of developing a twopass system and having the speed penalty when using the encoder.
A twopass system could help a bit to better manage potential out of data space situations...
I suppose I was suggesting a twopass more as part of the "primary" LAME build, not your variant. I agree with your comments, however; I can only see a twopass system being beneficial if a person is trying to create the most accurate MP3 possible, given the 320kbps limit. In your variant, I would think the additional time requirement would only be worthwhile for -V0+ or higher.
I think what I said holds true for mp3 in general.
AFAIK video encoding is essentially based on the relation between consecutive pictures in the video. A similar thing is with lossless audio encoding where the current wave sample is encoded as a formula based prediction from the previous samples + coding the prediction error. In these cases a multipass system might help.
Lossy codecs like mp3 however partition a track into frames each of which is encoded individually. The encoding process of the current frame does not depend on the contents of the neighboring frames. So what should a multipass process do? The only exception to this is bitreservoir usage. A multipass system can principally lead to a better bitreservoir management. But for quality settings not very high running out of data space is a minor issue. Even with very high quality settings we are hard-pressed to run out of data space for tonal frames, and we are hard-pressed not to run out of data space in case of a sequence of several short block granules, and a single-pass bitreservoir management for just one short block granule is possible (I do that with the extension variant).
So within the mp3 framework, it would be hard to find a niche where a multipass process could bring a real advantage.
I'm back having finished everything on my mind.
As I said in a previous post I wanted to get rid of unused bits by either filling up the unused bits with audio data or giving away the unused bits together with some audio data, whichever seems most appropriate. I was successful in the 270-300 kbps average bitrate range, but I did not succeed to generalize the approach to the entire bitrate spectrum.
I had a hard time thinking it over, and finally dropped the idea. After all filling up unused bits with audio data produced for only this reason, or - worth in terms of quality - giving away audio data just because of this is a bit questionable.
So I did something else: I call Omion's fast and lossless mp3packer tool from within lame3995f to do the job. All it takes is to put mp3packer.exe into lame3995f's folder. Of course lame3995f works also without having put mp3packer there, but the mp3 files will be somewhat larger.
While trying to realize my finally unsuccessful approach I had to care about speed because filling the otherwise unused bits with audio data turned out to be an iterative process. As a result I redesigned the iterative process for arriving at target bitrate. Though some of the other things I did are not in favor of speed, the final speed increase will be noticeable, especially with the higher quality settings.
With these changes I could bring things to the limits: keeping bitreservoir high is done now without any compromise, and target bitrate for short blocks is 460 kbps now for any -Vn+.
I did a lot of listening tests for testing about where to put the internal parameters for the various -Vn+ levels in terms of what are appropriate average bitrates in what overall sense. Sure this is a matter of taste to a certain extent. I ended up with the following systematics:
Users who care about universal good quality in the first place, but also about file size (the typical -V2 users) need a bitrate of 200 kbps +/- ~20 kbps according to their needs. So I designed -V5+ to -V2+ to be in this range.
Users who care more or less only about universal top quality can use -V1+ (~257 kbps) or -V0+ (~300 kbps).
You can use the --adbr_xxxx options to finetune longblock and shortblock behavior according to your needs.
Please tell me if you have other ideas of --adbr_xxxx defaults for the various -Vn+.
In the final 3995f version to be published in a new thread I'd like to give away the --adbr_xxxx options with the exception of --adbr_long. Please tell me if you don't agree.
Here (http://dl.getdropbox.com/u/2681777/Lame3.99.5f/lame-3.99.5f.zip) is 3.99.5f for download.
In the final 3995f version to be published in a new thread I'd like to give away the --adbr_xxxx options with the exception of --adbr_long. Please tell me if you don't agree.
Do you mean by that: to remove the other options? I only experimented with --adbr_long (and --lowpass) on the previous versions, so I'm not gonna say I need the others.
Yes, I'd like to remove all the speciial options of the functional extension except for --adbr_long.
Yes, I'd like to remove all the speciial options of the functional extension except for --adbr_long.
Personally, I
like having the additional options, but I can't say I
need them. I prefer to have standard -V0 behavior on long frames, and -V0+ behavior on short/start/stop frames, and I can approximate that with only the --adbr_long option.
I have a different question regarding --adbr_long, however. (This may have an obvious answer but I have not checked into it yet.)
Will your variant still allow silence to be encoded at 32kbps, and other low-complexity frames to be encoded at 128kbps or below, as default -V0 behavior will allow?
a) I can keep all the --adbr_xxx options if this is preferred. I just wanted to have things simpler again, but that's not essential to me.
b) Silence is encoded as in original Lame with the exception that bitreservoir is held at maximum. What this means for frame size see c)
c) Non-silent long block frames are encoded according to the --adbr_long value and the energy in the frame. In case the energy is very low, audio data bitrate can be significantly lower than the --adbr_long value. Frame size is chosen as the smallest frame to hold the corresponding audio data while keeping bireservoir size extremely close to 4000 bit (during the encoding process. After this was done for the intended purpose of providing maximum audio data space for short blocks, the final usage of mp3packer repackages anything into frames of the most appropriate size.)
I'm considering doing a listening test of the encoder.
I have had four successful 192kbps ABC/HR tests, and halb27 says at least 200kbps is needed, but Helix mp3 encoder VBR can't do over 230kbps over typical songs.
So I want to test somewhere in between (for example, 224k). I'm thinking of:
lame3995f -S -V3+ %i %o
lame3995 -S -V1 %i %o
lame3984 -S -q 0 -b 224 %i %o
helix/hmp3 %i %o -X2 -U2 -V148
bladeenc -quit -nocfg %i %o -224
Average bitrate of 25 samples I'll be using:
229 - this encoder V3+
228 - LAME 3.99.5 V1
225 - LAME 3.98.4 CBR
224 - Helix mp3 encoder VBR(2005)
224 - BladeEnc CBR(low anchor)
Is there a better option that I should use? Should I wait for another extension(s) that is coming soon?
Such a test is wonderful, Kamedo2. You are welcome to use version 3.99.5f, because I have put anything I had on my mind into this version.
But again, the question from your other listening thread of what samples to use for getting average bitrate drops in.
I'm also an advocate for the 'take a collection of normal music' for this. I can't see a problem with reproducibility because the collection can be a collection of 30 sec. snippets which can be published. Collection needn't be huge, but it should be more or less representative of the genres included in the test.
With my personal test set of various pop music average bitrates are:
3.99.5 -V1: 223 kbps
3.995.f -V3+: 206 kbps
3.99.5f -V2+: 220 kbps
Helix -X2 -U2 -V148: 226 kbps.
This corresponds pretty well with your results for 3.99.5 and Helix, but not with your 3.99.5f result. Did you put mp3packer.exe into the Lame3995f folder? For version f this is essential because I've optimized any internal detail for quality and rely on mp3packer to squeeze otherwise unused bits out of the output file.
As for 3.100a2: could you do a small a priori test to check whether it's really not worth while including it? While I agree that alpha versions should not participate my impression is that improvements on some problems are so strong that it may be worth while including it. Sure this version should not be treated like a final version when judging about the results.
Halb27, thank you for your great advice! I put mp3packer into the lame3995f folder and the bitrate was significantly reduced.
I think I should use V2+ then, what most people use, according to a poll.
I calibrated these encoders to around 224kbps on 63min of various pop songs.
219k - this encoder V2+
221k - LAME 3.99.5 V1
224k - LAME 3.98.4 CBR
225k - Helix mp3 encoder VBR148(2005)
224k - BladeEnc CBR(low anchor)
As for 3.100a2: Is 5 samples ABC/HR on 160k, LAME 3.99.5 VBR and LAME 3.100a2 VBR and LAME 3.100a2 CBR OK?
Fine.
Nice that you want to give 3.100a2 a try. The details of how to do it are all up to you IMO.
LAME 3.100a2 a priori test: finished. There might be a big improvement in 3.100a2.
http://www.hydrogenaudio.org/forums/index....mp;#entry816614 (http://www.hydrogenaudio.org/forums/index.php?showtopic=97769&st=25&gopid=816614&#entry816614)