HydrogenAudio

Lossy Audio Compression => MP3 => MP3 - Tech => Topic started by: halb27 on 2013-02-15 09:12:01

Title: lame3100i, a functional extension
Post by: halb27 on 2013-02-15 09:12:01
You can download it from here (http://dl.getdropbox.com/u/2681777/Lame3100i/lame3100i.zip).

What’s the functional extension?

It's an extension to Lame 3.100 alpha2.
It offers additional VBR quality settings -V5+ to -V0+ which cover the average bitrate range from 148 to 317 kbps (for a variety of pop music).

-V5+       148 kbps
-V4.5+    162 kbps
-V4+       176 kbps
-V3.5+    192 kbps
-V3+       208 kbps
-V2+       224 kbps
-V1.5+    240 kbps
-V1+       256 kbps
-V0.5+    291 kbps
-V0+       317 kbps

An alternative way to use the functional extension is --brV+ x, where x is the average bitrate (for a variety of pop music) you want to use, with x between 148 and 317. You can use for instance --brV+ 224 instead of -V2+.

What is it good for?

Lame’s moderate VBR quality settings like -V5 or -V4 usually yield a very good quality. That’s why many users are happy with these settings. Sometimes however tracks contain spots which are not encoded well. Many users want a better quality also for these rather rare events. From current experience Lame3.100 alpha2 quality of tonal problems seems to scale well with -Vn level, but temporal resolution can still be an issue.

-Vn+ uses -Vn as the encoding basis, but adds a certain amount of brute-force safety by forcing audio data bitrate to a target bitrate which depends on -Vn+ level. Moreover care is taken to always provide maximum possible audio data space for the encoding of short blocks which are used when the encoder thinks it is appropriate for a good temporal resolution. Also Lame's default lowpass is lowered a bit in order to make best use of the encoded bits (use --lowpass if you don't like this).

Emphasis is on issues with temporal resolution, but the higher quality levels starting with -V4+ can also have a positve effect for tonal problems.

In a sense -Vn+ combines the quality advantages of both VBR and CBR.

Recommendations

Users who care much about filesize and are content with the functional extension improving short block (pre-echo) behavior, can use -V5+ to -V4+.

Users who don’t like rather obvious issues in their music even when they’re rare but who also care about filesize are best to choose from -V3.5+ to -V1.5+ according to their needs.
For a significant potential for improving tonal issues -V3+ or better is recommended.

Users who don’t care much about filesize but much more about universal top quality are best served by using -V1+ or V0+, or anything in between. -V0.5+ is a good choice which is extremely close to maximum quality possible with lame3100i.

Installation

lame3100i.exe was compiled with Visual C++ 2010. For this reason it is necessary to install the Microsoft Visual C++ 2010 Redistributable Package vcredist_x86.exe. You can download it from http://www.microsoft.com/en-us/download/details.aspx?id=8328 (http://www.microsoft.com/en-us/download/details.aspx?id=8328).

lame3100i.exe uses the fast and lossless mp3packer tool internally to squeeze the otherwise unused bits out of the mp3 file. You can download mp3packer from http://www.hydrogenaudio.org/forums/index....st&p=282289 (http://www.hydrogenaudio.org/forums/index.php?showtopic=32379&view=findpost&p=282289). Put mp3packer.exe into the same folder where lame3100i.exe is located. Many thanks to Omion for this great tool.
In case there is no mp3packer.exe in lame3100i.exe’s folder lame3100i.exe will work, but the mp3 files will be somewhat larger than necessary.
Title: lame3100i, a functional extension
Post by: Kamedo2 on 2013-02-15 10:55:42
The feature brV+ seems to be working as advertised. Great.
Code: [Select]
cores\lame3100i\lame3100i --brV+ %v %i %o
%v  bitrate[bps] of pops and jazz albums(5 min snippet)
0 146932
4 146932
144 146932
148 146932
152 155511
156 157544
160 159483
164 161866
168 163641
172 165594
176 175418
180 179288
184 183231
188 187309
192 191600
196 195465
200 199285
204 203215
208 207176
212 212092
216 215688
220 219378
224 222879
228 227044
232 230911
236 235030
240 239413
244 243539
248 247469
252 251648
256 255503
260 259525
264 263342
268 267286
272 270995
276 274723
280 278480
284 282119
288 285841
292 289190
296 293361
300 297445
304 301341
308 305350
312 309225
316 312301
320 312732
324 312732
Title: lame3100i, a functional extension
Post by: Kamedo2 on 2013-03-19 19:29:11
My personal listening test of MP3 224kbps including this encoder is ongoing, and I've ABC/HR-ed 5 samples.
Lame3100i V2+, LAME3.99.5 V1, LAME3.98.4 cbr224k, Helix V146, BladeEnc cbr224k(low anchor)
Title: lame3100i, a functional extension
Post by: halb27 on 2013-03-19 22:45:43
Wonderful! I'm quite curious about the results.
Title: lame3100i, a functional extension
Post by: halo001 on 2013-03-30 12:34:50
I like what I experienced so far with lame3100i. Something just brought up on my mind, what about lame use CVBR (Constrained Variable Bitrate) instead of a Plain ABR (Average Bitrate) as Encoding Strategy?
Title: lame3100i, a functional extension
Post by: halb27 on 2013-03-30 21:46:52
What the functional extension does is exactly CVBR, at least to a major extent.
I welcome anything within main Lame devolopment that does something like this, no matter whether it's exactly what I've done or something else (maybe better).

However I must admit that to me official Lame's overall progress is so good that I'm not sure I would have done what I've done with Lame's current state at the time I started.
I personally still like the idea of CVBR, but I'm hard-pressed to give very convincing reasons for it with the arrival of Lame 3.100alpha2.
Title: lame3100i, a functional extension
Post by: halo001 on 2013-05-03 06:48:38
I agree with that. It will be close to perfection by the time lame 3.100 is officially released. 

Something about your work, I think lame developers are fully aware of your functional extension. Do you have any plans proposing this (functional extension) to be part of lame's official project?
Title: lame3100i, a functional extension
Post by: halb27 on 2013-05-03 13:24:03
I did. Guess CVBR isn't what they like to do.
Title: lame3100i, a functional extension
Post by: o-l-a-v on 2013-05-04 14:06:25
I did. Guess CVBR isn't what they like to do.


CVBR will result in better sound compared to VBR, both having same average bitrate/ same size?
Title: lame3100i, a functional extension
Post by: db1989 on 2013-05-04 18:53:04
I doubt that. At least in theory, constraining the bitrate reduces the ability of the encoder to allocate bits as it sees fit, hence potentially reducing the quality as some sections are forced to higher bitrates and others lower. Then you have this, three posts above yours:
I personally still like the idea of CVBR, but I'm hard-pressed to give very convincing reasons for it with the arrival of Lame 3.100alpha2.
Title: lame3100i, a functional extension
Post by: halb27 on 2013-05-04 18:55:44
Kamedo2 is doling a test ATM which may bring some insight.

Other than that for people who are concerned about average quality for a test set CVBR is definitely not attractive.
CVBR is for people who are concerned about worst case behavior, people who like say -V3 as the perfect match for their quality needs in the vast majority kind of music, but are not content with -V3 on certain occasion.
In this case a CVBR variant of -V3 might do the job. An average bitrate equivalent -Vx is the fair competitor to this, and it's a priori not clear what gives the better quality. We don't have sufficient experience with it to make it real clear.

No reason however to be very doubtful towards the functional extension's usefulness because there has been positive experience with it (listen to sec. 3.0 of eig for instance: the functional extension brings the solution which standard Lame fails to give even with -V0). I personally am interested mostly in real inadequate VBR behavior however. For me everything started with problem sample 'trumpet' in Lame 3.97 times where VBR provided a real bad quality so I was shocked. The issue was solved with 3.98/3.99. Meanwhile problem sample 'lead-voice' lead to an even bigger shock when using VBR. 'lead-voice' was the main real life background for the development of the functional extension. I don't agree with db1989's argumentation as general arguments as in terms of Lame 3.99 lead-voice is a strong proof that CVBR makes sense - CVBR eliminated the issue. With Lame 3.100.alpha2 the issue with lead-voice is gone (most of the time it's like this when I do a listening test) or is at least brought down to next-to-nothing (sometimes I can still ABX the problem).

That's the situation.
Title: lame3100i, a functional extension
Post by: Kamedo2 on 2013-05-05 20:57:12
Kamedo2 is doling a test ATM which may bring some insight.

Yes, I'm testing MP3 encoders including Lame3100i at 224kbps, and I've done 21 samples out of 25 samples (84% done). I think I can upload the result within 10 days.

Title: lame3100i, a functional extension
Post by: eahm on 2013-05-05 21:35:01
Kamedo2 is doling a test ATM which may bring some insight.

Yes, I'm testing MP3 encoders including Lame3100i at 224kbps, and I've done 21 samples out of 25 samples (84% done). I think I can upload the result within 10 days.

Isn't that bitrate too high to ABX? Wasn't ~175 or ~190 more appropriate? Will we be surprised?
Title: lame3100i, a functional extension
Post by: IgorC on 2013-05-05 21:55:21
IMO it's matter of training.  1+ year ago I've trained a lot for ABXing of high bitrates and wasn't happy with V2. Only V0 was good. Now I've tried V2 and basically it's (close to) transparent.
Title: lame3100i, a functional extension
Post by: Kamedo2 on 2013-05-05 22:07:50
Kamedo2 is doling a test ATM which may bring some insight.

Yes, I'm testing MP3 encoders including Lame3100i at 224kbps, and I've done 21 samples out of 25 samples (84% done). I think I can upload the result within 10 days.

Isn't that bitrate too high to ABX? Wasn't ~175 or ~190 more appropriate? Will we be surprised?

I'm successfully ABXing around 80% of the encoded test materials that had been already tested, but this is tough. ABX criteria is 15+/20(p = 0.02).
Title: lame3100i, a functional extension
Post by: Kamedo2 on 2013-05-16 18:08:42
The test of MP3 encoders including this encoder has finished, and I uploaded the result in here:
http://www.hydrogenaudio.org/forums/index....howtopic=100896 (http://www.hydrogenaudio.org/forums/index.php?showtopic=100896)
Title: lame3100i, a functional extension
Post by: halb27 on 2013-05-22 17:56:19
I'd like to report about recent development and future plans for the functional extension.

In the last few weeks I did some changes for a preliminary version 3100j. It was all about having details more balanced. One of the changes is worth mentioning: the energy (loudness) dependent minimum audio data bitrate. I've always concentrated on a reference energy level which is very moderate and had minimum audio data bitrate decrease/increase rather strongly for energy levels below/above the reference level. This is very welcome for low energy levels as minimum audio data bitrate requirements are low. But I've realized it's not so welcome for very loud spots in the music. It increases average bitrate of pop music pretty much, or the other way around: holds minimum audio data bitrate requirements for the reference energy level rather moderate in order not to increase average bitrate too much.
I decided to increase bitrate requirements for higher energy levels to a smaller amount. In return I could increase the bitrate requirements for the reference energy level and below.

I did intensive listening tests at -V3+ with the problem samples I care about. I couldn't hear any difference to version 3100i with one exception: harp40_1 improved so much that I could ABX the 3100j result against the 3100i result.

I also felt that in the quality range around -V1+ average bitrate is too high. The background is that minimum audio bitrate requirements are pretty strong already at -V2+ and should be increased by a much smaller extent for the higher quality levels.

With the preliminary j version I still had average bitrate increase up to 317 kbps when using -V0+. Severe bitrate increase started with -V0.5+.
But I want to reconsider this totally. I think of something like a parameter -Vplus x, 1 <= x <= 10, to be used together with the standard -V parameter. -Vplus x controls the minimum audio data bitratre requirements. For a low x only cheap requirements concerning only short block behavior are triggered (which come practically for free for very high quality levels). For a high x very expensive requirements are used (makes sense only for -V0). For x ~ 5 'normal' requirements are used which make sense for -V3 or better (and are cheap for the highest quality levels).
I still think of having the -Vx+ and --brV+ options, but with respect to Kamedo2's test I will consider having a lower average bitrate increase of -Vx+ against -Vx for all x, resp. use a higher -Vn for --brV+ x than with version i.

As for Kamedo2's test I will try to find out for the samples Trust, Quizas and Experiencia (3100i was inferior to 3.99.5 here), if original 3.100.a2 and 3.99.5 behave different or why 3100i does a relatively poor job here. Sure this might have an influence on further changes for 3100j.
Title: lame3100i, a functional extension
Post by: BFG on 2013-05-23 05:43:38
Thank you, as usual, for the work on your functional extension, and continued pursuit of perfection!

...did you intend to include a link to 3.100j in your previous post?
Title: lame3100i, a functional extension
Post by: halb27 on 2013-05-23 07:45:58
...did you intend to include a link to 3.100j in your previous post?

I didn't as for now it's a preliminary version, and I wrote what I want to do yet before going final.

However it's fully tested, and if you like to, I can publish it, and have 3100k do what I have planned for 3100j final.
Title: lame3100i, a functional extension
Post by: BFG on 2013-05-23 22:30:29
However it's fully tested, and if you like to, I can publish it, and have 3100k do what I have planned for 3100j final.

I appreciate that, but there is no need to publish the current version just for me.  I am very poor at ABXing and would not be able to provide any good testing for you.
Title: lame3100i, a functional extension
Post by: ShotCaller on 2013-05-24 00:00:31
I would be interested in checking out the latest version if you wouldn't mind uploading it halb27... Thanks!
Title: lame3100i, a functional extension
Post by: halb27 on 2013-05-24 05:35:18
OK, here (http://dl.getdropbox.com/u/2681777/Lame3100j/lame3100j.zip) comes version 3100j. As this is an intermediate version, I will not start a new thread for it.
Whatever comes out of my further plans will be published in version k.
Title: lame3100i, a functional extension
Post by: halb27 on 2013-05-25 14:39:01
I did some listening tests with those of Kamedo2's samples I mentioned.

Most important to me is Trust, as with a 4.0 score this sample had the worst 3100i result in his test.
I could rather easily ABX the issue in the leading applaud part when using 3.99.5 -V5 and -V4 as well as 3.100.a2 -V5 and -V4.
Doing an ABC/HR test with these quality levels revealed 3.99.5 to yield the better quality. I also tried -V3 and found 3.99.5 to be a tiny bit better, too, but the difference was so small to me that this doesn't really matter. Sure my hearing isn't by far as good as Kamedo2's.

I also tried Experiencia and Quizas, but my hearing is so bad that even at -V5 I didn't get distinguishing results.

Sample Trust shows that using 3.100.a2 as the basis for the recent versions of the functional extension may not have been a wise decision. Sure this one sample doesn't mean 3.99.5 would have been the better basis.

As a consequence I'd love to know: shall we have upcoming version k
- both on a 3.100.a2 and a 3.99.5 basis
- only on a 3.100.a2 basis
- only on a 3.99.5 basis?

What do you think?
Title: lame3100i, a functional extension
Post by: lock67ca on 2013-05-25 15:45:30
I'd personally rather wait until 3.100 is in final release before I used it for anything more than testing purposes, so I'm all for using 3.99.5
Title: lame3100i, a functional extension
Post by: GeSomeone on 2013-05-25 17:33:09
Probably one of those cases of win some, lose some.
If 3100a2 is generally an improvement for the range -V2 to -V0 it might be good to stick to it.
Title: lame3100i, a functional extension
Post by: halb27 on 2013-05-26 16:26:25
I am about to implement version k on the 3.100.a2 basis. It makes sense for the mere fact that I can do a bitwise comparison test with the 3100j encoding results. I will publish it when it is finished.
After that I will port the k version back to 3.99.5.
Title: lame3100i, a functional extension
Post by: ShotCaller on 2013-05-27 23:01:28
Thanks for posting 3100j halb27 and for all your work on this extension.

I was hoping though for an explanation for these --adbr switches... from what I understand, changing the values for these switches can produce less artifacts for certain samples. I really don't have the time to determine the best switches for each encode... so, what would you recommend for a "one size fits all" type command string (very high bitrate being no issue)? Would the --insane_factor switch set to 1.0 help me achieve this? Again, thanks for all you great work. It's comforting to know that my MP3 encodes will always (well, 99.9%) be transparent with the help of LAME developers and your functional extension.
Title: lame3100i, a functional extension
Post by: halb27 on 2013-05-28 00:43:33
You needn't care about the --adbr switches. Maybe I will give them away with version k.
For best quality just use a -Vx+ setting in the range -V0.5+ ... -V0+ according to your likings, but it's not necessary to go straight -V0+ IMO. harp40_1 is the only sample where I can hear tiny differences in this -Vx+ level range (totally negligible to me for -V0.2+ or better). You might try this sample and find out for yourself.
Title: lame3100i, a functional extension
Post by: Kamedo2 on 2013-05-31 18:04:15
I tested the --brV+ function. A 5min snippet of Pops and Jazz albums was used. It seems the option is properly working, just like the previous version.

(http://i42.tinypic.com/30dccwm.png)
Title: lame3100i, a functional extension
Post by: shadowking on 2015-12-14 13:17:28
It doesn't work for stereo files with near-mono content. It becomes just like CBR/ABR or a lossless mode that doesn't do M/S stereo / inter-channel correlation.

C:\stuff\music\test>lame3100l yaldonet.wav -V2 --cvbr -1
LAME 3.100 (alpha 2, Jul 23 2013 18:58:01) extension:l 32bits (http://lame.sf.net)
warning: alpha versions should be used for testing only
Using polyphase lowpass filter, transition band: 18671 Hz - 19205 Hz
Encoding yaldonet.wav to yaldonet.mp3
Encoding as 44.1 kHz j-stereo MPEG-1 Layer III VBR(q=2)
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  8921/8921  (100%)|    0:26/    0:26|    0:26/    0:26|  8.6451x|    0:00
32 [  21] *
40 [  0]
48 [  0]
56 [  0]
64 [  9] *
80 [  9] *
96 [ 334] ******
112 [2354] **************************************
128 [4252] ********************************************************************
160 [1627] ***************************
192 [ 249] %***
224 [  54] *
256 [  12] *
320 [  0]
-------------------------------------------------------------------------------
  kbps        LR    MS  %    long switch short %
  130.6        0.0 100.0        94.5  2.9  2.6



C:\stuff\music\test>lame3100l yaldonet.wav -V2
LAME 3.100 (alpha 2, Jul 23 2013 18:58:01) extension:l 32bits (http://lame.sf.net)
warning: alpha versions should be used for testing only
Using polyphase lowpass filter, transition band: 17249 Hz - 17782 Hz
Encoding yaldonet.wav to yaldonet.mp3
Encoding as 44.1 kHz j-stereo MPEG-1 Layer III VBR(q=2)
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  8921/8921  (100%)|    0:48/    0:48|    0:48/    0:48|  4.8021x|    0:00
32 [  19] *
40 [  1] *
48 [  0]
56 [  3] *
64 [  5] *
80 [  9] *
96 [  29] *
112 [  34] *
128 [  41] *
160 [ 218] **
192 [8044] ********************************************************************
224 [  89] %
256 [ 138] **
320 [ 291] ***
-------------------------------------------------------------------------------
  kbps        LR    MS  %    long switch short %
  195.2        0.0 100.0        94.5  2.9  2.6
Title: lame3100i, a functional extension
Post by: halb27 on 2015-12-14 20:27:07
The constrained vbr mode of my variant has a minimun bitrate approach. It doesn't care about the source being ping-pong stereo or near-mono. So yes, for such cases there is a significant bitrate bloat (resp. corresponding security margin) compared to standard Lame.

BTW I am developing a new variant which will work differently. It is based on the fact that usually standard Lame provides an absolutely great quality as well as the fact that for most of the cases where my method has a quality advantage this has only been so when using very high average bitrate.

I looked at the known cases where standard Lame provides an inappropriate quality.

For the first long block after a sequence of short blocks the psy model is lacking information. So some heuristics must do. This doesn't always work as can be seen with the 3.2 second issue of the eig sample. In this case I simply demand for a relatively high minimum bitrate and I have done so in my recent Lame variant versions. This comes virtually for free in terms of additional average bitratre.
Long blocks are made for tonal spots in the music. It can happen however that a long block is used and the music has a low level of tonality. This is the case with the lead-voice sample. In my upcoming version I detect this situation and demand for a pretty high minimum bitrate. This too requires next to no additional average bitrate.

So when using -Vx with the next version these problems will be fixed. Anything else will be absolutely the same as with standard Lame. When using -Vx I will use my minimum bitrate mechanism only to fix these issues, not to have a constrained VBR mode.

There ere other samples where standard Lame's quality is a bit inappropriate for the chosen quality level. Very tonal stuff belongs to this like with the samples herding_calls and trumpet_myPrince. harpsichord music belongs to this group as well (i.e. the harp40_1 sample). The situation isn't too bad though, quality scales with the -V level, but for a fully satisfying quality a high to very high average bitrate setting is necessary.
So there is room for improvement and I will try to do so with my next version. There will be an alternative -Q quality scale. When used my minimum bitrate mechanism will come to work together with a new mechanism which dynamically controls the -Vx level. Other than with my previous variants this is a selective mechanism controlled by the level of tonality. Unfortunately it looks like it can't be made very selective. At extremely high bitrate settings it wil work more or less the way my current variant does. But I expect to improve things at an average bitrate below 200 kbps.
Title: lame3100i, a functional extension
Post by: descrates on 2015-12-17 00:33:29
So when using -Vx with the next version these problems will be fixed. Anything else will be absolutely the same as with standard Lame. When using -Vx I will use my minimum bitrate mechanism only to fix these issues, not to have a constrained VBR mode.


yes, bug in original lame code maybe, in my mod too

http://www.mediafire.com/download/kxxti5bz...15_win32_mod.7z (http://www.mediafire.com/download/kxxti5bz2b9sjw7/lame_3.100a2_intel15_win32_mod.7z)
Title: lame3100i, a functional extension
Post by: halb27 on 2015-12-17 14:36:55
What is your mod about?
Title: lame3100i, a functional extension
Post by: descrates on 2015-12-22 02:07:00
What is your mod about?


320kbps tuning (i cannot remember how to build it)
-b 320 -Y -q0
-V0 -Y or -V 0 (my alternative)