Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Retune of FLAC compression levels (Read 18569 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Retune of FLAC compression levels

Hi all,

I'm considering retuning FLACs compression levels to work more like TAKs. Where TAK has p0, p0e, p0m, p1, p1e, p1m etc, FLAC will have its compression levels similarly grouped. levels 0, 1 and 2 will be the fastest decoding, 3, 4 and 5 will be slighly slower decoding, 6, 7 and 8 will be decoding the slowest. In each group, a higher levels gives higher compression and slower encoding. Beyond the grouping, material with a resolution above 48kHz will have presets 6, 7 and 8 go to lpc order 32, and presets 0, 1 and 2 change blocksize from 1152 to 4096, like the other presets.

A more thorough explanation of what exactly changes is here: https://github.com/xiph/flac/pull/576 There are also some links to results of comparisons.

The main reason for this change is that I think there are less 'useless' presets in here, presets are better differentiated.

CDDA input, encoding:
X

CDDA input, decoding:
X

Hi-res material, encoding. Colors are swapped, sorry for that
X

Hi-res material, decoding:
X

I'd like to hear some opinions on this change.
Music: sounds arranged such that they construct feelings.

Re: Retune of FLAC compression levels

Reply #1 –
Here's a x64 windows binary with the proposed settings

Also, I didn't notice the decoding graphs lacking an x-axis. The reason is that the differences are so small, the plotting software doesn't put any labels there (on a log axis it only places labels on multiples of 10, with sublabels on x2 and x5). The difference between the left and right side of the plot is only 15% for both 16-bit audio and 24-bit audio, meaning preset -0 decodes about 15% faster than preset -8.
Music: sounds arranged such that they construct feelings.

Re: Retune of FLAC compression levels

Reply #2 –
A quick test on my CDDA material shows a 26.5% decrease in encoding time and a 0.07% increase in filesize for flac -8 with this patch applied.

Re: Retune of FLAC compression levels

Reply #3 –
Sorry a bit of a newb question but does FLAC need to have 8 different preset levels?

Could it be broken down to a more simple scheme?

Preset Fast = FLAC preset 2
Preset Medium = FLAC preset 5
Preset Best = FLAC  preset 8

or

FLAC preset 0,1 get mapped to Flac preset 2
FLAC preset 3,4 get mapped to Flac preset 5
FLAC preset 6,7 get mapped to Flac preset 8
Who are you and how did you get in here ?
I'm a locksmith, I'm a locksmith.

Re: Retune of FLAC compression levels

Reply #4 –
Changing the number of presets would break old FLAC frontends, which ktf probably doesn't want to do. It's cleaner to rebalance the existing nine levels than to change the number of levels.

Re: Retune of FLAC compression levels

Reply #5 –
Quote from: A_Man_Eating_Duck on
Sorry a bit of a newb question but does FLAC need to have 8 different preset levels?
It probably doesn't need to, but it seems most people like some choice. TAK has 15 presets, WavPack has 4 presets and 6 different modes of extra processing (which can be combined in any way), Monkey's audio has 5 presets and OptimFROG has 12. So, FLAC isn't alone in this regard.

Quote from: Ziengaze on
A quick test on my CDDA material shows a 26.5% decrease in encoding time and a 0.07% increase in filesize for flac -8 with this patch applied.
Did you compare against unpatched current git, the latest release (1.4.2) or something else?
Music: sounds arranged such that they construct feelings.

Re: Retune of FLAC compression levels

Reply #6 –
I have to say that I rather think that this is a solution looking for a problem. The majority of users will simply use the default preset and most won't venture beyond that. For those few enthusiasts who wish to delve into the depths of what is possible, no doubt they enjoy playing with all the various current options. I'm not saying it's a bad idea, I'm just suggesting that it's not necessarily relevant.

Re: Retune of FLAC compression levels

Reply #7 –
Can those who run tests please let us know their CPUs?

Asking because there are known differences between Intel and AMD processors. For example, in ktf's tests (using an 1.3 version), -3 seems to be the fastest decoding, and that seems to be an AMD specific thing. More differences got known as HA users tested 1.4.x performances.

Short "just a couple of files" testing here, will do more when I am back to my test corpus at my computer (which BTW is now running a re-encode to 1.4.2 on a hard drive which is nearly full - that's when 50 cents of hard drive space is worth a bit more):

* Going full -l32 on high sampling rates:
My first reaction was, this will shock those who use "-8e", I checked one high-resolution file that now takes 4x the time of 1.4.2. That is even if the reduced number of apodization functions speeds up -8el12 to half the time of 1.4.2.

But the thing is, -e still has a mission on some high sampling rates, that is the problem. At least it had on 1.4.2, I'll test this one.
If there is a way to get a more clever semi-exhausting that tries the right thing, then maybe change the meaning of "-e"? (And introduce a "-E" for "YES I have the patience".)

* -0 with stereo decorrelation:
You probably thought over the following scenario, but ... what if somebody runs an on-the-fly encoding with a decoder that can only do dual mono? (I guess they won't use a new flac.exe anyway.) And 16 bit word length (cf. the limitations you put on 32-bit signals) - would anyone want dual mono as a point in itself? If so, they might specifically have chosen a preset that does dual mono.
If - and that's a big "if" - there is a point to have a dual mono preset, then what else than -0? 

* Going full 4096 is probably not a bad idea. Will test. Did test on 1.4.2, with some slight surprises. One was a penalty for 4096 over 3172.

(* Will test: Old -6 as new -5? I've argued that -6 wasn't much useful except for those who wanted the lightweight decoding of -l 8. Will test.)

* A_Man_Eating_Duck certainly has a point. I would ask how many use anything but -0, -5, -8, -8withsomeadditionaltweaks, but actually Bandcamp uses -6 ... which I've said bad things about. But hey, as long as there are -1/-2/-3/-4/-6/-7 around already, it doesn't hurt to give them some sensible meaning. (If that's a solution looking for a problem, then the problem was set up twenty years ago.)

Re: Retune of FLAC compression levels

Reply #8 –
Quote from: ktf on
Did you compare against unpatched current git, the latest release (1.4.2) or something else?
unpatched git master 9ee21a0
edit: on a Ryzen 1800

Re: Retune of FLAC compression levels

Reply #9 –
Quote from: Aleron Ives on
Changing the number of presets would break old FLAC frontends, which ktf probably doesn't want to do. It's cleaner to rebalance the existing nine levels than to change the number of levels.
Using the second method wouldn't break presets, it's just mean if preset 1 is used it will actually encode with preset 2 or if preset 6 is used it will encode with preset 8. It also aligns to the graphs as well.

This would mean there are less presets that you need to adjust the tuning for.
Who are you and how did you get in here ?
I'm a locksmith, I'm a locksmith.

Re: Retune of FLAC compression levels

Reply #10 –
Quote from: john33 on
I have to say that I rather think that this is a solution looking for a problem. The majority of users will simply use the default preset and most won't venture beyond that. For those few enthusiasts who wish to delve into the depths of what is possible, no doubt they enjoy playing with all the various current options. I'm not saying it's a bad idea, I'm just suggesting that it's not necessarily relevant.

Have to agree with john on this.

Personally..... All this hunting for rather minimal gains of this and that...... it's a waste of time imo. Things as it is, works just fine.

Re: Retune of FLAC compression levels

Reply #11 –
Me too. Though I think preset 2 should be retuned so that, if possible, its encoding speed and compression gain end up somewhere between those of presets 1 and 3.

Chris
If I don't reply to your reply, it means I agree with you.

Re: Retune of FLAC compression levels

Reply #12 –
Quote from: NetRanger on
Personally..... All this hunting for rather minimal gains of this and that...... it's a waste of time imo. Things as it is, works just fine.
I don't think ~ 0.3% improvement on average (-8 on hi-res material) with no slowdown is in any way minimal? Also -0 improves more than 1% with no slowdown. Lots of topic here discuss -p and -e, and those usually result in gains far less, at the cost of quite some slowdown.

I agree it might not be particularly necessary, but I don't think it is a waste of time.
Music: sounds arranged such that they construct feelings.

Re: Retune of FLAC compression levels

Reply #13 –
If the goal is to keep the existing 9 presets, I think I would do something like this:

0-2: Same as existing presets, but change -b to 2304.
6: Same as existing preset but increase -l to 10.

Other presets remain unchanged but for > 48k materials increase preset 7 to -l16 and preset 8 to -l20, and -b8192 for >= 176.4k materials.

-l is in general quite effective in reducing file size but higher values also reduce decoding speed, especially for higher sample rates like DXD.

As for APE, I never think "insane" is useful.

Re: Retune of FLAC compression levels

Reply #14 –
Hi all.

Instead of retuning presets it would be mutch more helpful to let the - commandline - encoder find the optimal "tune" for each machine, according to user's choices (speed, compression or both).

Hope that inspires !
Hybrid Multimedia Production Suite will be a platform-indipendent open source suite for advanced audio/video contents production.
Official git: https://www.forart.it/HyMPS/

Re: Retune of FLAC compression levels

Reply #15 –
For standard resolution, these retunings are not for size - apparently! - so that part of the criticism should be revised ... at least to target the setting that one should think twice before altering, namely the default. The proposed change will make -5 slower (no big deal if you compare new -5 on new hardware to fifteen years ago!), but I would scratch my head for a while over slowing down the default by some forty percent. Also it becomes slower than new -7.
It is also a stretch to re-assign default to "new -4 = old -5", but I think it should be considered. That would also keep the "logic" of these triplets 012, 345, 678, where the highest of each is for those who are willing to wait. Default would be the "middle of the middle".


-0 to -2 serve their very special purposes, namely to enforce fixed predictors.  Whether or not that is actually fastest, there are compatibility/interoperability reasons to keep them thay way.
Also in the IETF draft, ktf&co will recommend that whenever one wants maximum compatibility with decoders that suck, then stick to 1152 or 4096 samples in a block.  So while 2304 seems to improve, there is a case against it - at least for -0.
Going 4096 but keeping the -r will increase the partition size from 147 to 512, whether or not there is any reason to change that ... I see ktf proposes to keep -r3 for -0, but for -1 go to -r5 (a partition becomes 128 samples, that is close to 147).
So given the purpose of -0 to -2, then
-0 for fast encoding in a failsafe way.  (Which is why I question the stereo decorrelation.)
-2 to make the "best" out of fixed predictors.  From the charts it looks like the return to effort is very small, but it does serve that particular purpose.

Re: Retune of FLAC compression levels

Reply #16 –
Quote from: ktf on
Quote from: NetRanger on
Personally..... All this hunting for rather minimal gains of this and that...... it's a waste of time imo. Things as it is, works just fine.
I don't think ~ 0.3% improvement on average (-8 on hi-res material) with no slowdown is in any way minimal? Also -0 improves more than 1% with no slowdown. Lots of topic here discuss -p and -e, and those usually result in gains far less, at the cost of quite some slowdown.

I agree it might not be particularly necessary, but I don't think it is a waste of time.

I've personally never bothered with testing many switches for improving encoding speed and the size of the output. If the encoder/preset work as it should, then it's fine with me. Only used the preset -6 or -8 on my encodes for the last 4-5 years+

Re: Retune of FLAC compression levels

Reply #17 –
Quote from: NetRanger on
If the encoder/preset work as it should, then it's fine with me. Only used the preset -6 or -8 on my encodes for the last 4-5 years+
That is the reason why the developers set reasonable presets, so you don't have to tweak settings further ;)

(Can I ask why -6? Because it was the default at version 0.x? Not long since I saw someone still making that assumption from memory.)

Re: Retune of FLAC compression levels

Reply #18 –
Quote from: Porcus on
Quote from: NetRanger on
If the encoder/preset work as it should, then it's fine with me. Only used the preset -6 or -8 on my encodes for the last 4-5 years+
That is the reason why the developers set reasonable presets, so you don't have to tweak settings further ;)

(Can I ask why -6? Because it was the default at version 0.x? Not long since I saw someone still making that assumption from memory.)

To be honest...... i don't know really why i use -6. Could just be something i set at one time and then i've just kept it. But i would say i use -8 on 95% of my encodes these days.

Re: Retune of FLAC compression levels

Reply #19 –
A comparison between the 1.4.2 release from October and the retune version posted here.
~12GB of 44.1k/16bit files with an Intel 10600K on Windows (through Foobar).

version       enc. speed    size
1.4.2 -8       1310x         11,792,799
1.4.2 -7       1677x         11,803,842
retune -8     1794x         11,805,269

I ran every test a couple of times, although encoding speed was quite consistent.
Decoding speeds varied a bit more between runs, but I didn't see any significant difference there between the versions. And I just used Foobar's decoding speed test, with whatever flac version it uses for that.

Re: Retune of FLAC compression levels

Reply #20 –
Quote from: Brand on
version       enc. speed    size
1.4.2 -8       1310x         11,792,799
1.4.2 -7       1677x         11,803,842
retune -8     1794x         11,805,269
Similar results here with CD material. I don't think -8 needs speed ups in trade for compression.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Retune of FLAC compression levels

Reply #21 –
I would personally go even further.

Map all presets from 0-5 together into single preset. Use preset 5 as base and optimize it as much as you can without sacrificing speed.
Map all presets from 6-8 into single preset. Use preset 8 as base and optimize it as much as you can for higher compression while still keeping sane level encoding speed.

Edit: Before you start screaming at me, have you ever seen anybody using presets other than 5 (default) and 8 (best)?
gold plated toslink fan

Re: Retune of FLAC compression levels

Reply #22 –
Maybe we could keep the numbered levels as-is but also add new, non-numbered presets in addition to those levels so that anyone interested can use them, and people attached to their old presets for whatever reason won’t miss them.


The deprecated -completelyimpracticalcompression (or whatever it was, something along those lines) comes to mind

Re: Retune of FLAC compression levels

Reply #23 –
Quote from: Markuza97 on
Edit: Before you start screaming at me, have you ever seen anybody using presets other than 5 (default) and 8 (best)?
Bandcamp uses -6.
And I am pretty sure that -0 is in use, for those very particular reasons.

Re: Retune of FLAC compression levels

Reply #24 –
Quote from: Markuza97 on
Edit: Before you start screaming at me, have you ever seen anybody using presets other than 5 (default) and 8 (best)?

The difference between presets -0 and -8 are probably about 3% more compression at the cost of 3-5x longer encoding time.  People with older systems might not think the increase in time is worth it.