Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: FLAC v1.4.0 (Release) (Read 30579 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: FLAC v1.4.0 (Release)

Reply #25 –
I'm running a test on my files comparing v1.4.0 -8p and -8pe. -8p is already done but -8pe will take a total of 40hours to complete.

I'll post those results in this thread once done.
Who are you and how did you get in here ?
I'm a locksmith, I'm a locksmith.

Re: FLAC v1.4.0 (Release)

Reply #26 –
Since you are taking the big power bill, I shall not post my -8pe spoiler until then? ;-)

Meanwhile, the music I used for the testing, described here: https://hydrogenaud.io/index.php/topic,122960.0.html (to be updated)
Music might also be interesting ... y'know?  ;)

Sizes, compressed with 1.4.0 at default -5:
4 002 927 468 bytes of classical music, bitrate 598. Can be compressed to 3 982 303 322 at -8ep
4 000 248 235 bytes of heavier rock, bitrate, bitrate 971. Compressible to 3 978 679 443 at -8ep
4 030 323 496 bytes of "other" music, bitrate 704. Compressible to 3 996 086 171 at -8ep. This is the section that shrinks the most!

Then the "other" includes "Sunflower", this year's Ukraine charity EP from Dream Theater keyboardist Jordan Rudess. From a metal band, doing Chopin, one of his own compositions and a one minute rendition of Ukraine's national anthem, arguably as a tie-breaking criterion put in the "other" to get all three just across the 4 000 000 000 mark :-o That EP turned out to become two percent smaller in 1.4.0 than in 1.3.4, consistently across settings. Hm.
One file got missing from the initial run, so the above total times are, say, 1/37th too low. I won't bother to correct those times, it won't make big differences. The number of exceptions below are corrected.


So here is what happens. All "savings" etc are relative to the line immediately above it:
1.4.0 is compared to 1.3.4 at same setting
1.3.4 is compared to 1.4.0 at "lower setting". That lower setting is faster. And 1.4.0 wins on the most interesting presets.

1.3.4 @ -4.
1.4.0 @ -4. All sections smaller. 3 exceptions that make for larger files (Laibach: NATO 0.13 percent bigger, then Armand Van Helden and Kiss). At the other end: The Jordan Rudess Ukraine charity EP s(h)aves 2.35 percent. That EP benefits from 1.4.0 all the way.
1.3.4 @ -5. Worse in all three sections, though varying a bit. From .6% smaller to >2% bigger.
1.4.0 @ -5. 1.4.0 improves every file (from near-zero to two percent).
1.3.4 @ -6: classical section bigger than 1.4.0 at -5. The other two sections smaller.
1.4.0 @ -6: All files but one are smaller than 1.3.4. 
1.3.4 @ -7: classical section bigger than 1.4.0 at -6. The other two sections smaller.
1.4.0 @ -7: All files but one are smaller than 1.3.4. 
1.3.4 @ -8: All three sections (and all files but eight) lose against 1.4.0 @ -7. The "biggest exception" is 0.06 percent for Laibach: NATO
1.4.0 @ -8: 1.4.0 improves every file (from near-zero to two percent).
1.3.4 @ -8p: All three sections (and all files but twelve) lose against 1.4.0 @ -7; closest is the heavy corpus that is within 0.11%; biggest exception is 0.08 percent for Bach's harpsichords.
Also every section loses against 1.4.0 -7.
1.4.0 @ -8p: Every file improves over 1.3.4.  Still a couple of files improve one percent (Wovenhand: Black of the Ink full-length album) or even two for Jordan Rudess.
1.3.4 @ -8pe: All three sections (and all files but six) lose against 1.4.0 @ -8p; closest is the heavier section that is within 0.08%; biggest "single album exception" is 0.012 percent, yet again for Bach's harpsichords.
All three sections also lose against 1.4.0 @ -8. It wins at compression against 1.4.0 @ -7, except the classical section.
1.4.0 @ -8pe: Every file improves over 1.3.4.  Still Rudess saves 2.1 percent and Wovenhand 1.02.
-8pe took "only" three minutes more for 1.4.0.  Still short of six hours. 


So it seems that, the higher precision by and large makes for going down a preset and still saving space.
For classical music there are a couple of exceptions, but how many use 1.3.4 at -6 (which isn't a good setting!) and will consider going -5 when 1.4.0 arrives? And those who use 1.3.4 at -7 should hope they have computing power to stay at -7, rather than going to -6.

Re: FLAC v1.4.0 (Release)

Reply #28 –
cool, so I just saved 22,5 GB disk space by reencoding everything with 1.4.0 at -8ep (because why use a lower setting??)

Re: FLAC v1.4.0 (Release)

Reply #29 –
Quote from: SigHunter on
cool, so I just saved 22,5 GB disk space by reencoding everything with 1.4.0 at -8ep (because why use a lower setting??)
Yep, if you have 17 TB of CD rips encoded at -8ep. With -8ep taking six hours for twelve GB, that would translate to a year's reencoding time - YMMV, but I doubt you "just" did that  :))

We are not talking miracles. This is evolution without a leading "r".

(I actually didn't post improvement yet, but in my corpus, going 1.3.4 @ -8ep to 1.4.0 @ same, will save 0.1322035953 percent.)

Re: FLAC v1.4.0 (Release)

Reply #30 –
To be fair, if the advantage Flake used to have over FLAC (the encoder, not the format (obviously)) mattered to you enough to prefer using it, this should count as a "significant" improvement in some capacity, as the autocorrelation precision increase and additional compression improvements essentially remove that advantage.

(At least, as far as I know.)

Re: FLAC v1.4.0 (Release)

Reply #31 –
Congrats! Major point releases don’t happen every day you know. Million thanks to everyone who put their time into this and make FLAC what it is today.

Re: FLAC v1.4.0 (Release)

Reply #32 –
edit: what I just posted must have a grave error. Will re-run once and repost.

Re: FLAC v1.4.0 (Release)

Reply #33 –
Turns out this CPU is throttling in a way that is likely good for the purpose, but not so good for timing: starting cold and running two tasks where the temperature sensor starts throttling midway first task, and you get the idea. Anyway, a couple of re-runs later, the following timings should be reliable at ballpark level, and the results are so clear that I don't need more.

Also, new timings are done on all 38 and, rather than re-encoding I encoded .wav files.

Quote from: doccolinni on
To be fair, if the advantage Flake used to have over FLAC (the encoder, not the format (obviously)) mattered to you enough to prefer using it, this should count as a "significant" improvement in some capacity, as the autocorrelation precision increase and additional compression improvements essentially remove that advantage.

(At least, as far as I know.)

1.4.0 at -8 beats Flake at -8 in my test, at around half the encoding time.
Even if the heavier corpus makes for 0.01 percent bigger files, I think that conclusion is safe: The time difference from flake -8 up to flac -8p is about the same as down to flac -8, but saves you five to six times as much on that sub-corpus.  So if you are willing to go flac -8 to flake -8, you should be willing to go to flac -8p.

Some comparisons are given in the table, listed in order of decreasing total size, sizes are total = classical + heavy + other, and then time at the end.
Also I put WavPack into the table (not broken into genre), using only the -h mode [note at end to the very few HA'ers who don't know WavPack] and without double checking times. I admit to a fancy for -x4 for what it can do to high resolution, though it does not achieve the same on CDDA (where WavPack with and without the -x method are already well optimized, I guess) - while I think -x6 falls in the rightmost column although sometimes we cannot help ourselves using -8pe either.
 
codec/setting: total size= classical + heavier + othertime
flaccl -8 :  11988816395= 3991332277 + 3985599780 + 4011884338 220 sec
1.4.0 -7  : 11976656017= 3988461592 + 3983601785 + 4004592640 380 sec
WavPack -h:  11972040984 514 sec
flake -8  : 11970638661= 3987723686 + 3981441186 + 4001473789 1135 sec
1.4.0 -8  : 11969604531= 3986775555 + 3981858510 + 4000970466 551 sec
1.4.0 -8p : 11959947654= 3983291686 + 3979122460 + 3997533508 half an hour
1.4.0 -8ep: 11957068936= 3982303322 + 3978679443 + 3996086171 nearly six hoursNon-commonsense
flaccl -11:  11952388265= 3975139072 + 3975317627 + 4001931566 271 secNon-subset
flake  -11: 11929816035= 3972574928 + 3970861219 + 3986379888 1782 secNon-subset
WavPack -hx 11913860416833 sec
WavPack -hx411860280264two hrs and a half
WavPack -hx6 11848338084nearly five hours
Now if subset can be sacrificed, then damn this new computer has a GPU ... a bit surprised at what happens at the "other" section.

But what you point out is part of what I have been testing: "if you were willing to choose <x>, then you should be willing to choose 1.4.0 at <setting>". Reasoning as if the user makes deliberate choices out of CPU cost / size benefits as I measure them - most flawed assumptions, of course:
* users who run flac.exe at default most often don't consider -4/-6/-7/-8. (Cost/benefit argument: default is good enough, benefits are not worth the hassle of nerding yourself up on alternative settings.)
* users who run at -8 or higher most often don't consider -6 or -7.
* and even if they were, my audio and my hardware are not theirs


WavPack note for those any HA readers who don't know it:
First on FLAC: FLAC was developed for ultra-light CPU footprint at decoding - and nothing decodes as ultra-light as FLAC!
By the time FLAC was born, WavPack was already around - it can achieve smaller files that require more juice to decode, and mid-noughties, that could be noticeable in terms of battery life on already-then-low-powered portable players. For more on performance figures, see ktf's tests at http://www.audiograaf.nl/losslesstest  .
WavPack's "-x" extra processing settings are - alike heavier FLAC options, by and large - costly at encoding time, but not decoding time. -x (=-x1) is considered most reasonable (/"essential", depending on whom you ask), while -x4 to -x6 are rightfully considered very slow. I guessed flac -8pe could use a sanity check against those, so I included them.

Re: FLAC v1.4.0 (Release)

Reply #34 –
I submit the following compiles for testing. They are all Intel 19.2 compiles and the win32 compiles should run on XP as previously. I don't believe there is any speed difference of note, just simply using the latest compiler. :)

www.rarewares.org/files/lossless/flac-1.4.0-x64-Intel-19.2.zip
www.rarewares.org/files/lossless/flac-1.4.0-x86-Intel-19.2.zip
www.rarewares.org/files/lossless/flac_dll-1.4.0-x64-Intel-19.2.zip
www.rarewares.org/files/lossless/flac_dll-1.4.0-x86-Intel-19.2.zip

Re: FLAC v1.4.0 (Release)

Reply #35 –
Can anyone tell me what the differences between -8, -8e, -8p and -8ep are?

I didn't find a documentation concerning these encoding options!

...but by testing I did already learn something about the file sizes that are produced by these options:

"-8" > "-8 -e" = "-8e" > -"8p" > "-8ep" = "-8ep -e" = -8p -e"

Thus, I'll use now "-8ep -P 2048" as "extra command line options" with FLAC Frontend instead of "-e -P 2048" , even tough this saved only 322 KB on 364 MB on my test files (I came down from 382.372.861 bytes to 382.702.491 bytes).

Re: FLAC v1.4.0 (Release)

Reply #37 –
Another Question:

Can values for -A subdivide_tukey(....) higher than 3 cause compatibility problems?

I made some tests with "-8ep -A subdivide_tukey(9) -P 2048" as "extra command line options" with FLAC Frontend and this saved more or less 10 kb on every song.

But unfortunately using these options take a lot of time and it seems that flac.exe uses only one cpu core, which seems to be the main cause for the extremely long encoding time.

Thus, I'd like to use the "-A subdivide_tukey(9)" option only if there's a way to let flac.exe use more than one cpu core and if this option does not cause any compatibility problems.

Re: FLAC v1.4.0 (Release)

Reply #38 –
Quote from: rennie on
Can anyone tell me what the differences between -8, -8e, -8p and -8ep are?

I'll try. A bit of background first:
So FLAC was designed for light decoding, and therefore the encoded data aren't packed in a very complex format. Also, FLAC decoding depends very little upon encoding parameters.
Unlike e.g. Monkey's, where using a higher mode means you use a more complex and more decoding time-consuming algorithm, FLAC instead "tries many simple ones" and chooses the best. But it doesn't "brute force optimize" a lot: instead it has a model where it guesstimates which one will be best, and then it spends time packing that one.

Therefore also the answer to your compatibility question is that ... don't worry. (If you ever see something that requires the "--lax" option, that is a different thing: then decoders can start to object.)

So the presets. The presets -0 to -8 are just names for different parameters to fire at it. Nothing says that "-6" is twice as good in any way as "-3".  The difference between -7 and -8 is that the latter only tries more windowing functions, hoping to find - for each 4096 samples - one that does better than the -7 choice. If it is better for a small part of the signal, it is selected.

Part of the "guesstimation" can be overridden with -e. With -e, part of the model selection will be done by calculating everything and then selecting the best. Think that you have two ways x or y to drive from A to B: you look at a rough map and guess, it looks like x is shorter, you drive "x" and measure it and declare that to be the distance. But your map could have been too rough, so to be sure you can go back and measure y exactly as well. Oh and there was a possible shortcut there ... possible? Only one way to find out.
-e exhausts several options exactly. Takes time.

-p does something similar on a different aspect of the modelling.

The FLAC encoder has become so good by now, that the time spent on -e is not worth it - on CDDA. On high resolution material, -e makes more of an impact - but even then there are other time-consuming procedures that can be added, that seem to work better (at least on my test corpus). The -A subdivide_tukey(<high number>) makes for trying more different apodization functions (= try more statistical filters, you will have a greater chance of finding something that fits better).

-p seems to be harder to beat. When you are at -8 and want more, then -8p might still be better than more apodization functions. (After all, -8 seems better than -7p, and -8 does indeed use more functions than -7.)

Documentation? It isn't explained in detail at https://xiph.org/flac/documentation_tools_flac.html , no.  And the thing is, we don't really know precisely why this works better than that. There are good reason that this measure improves compression, but the reason why -p works better than -e is ... well, we have measured it on real-world signals and it turns out it does so.

As for your -8ep = -8ep -e, that is because the options can be concatenated.  FLAC reads "-8ep" as -8 -e -p.

Order isn't arbitrary though.  In your -8ep -A subdivide_tukey(9) it first selects -8 (which implies some "-A"!), then switches on the -e and -p brute forces - and then you give a "-A" that overrides the one initially selected by -8. Note that by choosing such a high argument in the subdivide_tukey, you also have a lot to brute-force-check, so this.is.slow and likely only to play with for testing.
For testing you can try to compare the following:
-8
-8e
-8 -A subdivide_tukey(9)
-8e -A subdivide_tukey(9)
-8p
-8pe
-8p -A subdivide_tukey(9)
-8pe -A subdivide_tukey(9)
... and I have a hunch that is the order of sizes, but not the order of times. And that anything but -8 and -8p are "not worth it".


"-P 2048" has nothing to do with the compression of the audio. -P controls is the padding - the amount of empty space set aside at the end of the tags section, before the audio starts.
The default is 8096 bytes up to 20 minutes (why larger for larger? Because those are often entire albums and have more tags), and by setting it to 2048 bytes, you saved 6 kilobytes.
But, padding is there for a purpose. If you make a small tag edit, it will write into the padding - but if you write a bigger one, so that tagging needs more space than your padding section has set aside, then you have to wait for the entire file being written anew. Reducing the -P makes for more frequent full file rewrites. Then on the other hand, 8096 is hardly enough for album art. It maybe was when FLAC was released.


Was this about it?

Re: FLAC v1.4.0 (Release)

Reply #39 –
1.3.4 - flac.exe" -s --ignore-chunk-sizes -8p - -o %d
Code: [Select]
Total size : 393 GB (422 709 930 380 bytes)
Duration : 5wk 6d 0:45:59.793
Sample rate : 44100 Hz (97.3%); 48000 Hz (2.7%)
Channels : 2
Bits per sample : 16 (96.8%); 24 (3.2%)
Avg. bitrate : 954 kbps
Codec : FLAC
Encoding : lossless
Tool : reference libFLAC 1.3.4 20220220
Embedded cuesheet : no
1.4.0 - flac.exe" -s --ignore-chunk-sizes -8p - -o %d
Code: [Select]
Total size : 393 GB (422 292 414 045 bytes)
Duration : 5wk 6d 0:45:59.793
Sample rate : 44100 Hz (97.3%); 48000 Hz (2.7%)
Channels : 2
Bits per sample : 16 (96.8%); 24 (3.2%)
Avg. bitrate : 953 kbps
Codec : FLAC
Encoding : lossless
Tool : reference libFLAC 1.4.0 20220909
Embedded cuesheet : no
1.4.0 - flac.exe" -s --ignore-chunk-sizes -8pe - -o %d
Code: [Select]
Total size : 393 GB (422 175 699 576 bytes)
Duration : 5wk 6d 0:45:59.793
Sample rate : 44100 Hz (97.3%); 48000 Hz (2.7%)
Channels : 2
Bits per sample : 16 (96.8%); 24 (3.2%)
Avg. bitrate : 952 kbps
Codec : FLAC
Encoding : lossless
Tool : reference libFLAC 1.4.0 20220909
Embedded cuesheet : no

Results
1.3.4 -8p  = 403127.60MB
1.4.0 -8p  = 402729.43MB (Savings of 398.17MB over 1.3.4 -8p)
1.4.0 -8pe = 402618.12MB (Savings of 509.48MB over 1.34 -8p, Savings of 111.31MB over 1.4.0 -8p)
Who are you and how did you get in here ?
I'm a locksmith, I'm a locksmith.

Re: FLAC v1.4.0 (Release)

Reply #40 –
So, savings are in 0,1% range, no? What kinds of music are in your collection?
Error 404; signature server not available.

Re: FLAC v1.4.0 (Release)

Reply #41 –
It's mainly rock and metal.
Who are you and how did you get in here ?
I'm a locksmith, I'm a locksmith.

Re: FLAC v1.4.0 (Release)

Reply #42 –
Quote from: A_Man_Eating_Duck on
Results
1.3.4 -8p  = 403127.60MB
1.4.0 -8p  = 402729.43MB (Savings of 398.17MB over 1.3.4 -8p)
1.4.0 -8pe = 402618.12MB (Savings of 509.48MB over 1.34 -8p, Savings of 111.31MB over 1.4.0 -8p)

509MB storage space savings sounds decent until you see how large the general data is, which is 422.x GB in size, at which point that 509MB savings completely loses it's appeal.

but as we all basically know... differences in file size/compression speed/decompression speed/battery life (and even conversion time from FLAC to MP3/Opus/AAC etc) are probably so small it really does not matter what one chooses as I think the difference in lossy bit rates is worth nit-picking on more than FLAC. so I figure with FLAC just stick to the default (-5) or maybe -8 if someone wants to save a little storage space for good measure as beyond that the amount of time to re-compress is simply too high for the barely any storage space savings, unless of course someone is obsessed with as minimal of file size as possible on FLAC.

still, it's nice to see a small improvement with current FLAC version.

but thanks for the 'quick' tests there anyways ;)
For music I suggest (using Foobar2000)... MP3 (LAME) @ V5 (130kbps). NOTE: using on AGPTEK-U3 as of Mar 18th 2021. I use 'fatsort' (on Linux) so MP3's are listed in proper order on AGPTEK-U3.

Re: FLAC v1.4.0 (Release)

Reply #43 –
No, differences are not that much, especially at that end when we are closing in on what a computationally simple data format like FLAC can do.

On high resolution material, 1.3.4 wasn't quite there.  Tested here. At -8, the 1.4.0 encode saved 3.37 percent. Then on a 400 GB corpus the saving would be 13 GB and not half a GB. That is quite something to notice.

Reasons to do this upgrade, include
* to iron out wrinkles that sometimes make for embarrassing results
* for the geeky art of engineering
* for sports - finally found what made Flake perform so well, now the reference implementation is more on top of its own game

Worth it to encode? Maybe. Worth it to re-encode? Oh, that's a different question.

Re: FLAC v1.4.0 (Release)

Reply #44 –
Quote from: A_Man_Eating_Duck on
Savings of 111.31MB over 1.4.0 -8p
S(h)avings between 0.025 percent and 0.03 percent, same as in my test.
I tested the same with 1.3.4. Then the impact was > 0.1 percent.  (With my 12 GB corpus, the additional "e" at the end of -8p would save 13 MB for 1.3.4, and only 3 MB now.)

-8pe would spend ages on nearly nothing. Now it spends ages on a quarter of nearly nothing, because the basic algorithm has improved to already catch what the "e" can accomplish.

This week's news:
Queen -e has passed away. She had her time.

Re: FLAC v1.4.0 (Release)

Reply #45 –
Quote from: john33 on
I submit the following compiles for testing. They are all Intel 19.2 compiles and the win32 compiles should run on XP as previously. I don't believe there is any speed difference of note, just simply using the latest compiler. :)

Thnx John :)

Re: FLAC v1.4.0 (Release)

Reply #46 –
Any idea what minimum CPU architecture these v1.4.0 binaries were built for?

Re: FLAC v1.4.0 (Release)

Reply #47 –
Which, specifically? Quite a few have been dropped in this thread.
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.0 (Release)

Reply #48 –
Quote from: sundance on
Any idea what minimum CPU architecture these v1.4.0 binaries were built for?
win32 - XP, x64 - 8.1.

Edit: Oops, I assumed OS! CPU should be anything that supports those OSs.

Re: FLAC v1.4.0 (Release)

Reply #49 –
Quote from: ktf on
Which, specifically? Quite a few have been dropped in this thread.
I tested these four x64 binaries: the "official" release, both of the latest Netranger (clang and gcc) and John's icl compile. Since I'm more interested in speed than shaving some 0.0xx% off the flac size, I only tested compression levels -7 and -8. The speed variation on my 5 years old CPU i7-7700 were minimal (at -7: 37.84s, 37.18s, 37.94s, 37.71s). So I assumed that similar architecture settings were used to run on ALL CPUs.
Some time ago HA users enzo and case were kind enough to provide v1.3.3 x64-binaries, that were noticeably faster than the official compile and one of the optimisations were afaik to demand at least a Nehalem CPU architecture (12 years old now). I still use these binaries today since I never had a faster one (at -7: 24,96s, at -8: 35,99s) -> they are still faster at -8 than any of the 1.4.0 binaries at -7, but because of optimisations in 1.4.0 result in almost the same flac file size.
So I wonder if 1.4.0 would also benefit from a more restrictive CPU support to enable more instruction set extensions (do we still use CPUs prior to Core-i?)
Just my 2cents