HydrogenAudio

FLAC v1.4.0 have been released. :)

https://github.com/xiph/flac/releases/tag/1.4.0

FLAC v.1.4.0 compiles now at Rarewares. :)

I didn't initially get the windows binaries on github to work via foobar2000's converter, and not in EAC either. Both just threw some errors. Only when I ran it from the commandline I got a dialog/popup telling that libFLAC.dll is missing. It's included in the zip, but I didn't think of extracting it since no previous version has needed it.
So for anyone running into mysterious errors, there's your solution to get it working.
I'll have to test the rarewares build too, now that it's out :)

john33, your x64 binaries seem to report themselves as 1.3.4 still, and write "reference libFLAC 1.3.4 20220909" as the encoder.

I didn't initially get the windows binaries on github to work via foobar2000's converter, and not in EAC either.

I think the point of the "4" in "1.4.0" is that it breaks some interoperability?

I'll summarize the changes here

- Improved compression. The difference is only small for CD audio, but can be very large (> 10%) for low-passed material like most "high-resolution" downloads that do not contain any material above 20kHz but are sampled at 96kHz or even higher.
- Presets 0, 1 and 2 got a little faster, 3, through 8 got a little slower.
- Many bug fixes of problems found by fuzzing. This is probably not something most users will notice, but it tries to ensure hard-to-find bugs will not creep in like this one (https://hydrogenaud.io/index.php/topic,121349.0.html).
- Encoding and decoding of 32 bps audio is now possible (integer, not float)
- Samplerates above 655'350Hz are now supported, up to 1'048'575Hz

I think the point of the "4" in "1.4.0" is that it breaks some interoperability?

I think it's that the flac.exe in the official release requires libFLAC.dll to be present or it won't run; they aren't static compiles like previous releases were.

john33, your x64 binaries seem to report themselves as 1.3.4 still, and write "reference libFLAC 1.3.4 20220909" as the encoder.

I think @john33 forgot to change the version in the Visual Studio files

I think the point of the "4" in "1.4.0" is that it breaks some interoperability?

That's what I first thought, but it seems to work fine once I put libFLAC.dll in the same dir as flac.exe

EZ CD Audio Converter 10.2 includes FLAC 1.4.0

https://www.poikosoft.com

FLAC v1.4.0 (Release)
Built on September 09, 2022

GCC v12.2.0 compile attached.

32bit build available on github asks for libgcc_s_dw2-1.dll. I'm on windows 7.

Yes, I see now libFLAC++ also has a unwanted dependency. I'll fix and replace

edit: I've replaced on github, will ask to have the Xiph download replaced

edit2: updated once again, now libFLAC++.dll also doesn't have unwanted dependencies

Quote from: Heliologue on 2022-09-09 19:02:39

john33, your x64 binaries seem to report themselves as 1.3.4 still, and write "reference libFLAC 1.3.4 20220909" as the encoder.

I think @john33 forgot to change the version in the Visual Studio files

@ktf is quite correct!! Apologies to all. Corrected in current downloads.

Clang compile from NetRanger is fastest on my machine (Core i3 3245, 32 bit Windows 7)
Compression -8
Rarewares and GCC from NetRanger ~110x
Official github ~120x
Clang from NetRanger ~155x

A v1.3.4 managed to snuck into my CLANG 1.4.0 x64 archive in my previous post.

Correct archive contents here

FLAC v1.4.0 (Release)
Built on September 09, 2022, CLANG 14.0.6

@ktf is quite correct!! Apologies to all. Corrected in current downloads.

File libFLAC_dynamic.dll inside flac_dll-1.4.0-x64.zip still reports version "reference libFLAC 1.3.4 20220909" when used from third party software like CueTools.

On my i9-10850K

flac 1.3.4 clang NetRanger (https://hydrogenaud.io/index.php/topic,122179.msg1009962.html#msg1009962)
Total encoding time: 7:03.515, 76.59x realtime

flac 1.4.0 clang NetRanger (https://hydrogenaud.io/index.php/topic,122949.msg1015344.html#msg1015344)
Total encoding time: 10:05.329, 53.58x realtime

flac 1.4.0 gcc12.2 NetRanger (https://hydrogenaud.io/index.php/topic,122949.msg1015320.html#msg1015320)
Total encoding time: 5:52.078, 92.13x realtime

I'll stick with the 1.4.0 GCC Version ;-)

Quote from: john33 on 2022-09-09 21:38:44

@ktf is quite correct!! Apologies to all. Corrected in current downloads.

File libFLAC_dynamic.dll inside flac_dll-1.4.0-x64.zip still reports version "reference libFLAC 1.3.4 20220909" when used from third party software like CueTools.

Ooops! Sorry about that; all fixed now. More haste, less speed should be the maxim, I think. ;)

According to the checksums, the 32-bit version is different, not the dll inside flac_dll-1.4.0-x64.zip which produces the wrong version info for me.

I guess someone should update https://xiph.org/flac/

@A_Man_Eating_Duck has been doing some testing on CDDA in the 1.3.4 thread (https://hydrogenaud.io/index.php/topic,122179.msg1015346.html#msg1015346). I've earlier posted results in the compression improvement thread, showing sometimes big impact on "high resolution" as ktf mentions above.
So with compression threads elsewhere, maybe one should refrain from posting here too, yet here might be where new users may look ... so a few speed figures.

I am running a test on a freshly selected CDDA corpus on a freshly acquired CPU that passes 10k at https://www.cpubenchmark.net/, I couldn't help myself playing with it. FLAC from CLI doesn't utilize more cores, but still. I started from -4, comparing 1.3.4 and 1.4.0, and the first question is how the (small) compression improvements relate to time taken. In particular:

How does 1.4.0 at preset N compare to 1.3.4 at the next preset N+1? (For N=8: compared to -8p.)

Corpus: 38 albums, FLAC-compressed sizes: 4 GB classical music, 4 GB hard rock / heavy synth, 4 GB neither (=poprockjazzsoulrap). So seconds on the same corpus, on internal SSD, FLAC to FLAC with -f:
270 for 1.3.4 @ -4
290 for 1.4.0 @ -4, beats 1.3.4 @ -5 at size
315 for 1.3.4 @ -5
333 for 1.4.0 @ -5, narrowly loses to 1.3.4 @ -6 by 0.008 percent size difference
400 for 1.3.4 @ -6
409 for 1.4.0 @ -6 cannot beat 1.3.4's -7, but -6 isn't a good preset.
427 for 1.3.4 @ -7
450 for 1.4.0 @ -7 beats 1.3.4's -8 by 0.17 percent. Even beats 1.3.4's -8p by nearly 0.1 percent!
569 for 1.3.4 @ -8
610 for 1.4.0 @ -8 Also beats 1.3.4's -8p, of course.
1501 for 1.3.4 @ -8p
1782 for 1.4.0 @ -8p (This starts to get expensive.)

- Presets 0, 1 and 2 got a little faster, 3, through 8 got a little slower.

That extra time is well spent, if it compresses faster to smaller size and we've had some years of Moore's law since the last compression change.

450 for 1.4.0 @ -7 beats 1.3.4's -8 by 0.17 percent. Even beats 1.3.4's -8p by nearly 0.1 percent!

That's absolutely nuts.

I know there's been some discussion regarding this already, but I forgot - is -8ep even marginally useful any more?

I know there's been some discussion regarding this already, but I forgot - is -8ep even marginally useful any more?

On most music -ep is indeed way too slow, but some gains can be achieved on untypical music (like dark ambient, etc)

Stahlwerk 9 "I Am Become Death", almost same encoding speed:
23.645.595 bytes (ratio=0,294): -8 -A subdivide_tukey(9) -p
23.231.317 bytes (ratio=0,289): -8 -e -p

Btw, i found that welch can be useful in combination with subdivide_tukey.

Quote from: Porcus on 2022-09-10 16:05:02

450 for 1.4.0 @ -7 beats 1.3.4's -8 by 0.17 percent. Even beats 1.3.4's -8p by nearly 0.1 percent!

That's absolutely nuts.

I know there's been some discussion regarding this already, but I forgot - is -8ep even marginally useful any more?

-8ep took nearly six hours using 1.3.4 - only to lose at compression to 1.4.0 at -8. But hey, it got close enough to defeat -7. I need to run a -7p to see if ...  O:)  but not before 1.4.0 has done -8pe, which will take the night.

To give you an idea of how FLAC allows you to run in molasses with steel shoes:
-5 took five minutes fifteen sec
-7 took 7 minutes
-8 took 9 minutes and a half
-8p took 25 minutes - and 1.4.0 @ -7 gives you smaller file at > triple speed.
-8pe took 5h56.


As for -e, I found that it could help on high resolution material - but I also found that I could outdo it with high subdivide_tukey. For CDDA resolution, neither -e or very high subdivide_tukey made the same impact, but it does not surprise me to see some signals where it helps. Of course the issue is that FLAC cannot know in advance where to apply brute force.

As for welch, I am mildly surprised that it works with subdivide_tukey, as it doesn't look much different from the tukey function. But maybe that is the reason: slightly different from something good, could be even better. At https://hydrogenaud.io/index.php/topic,120158.0.html I also found that some signals like the welch. I would have been less surprised to find flattop useful, which is why I tested it (https://hydrogenaud.io/index.php/topic,120158.msg1014187.html#msg1014187) ... bummer.

I'm running a test on my files comparing v1.4.0 -8p and -8pe. -8p is already done but -8pe will take a total of 40hours to complete.

I'll post those results in this thread once done.

Since you are taking the big power bill, I shall not post my -8pe spoiler until then? ;-)

Meanwhile, the music I used for the testing, described here: https://hydrogenaud.io/index.php/topic,122960.0.html (to be updated)
Music might also be interesting ... y'know?  ;)

Sizes, compressed with 1.4.0 at default -5:
4 002 927 468 bytes of classical music, bitrate 598. Can be compressed to 3 982 303 322 at -8ep
4 000 248 235 bytes of heavier rock, bitrate, bitrate 971. Compressible to 3 978 679 443 at -8ep
4 030 323 496 bytes of "other" music, bitrate 704. Compressible to 3 996 086 171 at -8ep. This is the section that shrinks the most!

Then the "other" includes "Sunflower", this year's Ukraine charity EP from Dream Theater keyboardist Jordan Rudess. From a metal band, doing Chopin, one of his own compositions and a one minute rendition of Ukraine's national anthem, arguably as a tie-breaking criterion put in the "other" to get all three just across the 4 000 000 000 mark :-o That EP turned out to become two percent smaller in 1.4.0 than in 1.3.4, consistently across settings. Hm.
One file got missing from the initial run, so the above total times are, say, 1/37th too low. I won't bother to correct those times, it won't make big differences. The number of exceptions below are corrected.


So here is what happens. All "savings" etc are relative to the line immediately above it:
1.4.0 is compared to 1.3.4 at same setting
1.3.4 is compared to 1.4.0 at "lower setting". That lower setting is faster. And 1.4.0 wins on the most interesting presets.

1.3.4 @ -4.
1.4.0 @ -4. All sections smaller. 3 exceptions that make for larger files (Laibach: NATO 0.13 percent bigger, then Armand Van Helden and Kiss). At the other end: The Jordan Rudess Ukraine charity EP s(h)aves 2.35 percent. That EP benefits from 1.4.0 all the way.
1.3.4 @ -5. Worse in all three sections, though varying a bit. From .6% smaller to >2% bigger.
1.4.0 @ -5. 1.4.0 improves every file (from near-zero to two percent).
1.3.4 @ -6: classical section bigger than 1.4.0 at -5. The other two sections smaller.
1.4.0 @ -6: All files but one are smaller than 1.3.4. 
1.3.4 @ -7: classical section bigger than 1.4.0 at -6. The other two sections smaller.
1.4.0 @ -7: All files but one are smaller than 1.3.4. 
1.3.4 @ -8: All three sections (and all files but eight) lose against 1.4.0 @ -7. The "biggest exception" is 0.06 percent for Laibach: NATO
1.4.0 @ -8: 1.4.0 improves every file (from near-zero to two percent).
1.3.4 @ -8p: All three sections (and all files but twelve) lose against 1.4.0 @ -7; closest is the heavy corpus that is within 0.11%; biggest exception is 0.08 percent for Bach's harpsichords.
Also every section loses against 1.4.0 -7.
1.4.0 @ -8p: Every file improves over 1.3.4.  Still a couple of files improve one percent (Wovenhand: Black of the Ink full-length album) or even two for Jordan Rudess.
1.3.4 @ -8pe: All three sections (and all files but six) lose against 1.4.0 @ -8p; closest is the heavier section that is within 0.08%; biggest "single album exception" is 0.012 percent, yet again for Bach's harpsichords.
All three sections also lose against 1.4.0 @ -8. It wins at compression against 1.4.0 @ -7, except the classical section.
1.4.0 @ -8pe: Every file improves over 1.3.4.  Still Rudess saves 2.1 percent and Wovenhand 1.02.
-8pe took "only" three minutes more for 1.4.0.  Still short of six hours. 


So it seems that, the higher precision by and large makes for going down a preset and still saving space.
For classical music there are a couple of exceptions, but how many use 1.3.4 at -6 (which isn't a good setting!) and will consider going -5 when 1.4.0 arrives? And those who use 1.3.4 at -7 should hope they have computing power to stay at -7, rather than going to -6.

Appreciate the effort.

Meanwhile, the music I used for the testing, described here: https://hydrogenaud.io/index.php/topic,122960.0.html (to be updated)

Hm, I see you opted not to go for the superior test material choice of 3 songs representing a wide spectrum of music (https://hydrogenaud.io/index.php/topic,122058.msg1009143.html#msg1009143).

cool, so I just saved 22,5 GB disk space by reencoding everything with 1.4.0 at -8ep (because why use a lower setting??)

cool, so I just saved 22,5 GB disk space by reencoding everything with 1.4.0 at -8ep (because why use a lower setting??)

Yep, if you have 17 TB of CD rips encoded at -8ep. With -8ep taking six hours for twelve GB, that would translate to a year's reencoding time - YMMV, but I doubt you "just" did that  :))

We are not talking miracles. This is evolution without a leading "r".

(I actually didn't post improvement yet, but in my corpus, going 1.3.4 @ -8ep to 1.4.0 @ same, will save 0.1322035953 percent.)

To be fair, if the advantage Flake used to have over FLAC (the encoder, not the format (obviously)) mattered to you enough to prefer using it, this should count as a "significant" improvement in some capacity, as the autocorrelation precision increase and additional compression improvements essentially remove that advantage.

(At least, as far as I know.)

Congrats! Major point releases don’t happen every day you know. Million thanks to everyone who put their time into this and make FLAC what it is today.

edit: what I just posted must have a grave error. Will re-run once and repost.

Turns out this CPU is throttling in a way that is likely good for the purpose, but not so good for timing: starting cold and running two tasks where the temperature sensor starts throttling midway first task, and you get the idea. Anyway, a couple of re-runs later, the following timings should be reliable at ballpark level, and the results are so clear that I don't need more.

Also, new timings are done on all 38 and, rather than re-encoding I encoded .wav files.

To be fair, if the advantage Flake used to have over FLAC (the encoder, not the format (obviously)) mattered to you enough to prefer using it, this should count as a "significant" improvement in some capacity, as the autocorrelation precision increase and additional compression improvements essentially remove that advantage.

(At least, as far as I know.)

1.4.0 at -8 beats Flake at -8 in my test, at around half the encoding time.
Even if the heavier corpus makes for 0.01 percent bigger files, I think that conclusion is safe: The time difference from flake -8 up to flac -8p is about the same as down to flac -8, but saves you five to six times as much on that sub-corpus.  So if you are willing to go flac -8 to flake -8, you should be willing to go to flac -8p.

Some comparisons are given in the table, listed in order of decreasing total size, sizes are total = classical + heavy + other, and then time at the end.
Also I put WavPack into the table (not broken into genre), using only the -h mode [note at end to the very few HA'ers who don't know WavPack] and without double checking times. I admit to a fancy for -x4 for what it can do to high resolution (http://www.audiograaf.nl/losslesstest/revision%205/Average%20of%20hi-res%20sources%20up%20to%20192kHz.pdf), though it does not achieve the same on CDDA (where WavPack with and without the -x method are already well optimized, I guess) - while I think -x6 falls in the rightmost column although sometimes we cannot help ourselves using -8pe either.
 
Now if subset can be sacrificed, then damn this new computer has a GPU ... a bit surprised at what happens at the "other" section.

But what you point out is part of what I have been testing: "if you were willing to choose <x>, then you should be willing to choose 1.4.0 at <setting>". Reasoning as if the user makes deliberate choices out of CPU cost / size benefits as I measure them - most flawed assumptions, of course:
* users who run flac.exe at default most often don't consider -4/-6/-7/-8. (Cost/benefit argument: default is good enough, benefits are not worth the hassle of nerding yourself up on alternative settings.)
* users who run at -8 or higher most often don't consider -6 or -7.
* and even if they were, my audio and my hardware are not theirs


WavPack note for those any HA readers who don't know it:
First on FLAC: FLAC was developed for ultra-light CPU footprint at decoding - and nothing decodes as ultra-light as FLAC!
By the time FLAC was born, WavPack was already around - it can achieve smaller files that require more juice to decode, and mid-noughties, that could be noticeable in terms of battery life on already-then-low-powered portable players. For more on performance figures, see ktf's tests at http://www.audiograaf.nl/losslesstest  .
WavPack's "-x" extra processing settings are - alike heavier FLAC options, by and large - costly at encoding time, but not decoding time. -x (=-x1) is considered most reasonable (/"essential", depending on whom you ask), while -x4 to -x6 are rightfully considered very slow. I guessed flac -8pe could use a sanity check against those, so I included them.

I submit the following compiles for testing. They are all Intel 19.2 compiles and the win32 compiles should run on XP as previously. I don't believe there is any speed difference of note, just simply using the latest compiler. :)

www.rarewares.org/files/lossless/flac-1.4.0-x64-Intel-19.2.zip (http://www.rarewares.org/files/lossless/flac-1.4.0-x64-Intel-19.2.zip)
www.rarewares.org/files/lossless/flac-1.4.0-x86-Intel-19.2.zip (http://www.rarewares.org/files/lossless/flac-1.4.0-x86-Intel-19.2.zip)
www.rarewares.org/files/lossless/flac_dll-1.4.0-x64-Intel-19.2.zip (http://www.rarewares.org/files/lossless/flac_dll-1.4.0-x64-Intel-19.2.zip)
www.rarewares.org/files/lossless/flac_dll-1.4.0-x86-Intel-19.2.zip (http://www.rarewares.org/files/lossless/flac_dll-1.4.0-x86-Intel-19.2.zip)

Can anyone tell me what the differences between -8, -8e, -8p and -8ep are?

I didn't find a documentation concerning these encoding options!

...but by testing I did already learn something about the file sizes that are produced by these options:

"-8" > "-8 -e" = "-8e" > -"8p" > "-8ep" = "-8ep -e" = -8p -e"

Thus, I'll use now "-8ep -P 2048" as "extra command line options" with FLAC Frontend instead of "-e -P 2048" , even tough this saved only 322 KB on 364 MB on my test files (I came down from 382.372.861 bytes to 382.702.491 bytes).

flake -8  :	11970638661	1135 sec
1.4.0 -8  :	11969604531	551 sec

https://www.youtube.com/watch?v=e3uOMCfopR8

Another Question:

Can values for -A subdivide_tukey(....) higher than 3 cause compatibility problems?

I made some tests with "-8ep -A subdivide_tukey(9) -P 2048" as "extra command line options" with FLAC Frontend and this saved more or less 10 kb on every song.

But unfortunately using these options take a lot of time and it seems that flac.exe uses only one cpu core, which seems to be the main cause for the extremely long encoding time.

Thus, I'd like to use the "-A subdivide_tukey(9)" option only if there's a way to let flac.exe use more than one cpu core and if this option does not cause any compatibility problems.

Can anyone tell me what the differences between -8, -8e, -8p and -8ep are?

I'll try. A bit of background first:
So FLAC was designed for light decoding, and therefore the encoded data aren't packed in a very complex format. Also, FLAC decoding depends very little upon encoding parameters.
Unlike e.g. Monkey's, where using a higher mode means you use a more complex and more decoding time-consuming algorithm, FLAC instead "tries many simple ones" and chooses the best. But it doesn't "brute force optimize" a lot: instead it has a model where it guesstimates which one will be best, and then it spends time packing that one.

Therefore also the answer to your compatibility question is that ... don't worry. (If you ever see something that requires the "--lax" option, that is a different thing: then decoders can start to object.)

So the presets. The presets -0 to -8 are just names for different parameters to fire at it. Nothing says that "-6" is twice as good in any way as "-3".  The difference between -7 and -8 is that the latter only tries more windowing functions, hoping to find - for each 4096 samples - one that does better than the -7 choice. If it is better for a small part of the signal, it is selected.

Part of the "guesstimation" can be overridden with -e. With -e, part of the model selection will be done by calculating everything and then selecting the best. Think that you have two ways x or y to drive from A to B: you look at a rough map and guess, it looks like x is shorter, you drive "x" and measure it and declare that to be the distance. But your map could have been too rough, so to be sure you can go back and measure y exactly as well. Oh and there was a possible shortcut there ... possible? Only one way to find out.
-e exhausts several options exactly. Takes time.

-p does something similar on a different aspect of the modelling.

The FLAC encoder has become so good by now, that the time spent on -e is not worth it - on CDDA. On high resolution material, -e makes more of an impact - but even then there are other time-consuming procedures that can be added, that seem to work better (at least on my test corpus). The -A subdivide_tukey(<high number>) makes for trying more different apodization functions (= try more statistical filters, you will have a greater chance of finding something that fits better).

-p seems to be harder to beat. When you are at -8 and want more, then -8p might still be better than more apodization functions. (After all, -8 seems better than -7p, and -8 does indeed use more functions than -7.)

Documentation? It isn't explained in detail at https://xiph.org/flac/documentation_tools_flac.html , no.  And the thing is, we don't really know precisely why this works better than that. There are good reason that this measure improves compression, but the reason why -p works better than -e is ... well, we have measured it on real-world signals and it turns out it does so.

As for your -8ep = -8ep -e, that is because the options can be concatenated.  FLAC reads "-8ep" as -8 -e -p.

Order isn't arbitrary though.  In your -8ep -A subdivide_tukey(9) it first selects -8 (which implies some "-A"!), then switches on the -e and -p brute forces - and then you give a "-A" that overrides the one initially selected by -8. Note that by choosing such a high argument in the subdivide_tukey, you also have a lot to brute-force-check, so this.is.slow and likely only to play with for testing.
For testing you can try to compare the following:
-8
-8e
-8 -A subdivide_tukey(9)
-8e -A subdivide_tukey(9)
-8p
-8pe
-8p -A subdivide_tukey(9)
-8pe -A subdivide_tukey(9)
... and I have a hunch that is the order of sizes, but not the order of times. And that anything but -8 and -8p are "not worth it".


"-P 2048" has nothing to do with the compression of the audio. -P controls is the padding - the amount of empty space set aside at the end of the tags section, before the audio starts.
The default is 8096 bytes up to 20 minutes (why larger for larger? Because those are often entire albums and have more tags), and by setting it to 2048 bytes, you saved 6 kilobytes.
But, padding is there for a purpose. If you make a small tag edit, it will write into the padding - but if you write a bigger one, so that tagging needs more space than your padding section has set aside, then you have to wait for the entire file being written anew. Reducing the -P makes for more frequent full file rewrites. Then on the other hand, 8096 is hardly enough for album art. It maybe was when FLAC was released.


Was this about it?

1.3.4 - flac.exe" -s --ignore-chunk-sizes -8p - -o %d

Total size : 393 GB (422 709 930 380 bytes)
Duration : 5wk 6d 0:45:59.793
Sample rate : 44100 Hz (97.3%); 48000 Hz (2.7%)
Channels : 2
Bits per sample : 16 (96.8%); 24 (3.2%)
Avg. bitrate : 954 kbps
Codec : FLAC
Encoding : lossless
Tool : reference libFLAC 1.3.4 20220220
Embedded cuesheet : no

1.4.0 - flac.exe" -s --ignore-chunk-sizes -8p - -o %d

Total size : 393 GB (422 292 414 045 bytes)
Duration : 5wk 6d 0:45:59.793
Sample rate : 44100 Hz (97.3%); 48000 Hz (2.7%)
Channels : 2
Bits per sample : 16 (96.8%); 24 (3.2%)
Avg. bitrate : 953 kbps
Codec : FLAC
Encoding : lossless
Tool : reference libFLAC 1.4.0 20220909
Embedded cuesheet : no

1.4.0 - flac.exe" -s --ignore-chunk-sizes -8pe - -o %d

Total size : 393 GB (422 175 699 576 bytes)
Duration : 5wk 6d 0:45:59.793
Sample rate : 44100 Hz (97.3%); 48000 Hz (2.7%)
Channels : 2
Bits per sample : 16 (96.8%); 24 (3.2%)
Avg. bitrate : 952 kbps
Codec : FLAC
Encoding : lossless
Tool : reference libFLAC 1.4.0 20220909
Embedded cuesheet : no

Results
1.3.4 -8p  = 403127.60MB
1.4.0 -8p  = 402729.43MB (Savings of 398.17MB over 1.3.4 -8p)
1.4.0 -8pe = 402618.12MB (Savings of 509.48MB over 1.34 -8p, Savings of 111.31MB over 1.4.0 -8p)

So, savings are in 0,1% range, no? What kinds of music are in your collection?

It's mainly rock and metal.

Results
1.3.4 -8p  = 403127.60MB
1.4.0 -8p  = 402729.43MB (Savings of 398.17MB over 1.3.4 -8p)
1.4.0 -8pe = 402618.12MB (Savings of 509.48MB over 1.34 -8p, Savings of 111.31MB over 1.4.0 -8p)

509MB storage space savings sounds decent until you see how large the general data is, which is 422.x GB in size, at which point that 509MB savings completely loses it's appeal.

but as we all basically know... differences in file size/compression speed/decompression speed/battery life (and even conversion time from FLAC to MP3/Opus/AAC etc) are probably so small it really does not matter what one chooses as I think the difference in lossy bit rates is worth nit-picking on more than FLAC. so I figure with FLAC just stick to the default (-5) or maybe -8 if someone wants to save a little storage space for good measure as beyond that the amount of time to re-compress is simply too high for the barely any storage space savings, unless of course someone is obsessed with as minimal of file size as possible on FLAC.

still, it's nice to see a small improvement with current FLAC version.

but thanks for the 'quick' tests there anyways ;)

No, differences are not that much, especially at that end when we are closing in on what a computationally simple data format like FLAC can do.

On high resolution material, 1.3.4 wasn't quite there.  Tested here. (https://hydrogenaud.io/index.php/topic,120158.msg1014183.html#msg1014183) At -8, the 1.4.0 encode saved 3.37 percent. Then on a 400 GB corpus the saving would be 13 GB and not half a GB. That is quite something to notice.

Reasons to do this upgrade, include
* to iron out wrinkles that sometimes make for embarrassing results
* for the geeky art of engineering
* for sports - finally found what made Flake perform so well, now the reference implementation is more on top of its own game

Worth it to encode? Maybe. Worth it to re-encode? Oh, that's a different question.

Savings of 111.31MB over 1.4.0 -8p

S(h)avings between 0.025 percent and 0.03 percent, same as in my test.
I tested the same with 1.3.4. Then the impact was > 0.1 percent.  (With my 12 GB corpus, the additional "e" at the end of -8p would save 13 MB for 1.3.4, and only 3 MB now.)

-8pe would spend ages on nearly nothing. Now it spends ages on a quarter of nearly nothing, because the basic algorithm has improved to already catch what the "e" can accomplish.

This week's news:
Queen -e has passed away. She had her time.

I submit the following compiles for testing. They are all Intel 19.2 compiles and the win32 compiles should run on XP as previously. I don't believe there is any speed difference of note, just simply using the latest compiler. :)

Thnx John :)

Any idea what minimum CPU architecture these v1.4.0 binaries were built for?

Which, specifically? Quite a few have been dropped in this thread.

Any idea what minimum CPU architecture these v1.4.0 binaries were built for?

win32 - XP, x64 - 8.1.

Edit: Oops, I assumed OS! CPU should be anything that supports those OSs.

Which, specifically? Quite a few have been dropped in this thread.

I tested these four x64 binaries: the "official" release, both of the latest Netranger (clang and gcc) and John's icl compile. Since I'm more interested in speed than shaving some 0.0xx% off the flac size, I only tested compression levels -7 and -8. The speed variation on my 5 years old CPU i7-7700 were minimal (at -7: 37.84s, 37.18s, 37.94s, 37.71s). So I assumed that similar architecture settings were used to run on ALL CPUs.
Some time ago HA users enzo and case were kind enough to provide v1.3.3 x64-binaries, that were noticeably faster than the official compile and one of the optimisations were afaik to demand at least a Nehalem CPU architecture (12 years old now). I still use these binaries today since I never had a faster one (at -7: 24,96s, at -8: 35,99s) -> they are still faster at -8 than any of the 1.4.0 binaries at -7, but because of optimisations in 1.4.0 result in almost the same flac file size.
So I wonder if 1.4.0 would also benefit from a more restrictive CPU support to enable more instruction set extensions (do we still use CPUs prior to Core-i?)
Just my 2cents

(do we still use CPUs prior to Core-i?)

Believe it or not, I've had people complaining that FLAC defaults to requiring SSE2, which was introduced in 2000. See here: https://sourceforge.net/p/flac/bugs/479/

I explained that FLAC can be configured such that a binary can be compiled supporting all CPUs back to the Intel 486, sacrificing performance on modern CPUs in the process. Apparently this wasn't enough because SSE-but-not-SSE2 processor would not get the best possible binary. In my view that means asking/demanding that I would write hand-optimized code for those CPUs.

I'm using FLAC 1.3.4 20190805 with -8pe and it is significantly faster than 1.4.0 with minimal increase in size:

1.3.4:
137.210.816 bytes @ 1:32 (average over 10 passes)

1.4.0 (GCC, the others I've tried are much slower):
137.203.644 byte @ 2:06 (average over 10 passes)

<offtopic>What OS do you run on a year-2000-CPU?  ;D </offtopic>
You can't please everyone. But you guys spend so much time and effort in optimising flac's algorithm and have to waste maybe some 4% performance to support all sort of old CPUs that only few people use.
I can only speak for myself: I'd have zero problems to keep using v1.3.3 on my ancient CPU if the default binary for v1.4.x would require a decent, modern architecture if there was a performance gain worth mentioning. Or I'd spend some time and effort and try to setup the compiler toolchain and build my own binary.

137.210.816 bytes @ 1:32 (average over 10 passes)

137 MB flac size in 92 seconds? What CPU did you use?
Sorry, didn't notice you encoded with -8ep... my bad.

For those using win32, but not XP, here are binaries that don't need the runtime dlls:

www.rarewares.org/files/lossless/flac-1.4.0-x86-nonXP-Intel-19.2.zip (http://www.rarewares.org/files/lossless/flac-1.4.0-x86-nonXP-Intel-19.2.zip)
www.rarewares.org/files/lossless/flac_dll-1.4.0-x86-nonXP-Intel-19.2.zip (http://www.rarewares.org/files/lossless/flac_dll-1.4.0-x86-nonXP-Intel-19.2.zip)

I'm using FLAC 1.3.4 20190805 with -8pe and it is significantly faster than 1.4.0 with minimal increase in size:

1.3.3 you mean?

This is a big "YMMV", -8pe took like one percent more time on 1.4.0 that 1.3.4 in my testing.

Not that one album is much to judge size gains that are this small, but for what it is worth: Can you post 1.3.3 at -8p times and size?


Edit:

<offtopic>What OS do you run on a year-2000-CPU?  ;D </offtopic>

Since I am discussing inferences from a single data point already ...
Right now at this computer, I am using a CPU launched ~nine years ago, on an OS launched ~thirteen years ago. What do you think, does the "four years older" rule make for a more or less reasonable result than upscaling the age of 22 by 13/9?

1.3.3 you mean?

This is a big "YMMV", -8pe took like one percent more time on 1.4.0 that 1.3.4 in my testing.

Not that one album is much to judge size gains that are this small, but for what it is worth: Can you post 1.3.3 at -8p times and size?

You are correct, it actually is FLAC 1.3.4 20220220 (GCC) on W10 x64 with an Intel i7 9700

with -8p I get :

137.223.277 bytes @ 0:14:xxx
original size: 242.961.644 bytes ( Btw, this is track #6 from Genesis' Foxtrot CD)


For the whole album numbers are (again with -8ep):

293.203.041 bytes @ 1:31:378 (the album has 6 tracks and the CPU 8 cores so all tracks are encoded simultaneously)

same but using 1.4.0:

293.177.359 bytes @2:08:866

@sundance - I compiled FLAC 1.4.0 the same way I created my previous fast 64-bit compile. Attached for your testing pleasure.

@ Jackal666InHex:

With these figures, the small gains from going 1.3.4 at -8pe to 1.4.0 at -8pe, are still cheaper than the "e".
Which means: if you don't think the 1.4.0 slowdown is worth it at -8pe, then -8pe should have been avoided in the first place, as that slowdown measures up as worth even less.

(And even even less now, as 1.4.0 has dwindled the shavings of the "e".)

Here are the calculations:

1.3.4:
137.210.816 bytes @ 1:32 (average over 10 passes)

1.4.0 (GCC, the others I've tried are much slower):
137.203.644 byte @ 2:06 (average over 10 passes)

7172 bytes saved in 34 seconds, meaning that a second extra processing time saves:
211 bytes. Very little indeed.


Then compared to this:

with -8p I get :

137.223.277 bytes @ 0:14:xxx

Going 1.3.4 at -8p to -8pe gains 137.223.277 - 137.210.816 = 12416 bytes. Takes 78 extra seconds, meaning that a second extra processing time saves:
159 bytes. That is less than 211.

@sundance - I compiled FLAC 1.4.0 the same way I created my previous fast 64-bit compile. Attached for your testing pleasure.

These are my test results:

flac140.exe -7 (64 bit):                     Global Time = 37.838
flac140-netranger-clang.exe -7 (64 bit):     Global Time = 37.178 *
flac140-netranger-gcc12-2.exe -7 (64 bit):   Global Time = 37.946
flac140-john-icl.exe -7 (64 bit):            Global Time = 37.713
flac140-Case.exe -7 (64 bit):                Global Time = 30.114  -> 19.0% faster than *
-> FLAC size: 1.167.093.760 Bytes

flac140.exe -8 (64 bit):                     Global Time = 54.999 #
flac140-netranger-clang.exe -8 (64 bit):     Global Time = 56.768
flac140-netranger-gcc12-2.exe -8 (64 bit):   Global Time = 55.100 
flac140-john-icl.exe -8 (64 bit):            Global Time = 57.346
flac140_Case.exe -8 (64 bit):                Global Time = 46.420  -> 15.6% faster than #
-> FLAC size: 1.166.278.656 Bytes

flac133_case.exe -7 (64 bit): Global Time = 24.964
flac133_enzo.exe -7 (64 bit): Global Time = 24.961
FLAC size: 1.168.025.915 Bytes

flac133_case.exe -8 (64 bit): Global Time = 36.012
flac133_enzo.exe -8 (64 bit): Global Time = 35.994
FLAC size: 1.167.279.038 Bytes

My conclusion:

• Case's optimised flac140 is (on my i7-7700) way faster than every other binary
• flac133_case.exe -8 and flac140_case.exe -7 create very similar flac sizes (delta=159 ppm), but v1.4.0 is much faster by 19.6%
• I can easily replace flac133_case -8 with flac140_case -7: "The King is dead, long live the King!"
• There is no replacement for flac133 -7, since flac140 -6 is not a good setting
• It's worth to have a optimised binary along with the "one-size-fits-all" standard build

Case, thank you very much for your build, you made my day.
Please don't get me wrong: This is not meant to start some sort of "flac builder bashing".
You guys provide us (the noobs that are not able to build their own binaries) with the lastest flac binaries. Thanks a lot for your ongoing efforts!
And I have to update my statement:

But you guys spend so much time and effort in optimising flac's algorithm and have to waste maybe some 4% 15-20% performance to support all sort of old CPUs that only few people use.

Would be very interesting if others (with other CPUs) also see some speed improvment.

No science but using flac 1.40 as exe in CUETools gives ~90x speed with Netrangers GCC compile while Cases compile is at ~85x on a Ryzen 5900x for -8 -p

Brief speed test at default -5, all x64 compiles, will later run a round with both the 32-bit and 64 bit compiles and also at -7p.
Why -7p when it isn't optimal? "p" because that triggers brute force on something (i.e. some routine has to be run several times, is it there the savings are or isn't it?) and -7 because -7 is a good one. Besides, -8p is slow and this will run for long.

Anyway, at -5, on an 11th gen mobile i7, reading .wav from a bitlocker'd internal SSD and writing to same.

254: Case build <-- fastest, the number is seconds
301: Official build
389: John33 Rarewares x64-Intel-19.2 from Reply 34
401: NetRanger CLANG from Reply #15
418: NetRanger GCC freshly downloaded from the edited Reply #10

Are these differences suspiciously big? Just a single run each, will run more overnight.

i3-12100, 59m30s 16/44 wave file on RAM drive @ -8p, all of them are 64-bit exe

51s john33 at Reply #34
41s Case at Reply #57
41s NetRanger at Reply #10
37s Xiph official

I repeated the tests several times and the results are consistent.

I can confirm that on higher compression presets, the official build takes the lead:

flac140 -8p (64 bit):            Global Time = 153.346
flac140_Case.exe -8p (64 bit):   Global Time = 165.111

i3-12100, 59m30s 16/44 wave file on RAM drive @ -8p, all of them are 64-bit exe

51s john33 at Reply #34
41s Case at Reply #57
41s NetRanger at Reply #10
37s Xiph official

I repeated the tests several times and the results are consistent.

Same as above, but a 2h13m30s 24/48 stereo file at -8 without p.
43s Case
44s Xiph
50s NetRanger
1m5s john33

Oh, I did not pay attention that NetRanger's Reply #10 was a wrong file, here are the correct results at Reply #15

1m9s for the 24/48 file at -8

54s for the 16/44 file at -8p

A brief question: Are these "equations" still valid in v1.4.0 ?

FLAC v1.3.3
  -0, --compression-level-0  =  -l 0  -b 1152 -r 3
  -1, --compression-level-1  =  -l 0  -b 1152 -M -r 3
  -2, --compression-level-2  =  -l 0  -b 1152 -m -r 3
  -3, --compression-level-3  =  -l 6  -b 4096 -r 4
  -4, --compression-level-4  =  -l 8  -b 4096 -M -r 4
  -5, --compression-level-5  =  -l 8  -b 4096 -m -r 5
  -6, --compression-level-6  =  -l 8  -b 4096 -m -r 6 -A tukey(0.5) -A partial_tukey(2)
  -7, --compression-level-7  =  -l 12 -b 4096 -m -r 6 -A tukey(0.5) -A partial_tukey(2)
  -8, --compression-level-8  =  -l 12 -b 4096 -m -r 6 -A tukey(0.5) -A partial_tukey(2) -A punchout_tukey(3)

(flac v1.4.0 produces different results with "-7" compared to "-l 12 -b 4096 -m -r 6 -A tukey(0.5) -A partial_tukey(2)")

(flac v1.4.0 produces different results with "-7" compared to "-l 12 -b 4096 -m -r 6 -A tukey(0.5) -A partial_tukey(2)")

Tested and differents results despite synonyms in MAN page / GUI

FLAC v1.4.0 (Release)
Built on September 14, 2022, CLANG 15.0.0

A brief question: Are these "equations" still valid in v1.4.0 ?

No, that is obsolete. But first, I notice you have a space before your question mark, so have a locale where the decimal point is a comma - we are more than the dot-users! - then your tukey(0.5) will be silently ignored.
Use scientific notation tukey(5e-1), it works in both locales.

On to it, and ktf might have to correct my misunderstandings for the Nth time:
1.4.0 introduces a "subdivide_tukey" which recycles calculations to cover more windowing functions for cheap.
If I have understood correctly, new -8 is a bit like throwing in an extra partial_tukey(3) at the end (except faster) - but they are not precisely the same, because in order to accommodate that recycling, tapering steepness had to be tweaked.

Expected if you in 1.4.0 compare
-8 -A tukey(5e-1) -A partial_tukey(2) -A punchout_tukey(3)
to
-8
to
-8 -A "tukey(5e-1);partial_tukey(2);partial_tukey(3);punchout_tukey(3)"
is that -8 should be "better than the midpoint between the former and the latter" by giving sizes about like the latter, but faster.

Does that hold up?

Oh, I did not pay attention that NetRanger's Reply #10 was a wrong file

Is it wrong after the edit? Or is it now "correct, just a different compiler"?
@NetRanger ?

Sorry, I did not read every post carefully. I read the thread and downloaded the file on Reply #10 today, so it is the "correct" one.

Here is the table of default settings. (https://github.com/xiph/flac/blob/229f4f45df72ac1fc8a2f7749043223d745c8e27/src/libFLAC/stream_encoder.c#L115) The default apodization functions were changed without updating the in-program help (although this document (https://xiph.org/flac/documentation_tools_flac.html) did get updated).

The table doesn't include block size, I'm not sure how that's chosen.

Is it wrong after the edit? Or is it now "correct, just a different compiler"?
@NetRanger ?

New compile with the latest Clang

Here my quick and dirty performance test of different flavors of FLAC 1.4.0 (Win64) on my Win10, i7-1185G7 Laptop.
Using 2L-086_stereo-DXD_15.flac (http://www.lindberg.no/hires/test/2L-086_stereo-DXD_15.flac) @ 352,8kHz / 24bit FLAC, recompressing FLAC-2-FLAC at -8

Long story short:

Case's compile: ~ 13,1 s
Xiph compile: ~ 13,9 s
GCC compile: ~ 16,0 s
Rarewares Intel 19 compile: ~ 23,8 s
NetRanger's recent CLANG15: ~ 26,8 s

PS C:\TEMP\FLAC14> Measure-Command { .\flac140case.exe -8 2L-086_stereo-DXD_15.flac -f -o 2L-086_stereo-DXD_154.flac.140case.flac | Out-Default }

flac 1.4.0
Copyright (C) 2000-2009  Josh Coalson, 2011-2022  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

2L-086_stereo-DXD_15.flac: wrote 437376780 bytes, ratio=0,976


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 13
Milliseconds      : 181
Ticks             : 131815177
TotalDays         : 0,000152563862268519
TotalHours        : 0,00366153269444444
TotalMinutes      : 0,219691961666667
TotalSeconds      : 13,1815177
TotalMilliseconds : 13181,5177



PS C:\TEMP\FLAC14> Measure-Command { .\flac140xiph.exe -8 2L-086_stereo-DXD_15.flac -f -o 2L-086_stereo-DXD_154.flac.140xiph.flac | Out-Default }

flac 1.4.0
Copyright (C) 2000-2009  Josh Coalson, 2011-2022  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

2L-086_stereo-DXD_15.flac: wrote 437376780 bytes, ratio=0,976


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 13
Milliseconds      : 944
Ticks             : 139447664
TotalDays         : 0,000161397759259259
TotalHours        : 0,00387354622222222
TotalMinutes      : 0,232412773333333
TotalSeconds      : 13,9447664
TotalMilliseconds : 13944,7664



PS C:\TEMP\FLAC14> Measure-Command { .\flac140clang.exe -8 2L-086_stereo-DXD_15.flac -f -o 2L-086_stereo-DXD_154.flac.140clang.flac | Out-Default }

flac 1.4.0
Copyright (C) 2000-2009  Josh Coalson, 2011-2022  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

2L-086_stereo-DXD_15.flac: wrote 437376784 bytes, ratio=0,976


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 26
Milliseconds      : 865
Ticks             : 268651043
TotalDays         : 0,000310938707175926
TotalHours        : 0,00746252897222222
TotalMinutes      : 0,447751738333333
TotalSeconds      : 26,8651043
TotalMilliseconds : 26865,1043



PS C:\TEMP\FLAC14> Measure-Command { .\flac140gcc.exe -8 2L-086_stereo-DXD_15.flac -f -o 2L-086_stereo-DXD_154.flac.140gcc.flac | Out-Default }

flac 1.4.0
Copyright (C) 2000-2009  Josh Coalson, 2011-2022  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

2L-086_stereo-DXD_15.flac: wrote 437376780 bytes, ratio=0,976


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 16
Milliseconds      : 29
Ticks             : 160290049
TotalDays         : 0,000185520890046296
TotalHours        : 0,00445250136111111
TotalMinutes      : 0,267150081666667
TotalSeconds      : 16,0290049
TotalMilliseconds : 16029,0049



PS C:\TEMP\FLAC14> Measure-Command { .\flac140rarewares.exe -8 2L-086_stereo-DXD_15.flac -f -o 2L-086_stereo-DXD_154.flac.140rarewares.flac | Out-Default }

flac 1.4.0
Copyright (C) 2000-2009  Josh Coalson, 2011-2022  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

2L-086_stereo-DXD_15.flac: wrote 437376782 bytes, ratio=0,976


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 23
Milliseconds      : 880
Ticks             : 238807805
TotalDays         : 0,000276397922453704
TotalHours        : 0,00663355013888889
TotalMinutes      : 0,398013008333333
TotalSeconds      : 23,8807805
TotalMilliseconds : 23880,7805



PS C:\TEMP\FLAC14>

Hm.  I just aborted a test run on the CLANG15 because the timings were unreasonably high. It didn't make it to the following nightly FOR loop, so I started it during lunch time today, saw the timings and went oh, daytime temperature must have made the laptop CPU throttle much earlier. But only on -7p, the results were anywhere as .halverhahn's.

Anyway, here are some timing differences based on several runs of each.  The CLANG15 compile is the only that wasn't done nighttime.
On this CPU, an 11th generation i7 mobile, I would have stuck to official build because well official is official and I use -8 regularly, but those who care so much more about speed as to stick to default -5 could have time to save on Case's build.

-5
256   Case compile (x64) from reply 57
271   Xiph (x64 only)
300   Rarewares-x64 from john33's Reply 34 (first link)
321   NetRanger CLANG14-64 from Reply 15
328   Rarewares-x86-nonXP from john33's Reply 34 (first link)
330   Rarewares-x86 from john33's Reply 34 (second link)
336   NetRanger GCC-32 from Reply 10
344   NetRanger GCC-64 from Reply 10
347   NetRanger CLANG14-32 from Reply 15
366   NetRangerCLANG15-w64 from Reply 68
395   NetRangerCLANG15-w32

The >400 numbers disappeared, likely less CPU throttling likely because the computer had been only running FLAC encoding previously - or possibly lower temperature at night.

-7p
 831   Xiph (x64 only)
 880   Case compile (x64)
 882   NetRanger GCC-64
1006   NetRanger GCC-32
1015   NetRanger CLANG14-64
1035   Rarewares-x64
1089   Rarewares-x86
1170   NetRanger CLANG14-32
1284   Rarewares-x86-nonXP
1508    NetRanger CLANG15-64
aborted: NetRanger CLANG15-32

What I did for -7p:
* .bat file which would first warm-up with FLAC encoding, to make sure the CPU heated up and the fan started running already before the first to go; then it would loop through three iterations
* I would select the median.  I would also take note that the median isn't unreasonably close to the minimum, so that any weird incident would have to manifest itself in three consecutive runs.

For -5 I did the same, except that the loop that was supposed to run -5 and -7p had a whitespace missing between the output name and the "-7p", instead running six iterations of -5. I picked the third fastest.

Also interesting here: what files are bit-identical? Files/music posted at https://hydrogenaud.io/index.php/topic,122960
For 76 files per encoder (38 albums, -5 and -7p), I put a duplicate finder to work.

* Rarewares x64 and Rarewares x86 produce files bit-identical to each other - but never identical to anything else.
(For x86, that means the "non-XP" files - they overwrote the other x86 files since I'm clumsy.)
If you are interested in what size difference it makes: The Rarewares files are now 24 003 098 799 bytes, which is 55 bytes smaller than the official build.

* Each NetRanger CLANG file (14 or 15, though 15-x86 was aborted after FLAC -5) was bit-identical to some GCC-compile generated file.
Deleted all.

* NetRanger GCC compiles: Out of 2*38 albums (two FLAC presets), 43 duplicate groups were generated between the x64 and x86 compiles.
Deleted down to 109 files.
* Remaining 109 GCC compiles: 53 were bit-identical to official x64 compile. A further two were distinct from the Xiph build but bit-identical to Case's build.

* Case's build (x64) matches official (x64) build in 66 of 76 files
Deleted down to 10 files. These ten are very close in size to official-generated files - max difference is seven bytes (Jordan Rudess), the nine others between 0 and 3. Bytes, not kilobytes!


After this carnage, we are left with full Rarewares file set, full official file set, and about a quarter of the Case and NetRangerGCCx86 and NetRangerGCCx64 file set (10+29+24 files left).
The Wovenhand album is there in five of six possible versions - only one bit-identical to official build is Case at -7p. Size differences are nothing to talk about, typically in the single-digit byte count, sometimes twenty - and the biggest difference seems to be 177 bytes for the the Philips {1990} High Tech Choruses at -7p. That thing has test signals including sines that haven't been FLAC faves ... and I am slapping my hands not to dig more into those possibly spurious combinations. (Anyway, you want the numbers? 339 152 854 NetRangerGCC32, ...951 for GCC64, 339 153 016 for Case, ...017 for the official, ...031 for Rarewares.)

I also noticed some minuscule size differences between different encodes (made with different builds) of the same files. The only data that differs between these encodes when running metaflac --list is the seektables, and more specifically the stream_offset values which differ with what seems like random values, all other values are exactly the same. The only 2 bit-identical files were produced by NetRanger's GCC and CLANG14 builds.
In practice, the seektable and tiny filesize differences probaly mean absolutely nothing. Of course of interest here is why different builds produce different seektables? I don't have enough (as in: none at all) technical knowhow to figure this out :)

The builds I tested were (all 64-bit versions on Win10)
Xiph build from GitHub
NetRanger GCC from post #10
NetRanger CLANG14 from post #15
John33/RareWares Intel 19.2 from post #34
Case's build from post #57
NetRanger CLANG15 from post #68

Run from the commandline and a simple flac -8 -V as that's how my entire collection is done (on various version of flac).

Of course of interest here is why different builds produce different seektables?

Didn't even think of the possibility ... but anyway, I stripped everything such from the hundredandforty (?) "other-than-official" and none of them became identical. Didn't bother to check that against official.

metaflac --list .txt dumps from one test case, just in case someone's interested
The track: https://supersillyus.bandcamp.com/track/the-enigmagician (edit: downloaded the WAV version of course)
Chosen for eclectic reasons and also because it's free / name your price, and 96 kHz / 24-bit
Also, file sizes just out of interest:

Case: 225 374 563
CLANG14: 225 374 556 (bit-identical with the GCC build)
CLANG15: 225 374 558
GCC: 225 374 556
Intel 19.2: 225 374 531 (edit2: official RareWares Intel 19.0 build from https://www.rarewares.org/lossless.php is bit-identical with this build)
Xiph: 225 374 562

As you can all see, different CPUs handle the different compiles all very... differently. That makes profiling optimizations rather difficult: gains of less than about 10% might not be gains on other CPUs at all.

The only data that differs between these encodes when running metaflac --list is the seektables, and more specifically the stream_offset values which differ with what seems like random values, all other values are exactly the same.

Can you check again whether they are the same when seektable is removed? As far as I read the format spec (https://xiph.org/flac/format.html#metadata_block_seektable), the stream_offset should be where the sample's frame starts, and if every frame is equal between two files, the respective frame must obviously start at the same byte count.

As for timing the compiles: that new laptop of mine is throttling at its own discretion, and when looping the damn thing it seems to matter whether the previous encoding was more or less CPU intensive. Dell models come and go, Dell fan speed control remains a mystery.
Still I suggest that the timings numbers I posted in #75 might be sane, as they did repeated -5 after -5 after -5 and then repeated -7p after -7p after -7p (and then nothing after that presumed higher load). Not the other way around, when long cooldown time and short execution time might have tainted the numbers more - and the fastest run was deleted too. So I think they are a pretty good indication - at this CPU and at this pretty fast storage medium.

Here is another compile that certainly appears faster on my system although no empirical measurements taken. It is compiled using the Intel c++ 2022 compiler.
www.rarewares.org/files/lossless/flac-1.4.0-x64-Intel-2022.zip (http://www.rarewares.org/files/lossless/flac-1.4.0-x64-Intel-2022.zip)

@ktf: Sorry for the n00b question:
November last year you posted a graph (https://hydrogenaud.io/index.php/topic,120158.msg1004764.html#msg1004764) showing the performance of flac_currentgit and your subblock variant. Is v1.4.0 encoding performance equal/similar to the subblock-newsettings?

Is v1.4.0 encoding performance equal/similar to the subblock-newsettings?

No, it is not. However, a few posts further down, those of August 2022 are up-to-date. See reply #90 and #94.

Thanks for the hint. It's easy to loose track when threads get longer...
Confirms my findings that new -7 is faster and at the same time better compressing than old -8. A real "sweet spot" with top speed and compression close to the max.

Confirms my findings that new -7 is faster and at the same time better compressing than old -8. A real "sweet spot" with top speed and compression close to the max.

-7 is good indeed. I've been singing its praises since the test builds, suggesting that -8 needed a beef-up to get a proper difference (https://hydrogenaud.io/index.php/topic,120158.msg1003585.html#msg1003585). (-8 did since change, running a test to see if it makes for much. Oh, and for the reply #72 table: the numbers are basis points - 1/100ths of a percentage point.)


-7 also beats ffmpeg "at any setting within subset", even those spending > 10x the time:

In my test, at least: Of the 38 CDs used, -7 won each and every one even when I tried -compression_level 8 -prediction_order_method search -exact_rice_parameters 1 combined with -lpc_type cholesky -lpc_passes 3/6/9 or 12; 9 made for smallest files, took a hour and a half and lost on every CD. Although one was so close that I had to metaflac away padding to be sure -7 won.
Not tested though: I quickly aborted the multi_dim_quant option, as it brought the speed down to 1/1000 of -7.

Even outside subset, ffmpeg at -compression_level 11 ended up about as -7 in size, and took more time than -8. I wonder if any ffmpeg can get to 1.4.0 at -8 sizes.  But if subset can be sacrificed, there are better options still (https://hydrogenaud.io/index.php/topic,122949.msg1015508.html#msg1015508). CUETools seems to have done brighter development on Flake than ffmpeg did, and if you have a decent GPU ... that test was done on a laptop.

Can you check again whether they are the same when seektable is removed?

Nope. They're not the same. I ran metaflac --remove-all --dont-use-padding on all the files, then metaflac --list (which is obviously the same since only the streaminfo block is left intact and that's identical for all files).

File sizes after the operation:
Case: 225 365 123
CLANG14: 225 365 116 (bit-identical with GCC)
CLANG15: 225 365 118
GCC: 225 365 116
Intel 19.0: 225 365 091 (bit-identical with 19.2)
Intel 19.2: 225 365 091
Xiph: 225 365 122

I also tested the Intel 2022 build, metaflac --list dump included
Before metadata removal: 225 374 529
After: 225 365 089

-7 is good indeed.

... and -7 was very good in the v1.3.x days. That's why I still use it today: With the speed of v1.3.3 in mind, I could not find a 1.4.0-setting that is at least as fast and compresses as well as its predecessor on -7.

WAV size = 1.907.269.136 Bytes
flac133_case.exe -7:  Global Time = 24.826 sec, FLAC size: 1.168.025.916 Bytes
flac140-case.exe -7:  Global Time = 29.569 sec, FLAC size: 1.167.014.383 Bytes (= 61,188% of WAV size) -> slower, but better compression
flac140-case.exe -6:  Global Time = 25.919 sec, FLAC size: 1.170.517.594 Bytes, still slower & worse compression

And with new -6 being slower and worse compressing than old -7, it's probably hard to find something "in between" being able to match old -7. But that's almost nitpicking.

Here is another compile that certainly appears faster on my system although no empirical measurements taken. It is compiled using the Intel c++ 2022 compiler.
www.rarewares.org/files/lossless/flac-1.4.0-x64-Intel-2022.zip (http://www.rarewares.org/files/lossless/flac-1.4.0-x64-Intel-2022.zip)

No difference from the one posted on Reply #34 (https://hydrogenaud.io/index.php/topic,122949.msg1015520.html#msg1015520) on my PC (https://hydrogenaud.io/index.php/topic,122949.msg1015615.html#msg1015615).
And yes, NetRanger's CLANG15 is the slowest one in the whole thread on my i3-12100. Would like to see some Ryzen results.

As you can all see, different CPUs handle the different compiles all very... differently. That makes profiling optimizations rather difficult: gains of less than about 10% might not be gains on other CPUs at all.

It may be the different compilers differ more these days alone for the fact mitigating CPU vulnerabilities at compiler level if i understand the mitigations correctly.

v1.4.0 from the Foobar2000 'Encoders Pack' (on Wine v6.13-staging) on a i5-3550 desktop quad-core CPU on some random music I got of 3hr4min2sec in length and just going by what I last see in the Foobar2000 window for current time shown before the window disappears...

-5 / 26sec / 1,064,210,521 bytes
-6 / 31sec / 1,063,104,966 bytes
-7 / 33sec / 1,058,278,040 bytes
-8 / 42sec / 1,057,783,228 bytes

Takeaways...

-FLAC7 is probably the sweet-spot when it comes to time vs file size combo as it's barely larger (a bit shy of 0.5MB) than FLAC8 but is 9 seconds quicker as you would have to go to FLAC5 for a worthwhile speed improvement, but then you would lose a bit over 5.9MB in size.

-FLAC6 is useless given it's barely faster (2 sec) than FLAC7 but is 4.8MB larger.

-While a bit more debatable, FLAC5 is probably better than FLAC6 given the 5sec speed improvement but only sacrifices about 1.1MB. so if someone goes with this mindset, FLAC6 don't really win any category here because FLAC8 wins in smallest file size, FLAC7 wins comfortably in the 'time vs file size' combo and FLAC5 has a worthwhile compression speed improvement where as FLAC6's best chance is against FLAC5 but even this is not really clear cut, but would probably favor FLAC5's time savings over the small enough file size difference between the two. although I guess a scenario that would favor FLAC6 over FLAC5 would be if someone was uploading it on a slower internet line, but then again at this point, FLAC7/8 would save even further, which still makes FLAC6 useless.

so while FLAC7 looks good, in the end, with modern computing power and storage space, it's not really going to matter too much on what someone chooses unless maybe if you plan on uploading a fair amount of that stuff online for backup in which case the smallest file size will be best, especially if you don't have a fast upload. but I guess if someone had a boatload of WAV files to compress or FLAC to re-compress, then FLAC7 might be slightly preferred.

p.s. still, I pretty much prefer FLAC8 in general since it's not like we spend tons of time compressing in general and I might as well get smallest file size (especially when compression times are still easily fast enough as there ain't much real world difference for there to be any real negative by using -8) for storage efficiency sake.

Quote from: john33 on 2022-09-16 10:11:10

Here is another compile that certainly appears faster on my system although no empirical measurements taken. It is compiled using the Intel c++ 2022 compiler.
www.rarewares.org/files/lossless/flac-1.4.0-x64-Intel-2022.zip (http://www.rarewares.org/files/lossless/flac-1.4.0-x64-Intel-2022.zip)

No difference from the one posted on Reply #34 (https://hydrogenaud.io/index.php/topic,122949.msg1015520.html#msg1015520) on my PC (https://hydrogenaud.io/index.php/topic,122949.msg1015615.html#msg1015615).
And yes, NetRanger's CLANG15 is the slowest one in the whole thread on my i3-12100. Would like to see some Ryzen results.

Just tried -7 on a concatenated CDDA file, 4h43m29s. All previous tests were also using a long single file and therefore single threaded.
Case: 49s
Xiph: 51s
NetRanger GCC v12.2.0: 1m1s
NetRanger CLANG15: 1m2s

Here are -8:
Case: 1m8s
Xiph: 1m8s
NetRanger GCC v12.2.0: 1m23s
NetRanger CLANG15: 1m32s

Ryzen? Coming up in a few days, a friend's Acer laptop is running over the week-end while they are away ... under the all-too-bold assumption that I didn't make my usual .bat bug.

I recall my once fascination for compiling on own platform, before laziness got the better of me - anyone else remembering how long to took to get a Gentoo compiling its way through everything in the early noughties? Fast forward to this Windows 10 work laptop, which even with OS preinstalled took a week of updates and reboots, I guess history surely rhymes.

v1.4.0 from the Foobar2000 'Encoders Pack' (on Wine v6.13-staging) on a i5-3550 desktop quad-core CPU on some random music I got of 3hr4min2sec in length and just going by what I last see in the Foobar2000 window for current time shown before the window disappears...

-5 / 26sec / 1,064,210,521 bytes
-6 / 31sec / 1,063,104,966 bytes
-7 / 33sec / 1,058,278,040 bytes
-8 / 42sec / 1,057,783,228 bytes

Was this one big file (making encoding effectively singlethreaded?)

EDIT: My calculations suggest -8 is 262x, which means over 4 minutes of audio encoded per second.  If this is singlethreaded, if you use -8 with 4 threads, then you'd probably get over 1000x.

EDIT#2: If you're ripping CDs, given that ripping the CD is likely to be the bottleneck by far even on lesser CPUs, I'd recommend -8 in almost all those cases.

EDIT#2: If you're ripping CDs, given that ripping the CD is likely to be the bottleneck by far even on lesser CPUs, I'd recommend -8 in almost all those cases.

Depends on the application? Does EAC still run the compression after ripping the entire CD? I don't remember whether you can start ripping the next while it still compresses ...?
dBpoweramp works tracks-based. IIRC, only the paid version uses multiple CPU threads (one for each track), so at worst you will have to wait for the last to complete.

In any case, won't be ripping more CDs in a day than you can re-encode with -8p overnight. Well maybe if you use automated CD changers, when you will surely be busy verifying the metadata.

Here is another compile that certainly appears faster on my system although no empirical measurements taken. It is compiled using the Intel c++ 2022 compiler.
www.rarewares.org/files/lossless/flac-1.4.0-x64-Intel-2022.zip (http://www.rarewares.org/files/lossless/flac-1.4.0-x64-Intel-2022.zip)

Practically tied to your earlier x64 bild. Compared to the figures in reply 75 (https://hydrogenaud.io/index.php/topic,122949.msg1015694.html#msg1015694), this one gets -5 down from 300 to 298, and -7p down from 1035 to 1029.
Again, median of three runs. Pretty consistent numbers - 1016, 1029, 1036 seconds for -7p.

Extreme (> 20 percent) 1.4.0 benefits found in the wild - by coincidence, and without going to 88.2 kHz or higher. 
Yes 48/24 files, but also a few 44.1/24 that save some ten percent.

https://lustmord.bandcamp.com/album/alter is available at "name your price".
(If you want to buy at Bandcamp, wait until "Bandcamp Friday" (https://isitbandcampfriday.com/) where they waive their cut. I've started buying from Boomkat instead after Bandcamp canceled my purchase post payment just because.)

A bit trying back and forth suggests they used -6, that gives straight 1.000 ratios.

Now look at 1.4.0 at -3:

C:\tmp\Lustmord & Karin Park - ALTER> C:\bin\flac-1.4.0-win\Win64\flac.exe -3f *.flac

flac 1.4.0
Copyright (C) 2000-2009  Josh Coalson, 2011-2022  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

Lustmord & Karin Park - ALTER - 01 Hiraeth.flac: wrote 52035408 bytes, ratio=0,981
Lustmord & Karin Park - ALTER - 02 The Void Between.flac: wrote 59625894 bytes, ratio=0,899
Lustmord & Karin Park - ALTER - 03 Perihelion.flac: wrote 32100437 bytes, ratio=0,845
Lustmord & Karin Park - ALTER - 04 Twin Flames.flac: wrote 83337766 bytes, ratio=0,914
Lustmord & Karin Park - ALTER - 05 Entwined.flac: wrote 75526535 bytes, ratio=0,956
Lustmord & Karin Park - ALTER - 06 Kindred.flac: wrote 53892745 bytes, ratio=0,839
Lustmord & Karin Park - ALTER - 07 Song of Sol.flac: wrote 55486784 bytes, ratio=0,898
Lustmord & Karin Park - ALTER - 08 Sele.flac: wrote 64500002 bytes, ratio=0,879

... saves nearly ten percent on the total.

Deleting, re-extracting from .zip, and running -8p (no "e" ...)

PS C:\tmp\Lustmord & Karin Park - ALTER> C:\bin\flac-1.4.0-win\Win64\flac.exe -8pf *.flac

flac 1.4.0
Copyright (C) 2000-2009  Josh Coalson, 2011-2022  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

Lustmord & Karin Park - ALTER - 01 Hiraeth.flac: wrote 47871959 bytes, ratio=0,903
Lustmord & Karin Park - ALTER - 02 The Void Between.flac: wrote 53662309 bytes, ratio=0,809
Lustmord & Karin Park - ALTER - 03 Perihelion.flac: wrote 28090623 bytes, ratio=0,739
Lustmord & Karin Park - ALTER - 04 Twin Flames.flac: wrote 71762047 bytes, ratio=0,787
Lustmord & Karin Park - ALTER - 05 Entwined.flac: wrote 65022030 bytes, ratio=0,823
Lustmord & Karin Park - ALTER - 06 Kindred.flac: wrote 48006888 bytes, ratio=0,748
Lustmord & Karin Park - ALTER - 07 Song of Sol.flac: wrote 48319869 bytes, ratio=0,782
Lustmord & Karin Park - ALTER - 08 Sele.flac: wrote 55920678 bytes, ratio=0,763

... and no, the 1.3.4 at -8pf doesn't do wonders, it is the 1.4.0. Download and test for yourself.
-e not tested. Probably makes a difference for 1.3.4.

I came up with this by the following coincidence: Spotify on my soon-falling-apart desktop computer suggested a new Lustmord tribute/remix CD with some damn interesting names, and then a crash. Went to laptop, googled, found it at "name your price" together with the one above: https://lustmord.bandcamp.com/album/the-others-lustmord-deconstructed

OK, so this was the one I actually googled for - and it is 44.1/24. Again, testing with -6 gives straight 1.000 ratios, and let's hit it with 1.4.0 at -8pf (again, no "e" here):

Lustmord - The Others -Lustmord Deconstructed- - 01 Enslaved - Eon.flac: wrote 42854726 bytes, ratio=0,922
Lustmord - The Others -Lustmord Deconstructed- - 02 MONO - Er Eb Os.flac: wrote 76787106 bytes, ratio=0,873
Lustmord - The Others -Lustmord Deconstructed- - 03 Ihsahn - Dark Awakening.flac: wrote 39634289 bytes, ratio=0,923
Lustmord - The Others -Lustmord Deconstructed- - 04 Jo Quail - Prime.flac: wrote 74264091 bytes, ratio=0,900
Lustmord - The Others -Lustmord Deconstructed- - 05 Bohren & der Club of Gore - Plateau.flac: wrote 118575065 bytes, ratio=0,986
Lustmord - The Others -Lustmord Deconstructed- - 06 hackedepicciotto - Trinity Past.flac: wrote 65398218 bytes, ratio=0,905
Lustmord - The Others -Lustmord Deconstructed- - 07 Ulver - Godeater.flac: wrote 113619735 bytes, ratio=0,981
Lustmord - The Others -Lustmord Deconstructed- - 08 Jonas Renkse - Er Eb Os.flac: wrote 70867405 bytes, ratio=0,971
Lustmord - The Others -Lustmord Deconstructed- - 09 Zola Jesus - Prime.flac: wrote 37000037 bytes, ratio=0,889
Lustmord - The Others -Lustmord Deconstructed- - 10 Spotlights - Of Eons.flac: wrote 76565255 bytes, ratio=0,935
Lustmord - The Others -Lustmord Deconstructed- - 11 The Ocean - Primal -State of Being-.flac: wrote 116101117 bytes, ratio=0,994
Lustmord - The Others -Lustmord Deconstructed- - 12 CROWN - Element.flac: wrote 58738240 bytes, ratio=0,999
Lustmord - The Others -Lustmord Deconstructed- - 13 Jaye Jayle - Er Eb Es.flac: wrote 44316148 bytes, ratio=0,928
Lustmord - The Others -Lustmord Deconstructed- - 14 Godflesh - Ashen.flac: wrote 70261802 bytes, ratio=0,979
Lustmord - The Others -Lustmord Deconstructed- - 15 Steve von Till a.k.a. Harvestman - Testament.flac: wrote 79120831 bytes, ratio=0,994
Lustmord - The Others -Lustmord Deconstructed- - 16 Årabrot - The Last Days -See the Light-.flac: wrote 86782738 bytes, ratio=0,997

Still three tracks do hit the ten percent mark. Again, it is the 1.4.0 that makes the difference; recompressing the download by 1.3.4 at -8pf gives at best .992.

44.1/24

I'm curious to see if the improvements persist if the bit depth is reduced to 16 (perhaps without dithering?).

EDIT: My calculations suggest -8 is 262x, which means over 4 minutes of audio encoded per second.  If this is singlethreaded, if you use -8 with 4 threads, then you'd probably get over 1000x.

The results are obviously multithreaded and foobar does it by default, otherwise a 2012 Ivy Bridge (no AVX2 support if relevant) would be faster than a 2022 Alder Lake.
https://hydrogenaud.io/index.php/topic,122949.msg1015750.html#msg1015750

Even a 2017 Kaby Lake i7 is slower than a 12th Gen i3 in both single and multithread performance.
https://cpu.userbenchmark.com/Compare/Intel-Core-i7-7700K-vs-Intel-Core-i3-12100/3647vs4126

v1.4.0 from the Foobar2000 'Encoders Pack' (on Wine v6.13-staging) on a i5-3550 desktop quad-core CPU on some random music I got of 3hr4min2sec in length and just going by what I last see in the Foobar2000 window for current time shown before the window disappears...

-5 / 26sec / 1,064,210,521 bytes
-6 / 31sec / 1,063,104,966 bytes
-7 / 33sec / 1,058,278,040 bytes
-8 / 42sec / 1,057,783,228 bytes

With foobar, go to View -> Console after encoding, this shows the encoding time and speed (x). It is also necessary to know the source formats and types of used storage media, these factors affect speed too. My tests used .wav source and RAM drive. If there is not enough RAM then at least SATA SSD should be used.

flac.exe in the Free Encoder Pack is x86 and slower than Case/Xiph's x64 builds on my PC as well.

Just tried -7 on a concatenated CDDA file, 4h43m29s. All previous tests were also using a long single file and therefore single threaded.
Case: 49s
Xiph: 51s
NetRanger GCC v12.2.0: 1m1s
NetRanger CLANG15: 1m2s

Here are -8:
Case: 1m8s
Xiph: 1m8s
NetRanger GCC v12.2.0: 1m23s
NetRanger CLANG15: 1m32s

Free encoder pack with the same test:
-7 1m
-8 1m21s

@Porcus: What does the "f" do in your compression setting (other than -f, --force : Force overwriting of output files)?

I'm curious to see if the improvements persist if the bit depth is reduced to 16 (perhaps without dithering?).

Go ahead folks, download for free - and buy some other Lustmord first Friday in October  :)

With foobar, go to View -> Console after encoding, this shows the encoding time and speed (x). It is also necessary to know the source formats and types of used storage media, these factors affect speed too. My tests used .wav source and RAM drive. If there is not enough RAM then at least SATA SSD should be used.

If using foobar2000 for speed testing, beware the tag transfer options - if checked, that takes the extra time of writing the file anew. Likely nothing to talk about on RAM drive, likely not much on a SATA SSD.
(For a "real-life timing", one should likely do all the tag transfers, because that is what you would do in practice. But that is another story.)

@sundance : Yes, overwrites. flac -f filename.flac will "recompress" the flac file: Write to a temp, and then delete the .flac and rename the temp.
I just discovered that flac -f filename.wav will first delete filename.flac and start writing filename.flac anew, not using a temp. If process aborts, this will leave you with a corrupted flac, while aborting recompression will leave you with the old one.

Go ahead folks, download for free

Well, I did, and they aren't 44.1/24 but 48/24. I'm not really sure how you managed that blunder. :P

The Lustmord & Karin Park album - 48/24 indeed - has some extreme measurements that I would not expect at that sample rate:

OK, reading down to TAK, it isn't that outrageous. Not the most WavPack friendly album - and TAK usually doesn't beat Monkey's High with that much of a margin. But still ... sure, not every album makes for the average size order.

But then? WavPack's "x4" doing well happens to high resolution. But FLAC ... is let's say it is usually not there in the order. And that improvement? The "-8" windowing functions have changed, but 1/6th at -5 is a lot.

And since you asked for 16 bit undithered:
... still 12 percent size gain from 1.3.1 at -5 to 1.4.0 at -5, and now also beating WavPack -hhx4.

Quote from: Porcus on 2022-09-17 08:51:22

Go ahead folks, download for free

Well, I did, and they aren't 44.1/24 but 48/24. I'm not really sure how you managed that blunder. :P

Well, you know, second line I wrote ...

Extreme (> 20 percent) 1.4.0 benefits found in the wild - by coincidence, and without going to 88.2 kHz or higher. 
Yes 48/24 files, but also a few 44.1/24 that save some ten percent.

[...]
OK, so this was the one I actually googled for - and it is 44.1/24.

The remix/tribute album is 44.1/24. The Lustmord & Karin Park album - with the giant differences - is 48/24. But from 44.1 to 48 isn't an ocean ...?

Well, you know, second line I wrote ...

Quote from: Porcus on 2022-09-16 23:34:35

Extreme (> 20 percent) 1.4.0 benefits found in the wild - by coincidence, and without going to 88.2 kHz or higher. 
Yes 48/24 files, but also a few 44.1/24 that save some ten percent.

[...]
OK, so this was the one I actually googled for - and it is 44.1/24.

Ah, I misremembered what you wrote. Sorry about that.

Either way, for this album, it is really not surprising that 1.4.0 does so much better than 1.3.4. This is the spectrum of a rather representative section around the middle of song #2 on the album:

Spoiler (click to show/hide)
Some sections are even more extreme, like this one in song #3:

Spoiler (click to show/hide)
Generally these songs are fairly lacking in high frequencies.

Could those humps around 22k be artefacts from what once was (re) sampled at 44100?
Not necessarily so that the final product is an upsample, it could be, say, the vocal tracks.

Could those humps around 22k be artefacts from what once was (re) sampled at 44100?

I don't think so. It looks like it's just shaped dither noise from when they were converted from the recording/editing precision of 32-bit to 24-bit. That exact same profile appears with perfect consistency throughout all the tracks. Here's the spectrum of the last 17.2 seconds of the last song of the album (#8):

Spoiler (click to show/hide)

Quote from: ThaCrip on 2022-09-16 16:53:47

v1.4.0 from the Foobar2000 'Encoders Pack' (on Wine v6.13-staging) on a i5-3550 desktop quad-core CPU on some random music I got of 3hr4min2sec in length and just going by what I last see in the Foobar2000 window for current time shown before the window disappears...

-5 / 26sec / 1,064,210,521 bytes
-6 / 31sec / 1,063,104,966 bytes
-7 / 33sec / 1,058,278,040 bytes
-8 / 42sec / 1,057,783,228 bytes

Was this one big file (making encoding effectively singlethreaded?)

EDIT: My calculations suggest -8 is 262x, which means over 4 minutes of audio encoded per second.  If this is singlethreaded, if you use -8 with 4 threads, then you'd probably get over 1000x.

EDIT#2: If you're ripping CDs, given that ripping the CD is likely to be the bottleneck by far even on lesser CPUs, I'd recommend -8 in almost all those cases.

I get why you asked because if that were true the speed would be a lot less. but It's 40 standard (16/44.1) FLAC music files of 3hr4min2sec in total re-encoded to FLAC v1.4.0 (from about half at FLAC v1.3.3 and other half at FLAC v1.3.2) with the Foobar2000 Endoder's Pack. so it's multi-threaded. I got a i5-3550 CPU (Foobar2000 is running through Wine v6.13-staging as I suspect different Wine versions may change speed a bit(?)) which is four cores and you can see the CPU load is high during conversion.

With foobar, go to View -> Console after encoding, this shows the encoding time and speed (x). It is also necessary to know the source formats and types of used storage media, these factors affect speed too. My tests used .wav source and RAM drive. If there is not enough RAM then at least SATA SSD should be used.

flac.exe in the Free Encoder Pack is x86 and slower than Case/Xiph's x64 builds on my PC as well.

I found the Xiph's FLAC v1.4.0 and used the 'flac.exe' in the 'Win64' folder and had to extract the 'flac.exe' along with 'libFLAC.dll' to Foobar2000's 'encoders' folder as just the 'flac.exe'(64bit) alone would throw errors.

but at least based on reading the files from a 6TB hard drive and writing to another 5TB hard drive, the time is pretty much the same... "0:41.522, 265.93x realtime", ran it again "0:40.479, 272.79x realtime". so generally 40-41 seconds which is basically the same as the default from 'Encoders Pack'. so I might as well stick with the default 'Fobar2000's Encoders Pack' FLAC version given there is no real differences, at least on standard hard drive usage.

but just as a quick test I decided to try converting back to WAV temporarily. so basically the WAV files are on 6TB hard drive and then from there I converted those WAV files back to FLAC8 to a 5TB hard drive and pretty much same results (i.e. "0:41.710, 264.74x realtime"). but converting from FLAC (on 6TB HDD) back to WAV (on 5TB HDD) things, as expected, were faster "0:23.562, 468.65x realtime" as you could see the speed really shoot up at times and then pause briefly and resume etc, which I suspect might be the CPU waiting for hard drive to catch up(?). but the average speed is what it shows there at 468x.

but I figure if the hard drive becomes the bottleneck, and one needs a RAM drive or SSD to get true max speed, at that point it's probably not worth proceeding further simply because I typically use my hard drives over the SSD (my main boot drive is a Samsung 850 EVO 250GB SSD (SATA)) for general data writing like FLAC or other larger files like video etc.

thanks for that info.

p.s. but using the default Foobar2000's encoders pack LAME(32bit) vs the 64bit one, the 64bit does shave some time off by about 5-6 seconds at LAME V5. so there is a legitimate difference here even though not much.

Yeah, I just used the 64 bit build of 1.4.0 from Rarewares on one long file on an i7-4770 with a SSD and an unreasonable amount of RAM using Wine 5.0.3 on Debian stable and got something like 100x or so using -8.

I'm kinda curious how much better/worse a native compile of FLAC would do, but not enough to actually try to build FLAC myself.

Honestly, though, I'd still use -8 if I was ripping CDs as in practice you'd probably only be talking a second or two difference per CD given that unless you're ripping from multiple drives at once the CD drive is more or less always the bottleneck except for the last track.

If you're, say, encoding 10 Grateful Dead shows in one batch, though...

Noise / experimental album by Henrik Nordvargr: both -r 8 and high subdivide_tukey works very well here
268 597 254 bytes (~932 kbps): flac -8 -e -p
266 886 449 bytes (~926 kbps): wavpack -hhx6
266 472 563 bytes (~924 kbps): flac -l 12 -b 4096 -m -r 8 -A subdivide_tukey(3) -e -p
265 899 335 bytes (~922 kbps): flac -l 12 -b 4096 -m -r 8 -A subdivide_tukey(9) -p

Times? In particular, which was the slowest of the two latter? The size gains of -e seem to have dwindled on most CDDA-resolution material.
Also, CPU ... and album, Sir Björk(k) has released a few.

(The two latter are synonymous to the following, "-8" already covering the rest
-8ep -r 8
-8p -r 8 -A subdivide_tukey(9)
)

Times? In particular, which was the slowest of the two latter? The size gains of -e seem to have dwindled on most CDDA-resolution material.
Also, CPU ... and album, Sir Björk(k) has released a few.

Album is "Otyglad Best". -8 -r 8 -A subdivide_tukey(3) -e -p is slowest, -8 -r 8 -A subdivide_tukey(9) -p slightly faster and ~2.5 times slower than hhx6.

~2.5 times slower than hhx6.

Oh, but ... https://hydrogenaud.io/index.php/topic,120158.msg1015896.html#msg1015896 .  speed=4.08e-06x

Anyway, this speed is impractical and shouldn't be of much use (though, interesting for testing purposes). There is material where -r 8 helps, but at https://hydrogenaud.io/index.php/topic,120158.msg1003738.html#msg1003738 I found the size impact atop -8 -A lotsofthem to be very small (~ 1/1000 of a percent) and of ambiguous sign. Note that the "4" in "Ax4" is my big misinterpretation, and anyway obsoleted by subdivide_tukey; my point was anyway how the double precision fix has killed -e.
Which was also the reason why I asked about those timings of yours, and they agreee: there is a way to outcompress -e in less time.

but at least based on reading the files from a 6TB hard drive and writing to another 5TB hard drive, the time is pretty much the same... "0:41.522, 265.93x realtime", ran it again "0:40.479, 272.79x realtime". so generally 40-41 seconds which is basically the same as the default from 'Encoders Pack'. so I might as well stick with the default 'Fobar2000's Encoders Pack' FLAC version given there is no real differences, at least on standard hard drive usage.

So around 46MB/sec, unlikely to bottleneck transfer between two separate HDDs unless the disks are nearly full. Running the same set of files repeatedly also reduces I/O bottleneck due to OS caching.

Speedups for x86_64 CPUs having the FMA instruction set extention are added

I don't know about Wine but i5-3550 does not support FMA, which the Xiph build uses. However single thread speed may not be as simple as dividing the results by 4 due to Intel Turbo Boost and other unknown factors.

but just as a quick test I decided to try converting back to WAV temporarily. so basically the WAV files are on 6TB hard drive and then from there I converted those WAV files back to FLAC8 to a 5TB hard drive and pretty much same results (i.e. "0:41.710, 264.74x realtime"). but converting from FLAC (on 6TB HDD) back to WAV (on 5TB HDD) things, as expected, were faster "0:23.562, 468.65x realtime" as you could see the speed really shoot up at times and then pause briefly and resume etc, which I suspect might be the CPU waiting for hard drive to catch up(?). but the average speed is what it shows there at 468x.

For decoding speed test (foobar does not use flac.exe for decoding), one may use foo_benchmark as it does not write files to disk. Someone's results with i5-3210M at around 800x with single thread:
https://www.audiosciencereview.com/forum/index.php?threads/wav-pcm-vs-flac.26171/post-896508

If decoding from a slow drive (or over a slow connection), then recompressing flac -> flac means that less data has to be transferred compared to compressing wave/aiff -> flac. So flac -> flac could be faster in real-life conditions.

800x decoding speed at CDDA translates is very fast indeed, translating to 141 megabytes per second; yet that falls short of SATA throughput figures. But how are those in a real-life condition if you use the same bus to read and write?

The test below used this SSD, 256GB version, which I purchased in 2015:
https://www.anandtech.com/show/7908/adata-sp920-128gb-256gb-512gb-1tb-review

The files used are 5 separate CDDA flac images without metadata (they are on separate .cue files), total duration 4:32:22

The flac files are copied to the SSD, then Windows is rebooted to eliminate cache, and converted by foobar using Case's flac.exe with -8. Windows is also installed in the same SSD because I only have 1 HDD and 1 SSD in my machine.

Total encoding time: 0:22.250, 734.46x realtime

As a test control the same 5 images are also converted to .wav, rebooted, and encoded to flac -8.

Total encoding time: 0:20.969, 779.32x realtime

That encodes a month of audio in an hour.  Which is ... uh ... fast.

And you had five images, but you surely don't have five CPU cores/threads. If you have four, then fb2k will encode the first four and start on the fifth when the first is done - it won't parallelize in the end, so "if all four are equal then five take the same time as eight"?

(Sure it might be that the otherwise-least-busy core will finish first and then the fifth will get the presumed fastest, blah blah blah.)

Believe it or not, I've had people complaining that FLAC defaults to requiring SSE2, which was introduced in 2000.

That is true for Intel. AMD introduced SSE2 I think in 2004? But still a lot of time ago...

That encodes a month of audio in an hour.  Which is ... uh ... fast.

And you had five images, but you surely don't have five CPU cores/threads. If you have four, then fb2k will encode the first four and start on the fifth when the first is done - it won't parallelize in the end, so "if all four are equal then five take the same time as eight"?

(Sure it might be that the otherwise-least-busy core will finish first and then the fifth will get the presumed fastest, blah blah blah.)

The SSD currently has 107GB of free space, and I don't have over 100GB of lossless CDDA files unless I also convert my MIDI, chiptunes and game rips to audio with over one year of duration. Of course I won't do this, it will significantly shorten the life of my 7 years old SSD.

i3-12100 is 4C8T, with standard CPU-Z benchmark like this in my previous post:
(https://hydrogenaud.io/index.php?action=dlattach;attach=23346;image)

For Intel, AVX2 and FMA3 were introduced in Gen4 Core i Haswell CPUs in 2013, and my CPU looks like this in AVX2 benchmark:


In either case, they have similar Multi Thread Ratio. My results are similar to others too:
https://wolflsi.pixnet.net/blog/post/69948036-intel%E7%AC%AC%E5%8D%81%E4%BA%8C%E4%BB%A3alder-lake-core-i3-12100%E7%B0%A1%E6%B8%AC


I upgraded in February and only bought the minimum required hardware (CPU, mainboard, 2x8GB DDR4 3200), everything else is old and the m.2 slot is still empty. For a brand new system I would probably get a 1TB NVMe SSD.

I don't know / remember the encoding settings of the old flac files, but here are the file sizes:

wav
2882686684 bytes

Old flac
1566101628 bytes

New flac 1.4.0 at -8
1559797419 bytes

CD1: Loud pop songs + some speech tracks.
https://vgmdb.net/album/69558

CD2: Decently mastered pop songs.
https://vgmdb.net/album/5216

CD3: Decently mastered synth music, mainly fast-paced.
https://vgmdb.net/album/1253

CD4: Decently mastered synth music, but more calm.
https://vgmdb.net/album/2155

CD5: Classical music, sometimes contain narrations.

That CPU has 4 physical cores and 8 threads. fb2k encoded all five albums at once then? Not unlikely it would be even faster if you had run eight albums.

No wonder people use -p.
(And -e, but they shouldn't for 1.4.0. Preliminary testing suggests that even -8pe can be equaled at size in shorter time.)

Again simply using flac.exe within CUEtools on a long album ends at following speeds for a Ryzen 5900x using -8 -p.

Xiph official 98x
Netranger GCC 12.2 90x
Case 83x
john33 Intel 2022 76x
Netranger CLANG 15 52x

Without Ryzen, Intel i3 could still be 2C4T in 2022, and Meltdown hurt Intel's reputation a lot too. So I wonder why Wombat is the only one who posted Ryzen results until now.

I once even was official AMD Reseller when Slot A was king. I still have an AMD file folder around here. This is more than 20 years back when i had my own business.

Some size results from some overnight runs, aggregates over 38 CDs.

-e is dead, did I say that loud enough?
-8pe made bigger files and is five times slower than -8p -A subdivide_tukey(5) (you can increase to subdivide_tukey(7) and still save time).
-8e made bigger files slower than -8p.  And in line with the previous, yes -A subdivide_tukey(5) could beat -e at size here too.
"-p" has something for it. The size differences aren't big at this level, but -8 to -8p size difference was pretty much spot-on double the one from -8 to -8e.
I didn't get any -8 -A subdivide_tukey(N) to touch -p size-wise, and I stopped at (8) when it looks like it got slower than -p

-r, tested only on -7, -7p, -8, -8p.
As should, -r8 makes smaller files than -r7 than -r 6 [which -7 and -8 use] than -r 5. I've earlier had ambiguous results with -r 7 vs -r 8.
The size effect of changing -r is small. Smaller than -p and smaller than going -7 to -8 and -8 to -8 -A subdivide_tukey(4) (which is the next).

I wonder why Wombat is the only one who posted Ryzen results until now.

Week-end crash. Will find some.

Shouldn't the exhaustive model search be trying all of them, and then picking the best one? It seems very strange to me that -e makes the compression worse; the worst case scenario should be that the file size remains the same, because the non-exhaustive search was already using the best model, but -e making the size go up is quite counter intuitive.

-e making the size go up

Well, it doesn't. Read Porcus's message again - none of the comparisons are between [some parameters with e] and [those exact same parameters except e].

What is true is that the addition of -e does not achieve as much of a compression improvement as does the addition of another parameter, such as -A subdivide_tukey(5) or -p.

Yep. -e reduces size, but not so much as -p or "going up two subdivide_tukey steps" will do (subdivide_tukey(3) is included in -8 already) - and -e spends more time achieving less.

Note that the "exhaustive" in -e doesn't cover all modelling aspects. It brute-forces part of it, whereas -p brute-forces another part. subdivide_tukey(5) only makes for the encoder to estimate more function choices before picking the guesstimated best - and it seems that trying more (without full accuracy) is more efficient than trying fewer but brute-forcing them.

Scott Lawlor and Dronny Darko "Polar Night on Titan" (2015) (https://archive.org/details/petroglyph358ScottLawlor_DronnyDarko-PolarNightOnTitan). Dark ambient, drone

100.0% - 765.928.844 bytes
34.30% - 262.710.949 bytes (484 kbps): FLAC 1.3.4 -5
33.80% - 258.918.920 bytes (477 kbps): FLAC 1.3.4 -8
33.75% - 258.472.525 bytes (476 kbps): FLAC 1.3.4 -8p
33.26% - 254.765.189 bytes (469 kbps): FLAC 1.3.4 -8pe
33.09% - 253.431.855 bytes (467 kbps): FLAC 1.4.0 -3
32.96% - 252.451.802 bytes (465 kbps): WavPack 5.5.0 -hh -x1
32.91% - 252.139.633 bytes (465 kbps): ffmpeg -compression_level 3 -frame_size 4096
32.71% - 250.531.950 bytes (462 kbps): CUETools.Flake 2.2.2 -2
32.68% - 250.301.234 bytes (461 kbps): CUETools.Flake 2.2.2 -3
32.66% - 250.157.962 bytes (461 kbps): ffmpeg -compression_level 5 -frame_size 4096
32.53% - 249.155.643 bytes (459 kbps): ffmpeg -compression_level 8 -frame_size 4096
32.51% - 248.997.530 bytes (459 kbps): ffmpeg -compression_level 10 -frame_size 4096
32.49% - 248.821.318 bytes (458 kbps): FLAC 1.4.0 -4
32.48% - 248.753.600 bytes (458 kbps): FLAC 1.4.0 -5
32.46% - 248.611.049 bytes (458 kbps): FLAC 1.4.0 -6
32.45% - 248.531.637 bytes (458 kbps): CUETools.Flake 2.2.2 -4
32.44% - 248.444.048 bytes (458 kbps): CUETools.Flake 2.2.2 -5
32.42% - 248.321.082 bytes (458 kbps): CUETools.Flake 2.2.2 -6
32.42% - 248.305.277 bytes (457 kbps): CUETools.Flake 2.2.2 -7
32.37% - 247.965.156 bytes (457 kbps): ffmpeg -compression_level 10 -frame_size 4096 -lpc_type 3 -lpc_passes 10 -exact_rice_parameters 1
32.35% - 247.791.048 bytes (457 kbps): CUETools.Flake 2.2.2 -8
32.32% - 247.585.181 bytes (456 kbps): ffmpeg -compression_level 10 -lpc_type 3 -lpc_passes 10 -exact_rice_parameters 1
32.28% - 247.269.005 bytes (456 kbps): CUETools.Flake 2.2.2 -8 -e 32 -s search --window-method search
32.28% - 247.264.261 bytes (456 kbps): FLAC 1.4.0 -7
32.27% - 247.197.583 bytes (455 kbps): FLAC 1.4.0 -8
32.23% - 246.889.401 bytes (455 kbps): FLAC 1.4.0 -8p
32.22% - 246.789.073 bytes (455 kbps): FLAC 1.4.0 -8 -A subdivide_tukey(9) -p
32.20% - 246.664.559 bytes (454 kbps): FLAC 1.4.0 -8pe
32.06% - 245.520.054 bytes (452 kbps): WavPack 5.5.0 -hh -x4
31.92% - 244.459.886 bytes (450 kbps): WavPack 5.5.0 -hh -x6
30.86% - 236.360.371 bytes (435 kbps): TAK 2.3.3 -p4m

Scott Lawlor and Dronny Darko "Polar Night on Titan" (2015) (https://archive.org/details/petroglyph358ScottLawlor_DronnyDarko-PolarNightOnTitan). Dark ambient, drone
(...)
30.86% - 236.360.371 bytes (435 kbps): TAK 2.3.3 -p4m

TAK FTW!

33.26% - 254.765.189 bytes (469 kbps): FLAC 1.3.4 -8pe
33.09% - 253.431.855 bytes (467 kbps): FLAC 1.4.0 -3

Sapienti sat.

I see. I misinterpreted this:

-8pe made bigger files and is five times slower than -8p -A subdivide_tukey(5) (you can increase to subdivide_tukey(7) and still save time).

as -8pe also containing subdivide_tukey(5), thus the addition of -e made things worse, but it was actually the omission of -A.

-8 includes -A subdivide_tukey(3).

I see above that -e still works somewhere out in space (that album title does give you the right idea of the content yes).
Question though: is ffmpeg -frame_size 4096 (or 4608) subset until compression level 10 but not until 12?

Of course, TAK slays everything that's fast and a lot of slower codecs too. Filling some numbers into @Gravity Stupor 's list, those in italics are mine.
246.889.401 bytes (455 kbps): FLAC 1.4.0 -8p
246 867 738 bytes for -8p -r8, that helps only a a tiny bit
246.789.073 bytes (455 kbps): FLAC 1.4.0 -8 -A subdivide_tukey(9) -p
246.664.559 bytes (454 kbps): FLAC 1.4.0 -8pe
[...]
237 725 244 Monkey's "Insane" is overTAKen by something faster
236 462 439 OptimFROG at default (= -- preset 2)
236.360.371 bytes (435 kbps): TAK 2.3.3 -p4m
235 190 251 OptimFROG --preset 3

Question though: is ffmpeg -frame_size 4096 (or 4608) subset until compression level 10 but not until 12?

ffmpeg works a little different than FLAC.

• ffmpeg uses a block time instead of a blocksize, which is then translated to the nearest standard blocksize. Blocktime for settings 0, 1 and 2 is 27ms, for settings 3 until 12 it is 105ms. For 44.1kHz and 48kHz this means blocksize is 4608 for settings 3 and up
• Predictor order is 8 for levels 3, 4, 5, 6 and 7, is 12 for levels 8, 9 and 10 and 32 for levels 11 and 12
• -e is set for levels 10 and 12. Estimation (as FLAC uses without -e) is at levels 3, 4 and 5. Levels 6, 7, 8, 9 and 11 use something in between
• -p has no equivalent in ffmpeg

TL;DR: ffmpeg compression levels 11 and 12 for FLAC are non-subset when used for audio with a samplerate of 48kHz or lower.

Ah, thanks. So ffmpeg selects subset-compliant block sizes up to ... found in this post (https://hydrogenaud.io/index.php/topic,120158.msg1003304.html#msg1003304), 192000. Above which, -frame_size 16384 can enforce it. Predictor order on the other hand, cannot be overridden by user (at least not by any documented option) and is what throws 11 and 12 off subset.

The testing I have done on a limited 96 kHz corpus indicates that ffmpeg's blocking strategy is quite sane: Reference FLAC could benefit from setting block size to 8192 for 96 kHz files (https://hydrogenaud.io/index.php/topic,120158.msg1001334.html#msg1001334); you suggested that windowing could do the same (https://hydrogenaud.io/index.php/topic,120158.msg1001823.html#msg1001823), and I if that holds up to testing, I guess that those who are willing to wait for creative smaller-size-than-8 combinations might be satisfied even if it takes longer time running all the functions.
Also that testing suggests that for CDDA, reference FLAC is so well tuned to its 4096 choice that 4608 is even slightly harmful. Is anyone curious enough to test 4608 vs 4096 on 48 kHz? ;-)

Predictor order on the other hand, cannot be overridden by user (at least not by any documented option) and is what throws 11 and 12 off subset.

I guess it is time to show you some more documentation then  :))  O:)

$ ./ffmpeg -h encoder=flac
ffmpeg version n5.0 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 11.2.0 (Rev9, Built by MSYS2 project)
  configuration: --disable-sdl2 --disable-zlib --disable-iconv --disable-bzlib --disable-lzma --disable-w32threads --disable-pthreads
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
Encoder flac [FLAC (Free Lossless Audio Codec)]:
    General capabilities: dr1 delay small
    Threading capabilities: none
    Supported sample formats: s16 s32
FLAC encoder AVOptions:
  -lpc_coeff_precision <int>        E...A...... LPC coefficient precision (from 0 to 15) (default 15)
  -lpc_type          <int>        E...A...... LPC algorithm (from -1 to 3) (default -1)
     none            0            E...A......
     fixed           1            E...A......
     levinson        2            E...A......
     cholesky        3            E...A......
  -lpc_passes        <int>        E...A...... Number of passes to use for Cholesky factorization during LPC analysis (from 1 to INT_MAX) (default 2)
  -min_partition_order <int>        E...A...... (from -1 to 8) (default -1)
  -max_partition_order <int>        E...A...... (from -1 to 8) (default -1)
  -prediction_order_method <int>        E...A...... Search method for selecting prediction order (from -1 to 5) (default -1)
     estimation      0            E...A......
     2level          1            E...A......
     4level          2            E...A......
     8level          3            E...A......
     search          4            E...A......
     log             5            E...A......
  -ch_mode           <int>        E...A...... Stereo decorrelation mode (from -1 to 3) (default auto)
     auto            -1           E...A......
     indep           0            E...A......
     left_side       1            E...A......
     right_side      2            E...A......
     mid_side        3            E...A......
  -exact_rice_parameters <boolean>    E...A...... Calculate rice parameters exactly (default false)
  -multi_dim_quant   <boolean>    E...A...... Multi-dimensional quantization (default false)
  -min_prediction_order <int>        E...A...... (from -1 to 32) (default -1)
  -max_prediction_order <int>        E...A...... (from -1 to 32) (default -1)

See the last two options.

That's what I get for reading https://www.ffmpeg.org/ffmpeg-codecs.html#Options-10 rather than memorizing the simple help syntax ...  :-[

Scott Lawlor and Dronny Darko "Polar Night on Titan" (2015) (https://archive.org/details/petroglyph358ScottLawlor_DronnyDarko-PolarNightOnTitan). Dark ambient, drone
32.22% - 246.789.073 bytes (455 kbps): FLAC 1.4.0 -8 -A subdivide_tukey(9) -p
32.20% - 246.664.559 bytes (454 kbps): FLAC 1.4.0 -8pe
32.06% - 245.520.054 bytes (452 kbps): WavPack 5.5.0 -hh -x4
31.92% - 244.459.886 bytes (450 kbps): WavPack 5.5.0 -hh -x6
30.86% - 236.360.371 bytes (435 kbps): TAK 2.3.3 -p4m

It would be interesting to see a file with poor compression ratio as well, and unlike some of the Merzbow, this track is also poor on 7z/xz as well.
https://youtu.be/wJW9AIwHSGY

File size:
100.00% 40,388,588 Syvalion arrange MMIX.wav
83.143% 33,580,441 -8ep.flac
83.135% 33,577,007 -l 12 -b 4096 -m -r 8 -A subdivide_tukey(9) -p.flac
82.340% 33,256,002 hhx6.wv
81.436% 32,890,956 insane.ape

Decoding speed (foo_benchmark):
FLAC: 1359.678x realtime
WavPack: 315.513x realtime
Monkey's Audio: 77.892x realtime

78x decoding speed could be sane for some synth engines in foo_midi, but not very sane for CDDA.

File size:
100.00% 40,388,588 Syvalion arrange MMIX.wav
83.143% 33,580,441 -8ep.flac
83.135% 33,577,007 -l 12 -b 4096 -m -r 8 -A subdivide_tukey(9) -p.flac
82.340% 33,256,002 hhx6.wv
81.436% 32,890,956 insane.ape

LMAO
83.117% 33,569,736 -8p -b2048.flac

So, not even that much trial and error deletions by windowing could overtake the benefit of having two smaller blocks with different predictors.
@bennetng : try CUETools.flake with and without --vbr and check the difference?

@ktf:
* On -7: Have I understood correctly that the difference between new -7 (with -A subdivide_tukey(2)) and -7 -A tukey(5e-1);partial_tukey(2) is really only the tapering applied in the two parts, apart from that they are the same and should take the same time? Or does the new -7 calculate another two "unpunched" tukeys and tests them both and should be expected to every now and then be better?
Asking because the "old -7 apodizations" takes as good as same time, but made for 35 of 38 files smaller. Not much; the overall impact is 0.002 percent (after that percentage "being halved by" those 3 of 38 that swung the other way).
* On -8: Have I understood correctly that the subdivide_tukey(3) would roughly correspond to adding another "partial_tukey(3)" to old -8 - in addition to the tapering considerations as in -7, whether I got them right or wrong?
Issue is, -8 appears slightly faster than "old -8 apodizations" and makes smaller files.

So, not even that much trial and error deletions by windowing could overtake the benefit of having two smaller blocks with different predictors.
@bennetng : try CUETools.flake with and without --vbr and check the difference?

I am not familiar with this encoder, could you suggest a complete commandline that you think is optimal?

Optimal I don't know, but some were given in Reply 128 (https://hydrogenaud.io/index.php?msg=1016038).
Flake compression levels up to -8 are subset, so ... <path-to-cuetools>\CUETools.Flake.exe -8 --vbr 4 wavefile.wav
Replacing the "4" by "3", "2", "1" apparently gives various willingness to vary. It is not obvious what gives smallest files. "--vbr 0" is apparently synonymous to deleting the vbr thing and using constant block size.

My curiosity was rather about whether a variable block size matters, comparing Flake to Flake rather than to see if it can win against a different encoder.

CUETools.Flake.exe -8 --vbr n input.wav
33,626,598 vbr0.flac
33,602,041 vbr1.flac
33,585,224 vbr2.flac
33,583,900 vbr3.flac
33,577,870 vbr4.flac

https://youtu.be/iMH49ieL4es
63,014,828 Katamari on the Rocks (Main Theme).wav
45,665,993 flac140 -8pe.flac
45,648,260 flac140 -8p -b2304.flac
45,643,706 CueToolsFlake -8 --vbr 4.flac

* On -7: Have I understood correctly that the difference between new -7 (with -A subdivide_tukey(2)) and -7 -A tukey(5e-1);partial_tukey(2) is really only the tapering applied in the two parts, apart from that they are the same and should take the same time?

New should be slightly faster because it skips the zeroed part of the block, which partial_tukey does not do

Asking because the "old -7 apodizations" takes as good as same time, but made for 35 of 38 files smaller. Not much; the overall impact is 0.002 percent (after that percentage "being halved by" those 3 of 38 that swung the other way).

Yes, there might be tiny differences.

* On -8: Have I understood correctly that the subdivide_tukey(3) would roughly correspond to adding another "partial_tukey(3)" to old -8 - in addition to the tapering considerations as in -7, whether I got them right or wrong?

Yes

Issue is, -8 appears slightly faster than "old -8 apodizations" and makes smaller files.

I don't get it? Faster and smaller files, what is the issue?

I let CUETools.Flake run all the four (plus none) variable block size settings through that 38 CD corpus, and the results are not as dramatic as in @bennetng 's file in 138 and 142.  Therein, --vbr 4 made for 0.14something percent (not percentage point). In my 38 files,
Flake -8 files were slightly bigger than reference FLAC -8 files
Flake -8 --vbr 4 files were slightly bigger than FLAC -8p files.
From FLAC -8, the -p made for 0.069 percent reduction (not percentage points)
From Flake -8, the --vbr 4 made for 0.074 percent reduction (ditto). The classical music section did disagree, there -8 made for slightly smaller. And the classical music section also made --vbr 1 worse than --vbr 0 overall. But --vbr 3 was worst overall and in two sections.

I have not bothered to check the --vbr documentation, when I have tested it earlier I've always used --vbr 4, thinking that hey if I want to check how the variable block size strategy works, I'll pick whatever seems to be labeled the most aggressive. And anything that makes for a "-p" size difference at -8, would probably be interesting to ... urrr, to anyone who might consider -8p.

I don't get it? Faster and smaller files, what is the issue?

:)
No, it is good - the issue is to make sense of my numbers. As -8 does additional work compared to "old -8", it should not speed up - in the very least, not so much more than -7 over "old -7".
Will have to run another overnight job and possibly blame the mysterious ways of the fan control.

https://vgmdb.net/album/331
https://www.youtube.com/playlist?list=PL8woW7Kc7z-brUycWctb2nSB9P7IM5bbY

Whole image:
555,153,554 flac140 -8p.flac
554,139,415 CueToolsFlake -8 --vbr 4.flac
554,012,449 flac140 -8p -b2048.flac
553,841,524 CueToolsFlake -8 --vbr 4 -s search.flac
537,495,392 APE High.ape

Changing other CueTools.Flake settings make things much slower and therefore not tested.

FLAC 1.4.1 is out.

FLAC 1.4.1 is out.

God damn it, now we have to test it all over again :)

Quote from: ssjkakaroto on 2022-09-22 15:47:32

FLAC 1.4.1 is out.

God damn it, now we have to test it all over again :)

Changes aren't that much. Mainly, the help files were wrong - so flac --help and --explain showed old obsolete information. According to https://hydrogenaud.io/index.php/topic,123014.msg1016154.html#msg1016154 , "Binary might be an unmeasurable tiny amount faster because the binary is slightly smaller."

So I guess performance tests on official 1.4.0 are still valid. But, people are pumping out new compiles faster than they can be tested.
And faster than the damn overzealous antivirus engines can get their act together. I got a ton of them new builds blocked (https://hydrogenaud.io/index.php/topic,123025.msg1016295.html#msg1016295).

No idea where is the best thread to ask generic flac 1.4.x new features questions, so ... summoning @ktf by mention.

* subdivide_tukey(1) is supposed to be tukey(5e-1), right?
* but, -A "subdivide_tukey(3);subdivide_tukey(1)" is not the same as -A "subdivide_tukey(3);tukey(5e-1)" - they produce different-size files. Indeed it looks like that in -A "subdivide_tukey(3);subdivide_tukey(1)" is the same as - A "subdivide_tukey(1);subdivide_tukey(3)" is the same as -A "subdivide_tukey(3)", with the lowest one simply ignored.
* however, the subdivide_tukey(1) is not subsumed in the (3) routine. Different taper.

And it isn't that simple:

* The following yield the same file sizes:
-A "subdivide_tukey(3);subdivide_tukey(2)"
-A "subdivide_tukey(3/5e-1);subdivide_tukey(2)"
-A "subdivide_tukey(3);subdivide_tukey(2/5e-1)"
-A "subdivide_tukey(3/5e-1);subdivide_tukey(2/5e-1)"

* But look:
-A "subdivide_tukey(3/6e-1);subdivide_tukey(2/6e-1)" gives same sizes as -A "subdivide_tukey(3);subdivide_tukey(2/6e-1)",
but different from
-A "subdivide_tukey(2/6e-1);subdivide_tukey(3/6e-1)"

-A "subdivide_tukey(2/8e-1);subdivide_tukey(8/8e-1)" gives same sizes as -A "subdivide_tukey(2);subdivide_tukey(8/8e-1)"
-A "subdivide_tukey(2/2e-1);subdivide_tukey(8/8e-1)" gives same sizes as -A "subdivide_tukey(2/8e-1);subdivide_tukey(8)"



Bug, feature or ... uh, engineering compromise?

* subdivide_tukey(1) is supposed to be tukey(5e-1), right?

I see in the code subdivide_tukey(1) is simply dropped.

* But look:
-A "subdivide_tukey(3/6e-1);subdivide_tukey(2/6e-1)" gives same sizes as -A "subdivide_tukey(3);subdivide_tukey(2/6e-1)",
but different from
-A "subdivide_tukey(2/6e-1);subdivide_tukey(3/6e-1)"

-A "subdivide_tukey(2/8e-1);subdivide_tukey(8/8e-1)" gives same sizes as -A "subdivide_tukey(2);subdivide_tukey(8/8e-1)"
-A "subdivide_tukey(2/2e-1);subdivide_tukey(8/8e-1)" gives same sizes as -A "subdivide_tukey(2/8e-1);subdivide_tukey(8)"

Bug, feature or ... uh, engineering compromise?

No clue really. I should run a debugger to find out what is happening there.