And that's with ~8KB of padding.  The smallest I can make your example while still subset (if you consider variable blocksize to be subset) is this:

So, the CPU time is 'single-core time' I guess? So when using 4 CPU cores wall time would be a quarter of that?

I think there are plenty people on this forum that would be willing to use this, creating a subset, fully spec-compliant FLAC file that might play in their hardware system. Half a percent is quite an improvement, even at the cost of having the encoder run near real-time on a single core. So once again: very impressive.

Sadly non-standard blocksizes and variable blocksize streams are not well supported features in many hardware decoders, see here, but I think many people here won't mind.

With the above in mind these settings might be reasonable for subset (again I haven't tested queue depth or gasc much, nor chunk with decent tweak/merge, some of these are more guess than knowledge). Seems reasonable to pair stronger flac settings with stronger flaccid settings, it could be tweaked many ways. In probably ascending difficulty

• Quick
• --mode gasc --blocksize-limit-lower 1536 --tweak 0 --merge 0 --analysis-comp 6 --output-comp 6

Probably far from optimal but it might be a good starting point. I'd consider the first peakset setting on the list as probably the most balanced.

H:\>flaccid --in phantom.wav --out phantom.flac --mode gasc --blocksize-limit-lower 1536 --tweak 0 --merge 0 --analysis-comp 6 --output-comp 6
phantom.wav     settings        mode(gasc);lax(0);analysis_comp(6);analysis_apod((null));output_comp(6);output_apod((null));tweak(0);merge(0);blocksize_limit_lower(1536);blocksize_limit_upper(4608)
effort  analysis(2.734);tweak(0.000);merge(0.000);output(0.000)  subtiming       analysis(0.00000);tweak(0.00000);merge(0.00000) size    518191375        cpu_time        62.12700

Differences found in compared tracks; the tracks became identical after applying offset and truncating first/last samples.
Extra leading/trailing sections contained non-null samples.

Comparing:
"H:\phantom.flac"
"H:\phantom.wav"
Differences found: length mismatch - 1:40:31.333583 vs 1:40:31.333333, 265981811 vs 265981800 samples.
Compared 265981800 samples.
Discarded last 11 samples containing non-null values from the longer file.
Differences found within the compared range: 531100405 values, 0:00.000000 - 1:40:31.333311, peak: 1.631927 (+4.25 dBTP) at 0:40.642426, 1ch
Channel difference peaks: 1.631927 (+4.25 dBTP) 1.481720 (+3.42 dBTP)
File #1 peaks: 0.999969 (-0.00 dBTP) 1.000000 (0.00 dBTP)
File #2 peaks: 0.999969 (-0.00 dBTP) 1.000000 (0.00 dBTP)
Detected offset as -11 samples.

Comparing again with corrected offset...
Compared 265981800 samples, with offset of -11.
Discarded 11 leading samples containing non-null values from file #1.
No differences in decoded data found within the compared range.
Channel peaks: 0.999969 (-0.00 dBTP) 1.000000 (0.00 dBTP)

• Peakset now supports multiple windows of user-defined size (in millions of samples, with --peakset-window size), so peakset no longer requires access to the entire file at once. This paves the way for RAM to not scale with input size (in a future commit when the loader no longer reads the entire input before encoding begins, big TODO)
• Add --no-seek option to disable output stream seeking, meaning the header is not updated after encoding. Requires --no-md5 to be set so that the user understands that hashing has to be disabled
• Add output pipe support (with --out -). Either fully cached in RAM to allow header to be updated before writing, or used in tandem with --no-seek to write to pipe as soon as output frames are created
• Fixed out-of-spec issue with blocksize_min in header when blocksize of partial last frame of variable encode is less than 16 samples. The partial last frame of variable encodes is still represented in the header but that's within spec (blocksize_min/max are bounds, nothing is mentioned about frame sizes but I assume the partial frame should always be present there even with fixed encodes as there is a tiny chance it's the biggest frame size)
• Fixed potential issue with peakset and small queues, set->blocksize_min strikes again. set->blocksize_min has now been eliminated where possible to reflect its change of use
• Fixed rare case where a variable encode looks like a fixed encode in the header. This probably occurs when input is very small, it can occur when the settings used are pathological or wildly inefficient, it is very unlikely to occur under normal circumstances
• Fixed stat gathering and made it multithread-safe. Subtiming eliminated and everything gathered properly instead of using mode-specific shortcuts
• Fixed subset with high samplerates still being restricted slightly too strictly
• Fully convert modes to using simple_enc instead of manually defining edge cases. Fixed implementation dramatically simplified
• Boilerplate sucked out of beginning and end of all mode implementations to simplify them
• flanal utility to validate flac files is now in the repo and has been updated slightly to work with input created with --no-seek

...
(There must be some nice thesis idea for a clever jr researcher here: find a heuristic that takes as input your typical .flac file with a 4096 frame size, does a quick analysis on the predictor and the ... hm, rice partition order and parameter I guess? - to decide what frames to split. I've seen the IEEE publish far less interesting ideas than that.)

The partition order MUST be so that the block size is evenly divisible by the number of partitions. This means for example that for all odd block sizes, only partition order 0 is allowed. The partition order also MUST be so that the (block size >> partition order) is larger than the predictor order. This means for example that with a block size of 4096 and a predictor order of 4, partition order cannot be larger than 9

     Amount saved in pass               Total saved by a certain point
pass    1    2    4    8   16   32   64     1      2      4      8     16     32     64
 1   3900 3900 3900 3900 3900 3900 4012
 2   1982 1982 1982 1982 1982 1982 1681
 3   1165 1165 1165 1165 1165 1376 1491
 4     69  146  803  803  803 1099 1662
 5   1189 1032  740  740  740  125  295
 6    126  150  527  527  527  863   40
 7    467  460  260  260  606  123   11  8898   8835   9377   9377   9723   9468   9192
 8    489  484  411  411   87    9
 9     34  314  304  304  486  820
10    140  58    77  397   63  152
11    431 358   315   62    7    9       9992  10049  10484  10551  10366  10458
12     95  88    87  176  550
13     25 234   140   17  74
14     50  25   104  428  11            10162  10396  10815  11172  11001
15    123  77    13   60
16      9 372   368    7                10294  10845  11196  11239
17    436  73    72
18     17  37    75
19     83  91    15
20     15  13   167                     10845  11059  11525
21     10        16
22      1         5                     10856         11546
23    181
24      3
25     10
26      8
27    100
28      3                               11161

Maybe adjacent frames that have similar models are ripe for merging, mostly this will not be subset.

That may give quite different treatment when subset is imposed, limiting to partition order 8, i.e. 1/256th of the block. Sure there are exceptional signals where constraining yourself to subset means you should do -b2048 or maybe even lower only to get a smaller partition (in samples) at -r8. And when your project is to tweak block size, it could very well be that those signals are many enough to care about. Assuming subset is a constraint.

(It would likely need to be that in the reference encoder. I have no idea whether there is any subset-compliant decoder - meaning it would handle partition size of 1 sample and block size 64 - that would choke on partition size of 32 samples and block size 4096. But anyway ...)

As for this:

Quote from: cid42 on 2023-02-14 17:51:40

Maybe adjacent frames that have similar models are ripe for merging, mostly this will not be subset.

Yes, that is merging. I was talking splitting - which for all that I know, may not even be on the table in what you are working on.
(Generally I don't even have any idea how much a "good" (like in -8 with a reasonable -q) predictor vector differs from an "optimal" of the same -q. I know I can run some encodes and parse -a output, but ... not happening with my clumsiness at coding.)

• Presets 0..8 match ./flac so that people can encode fixed with settings they're familiar with, just multithreaded
• Preset 9 is currently gasc starting at blocksize 1536 with analysis setting 3m and output setting 8. This is slightly more space-efficient than -8 in testing but it's not guaranteed
• Preset 10 is currently peakset using 1152,2304,3456,4608 with analysis setting 3m and output setting 8
• More variable presets will come and the two that exist might change
• These options from ./flac work with all presets, appended to the preset number: e/l/m/p/q/r
• -b from ./flac to pick blocksize also works, with the fixed presets 0..8 only
• --preset-apod overrides the apod settings used by the preset, it doesn't append

• Reworked input handling to now read only as much input as is immediately needed by analysis
• --input-format forces input to be read as a particular format. Useful for piped input or when extension is different from typical. Valid formats are: flac wav cdda
• flac input can now be piped by specifying '--in -' in tandem with '--input-format flac'
• Raw input reworked to only be for CDDA, automatically chosen for files with .bin exension. Pipe works and requires '--input-format cdda'
• Initial wav support implemented but it's basic. No pipe, 16 bit only, not very well tested
• Hashing is now done by input, greatly simplifying the rest of the interface
• Total samples of input no longer used during encode so it passes flac-test-files 45, now all flac-test-files pass
• If total samples is available from input header (flac/wav), it's checked on completion
• If MD5 is available from input header (flac) and we're hashing, it's checked on completion

• Seektable support added with --seektable val. -1 (default) to let program size the seektable based on input header (flac/wav), fallback to 100 seekpoints. 0 disables the seektable, n>0 tells the program to use that many seekpoints. The program reserves the right to override users choice if it detects that it's massive overkill
• Optional metadata preservation from flac input files with --preserve-flac-metadata. Preserves the following meadata blocktypes: Application, Vorbis, Cuesheet, Picture, Undefined, aka everything that can be preserved
• Added a --metacomp option to flanal allowing it to check if --preserve-flac-metadata losslessly preserved the metadata

./common.h:34:29: error: unknown type name 'FLAC__StaticEncoder'; did you mean 'FLAC__StreamEncoder'?

gcc flaccid.c gasc.c gset.c load.c chunk.c fixed.c common.c peakset.c seektable.c -oflaccid -I../flac/include '/home/u20/Documents/mountain/runtime/flac/src/libFLAC/.libs/libFLAC-static.a' -logg -lm -fopenmp -Wall -O3 -funroll-loops  -Wall -Wextra -Wstrict-prototypes -Wmissing-prototypes -Waggregate-return -Wcast-align -Wnested-externs -Wshadow -Wundef -Wmissing-declarations -Winline  -Wdeclaration-after-statement -fvisibility=hidden -fstack-protector-strong -lcrypto -DUSE_OPENSSL

I do plan on eventually updating flaccid to the latest flac and implementing the few things it needed to make it less clunky in general, but how best to do that may need a rethink now that libflac can multithread itself. Let me know how you get on.

Also, currently someone is writing a PR (still a draft) to add multithreading support in `flac`

 clang -static -flto -s -fprofile-instr-generate -fcoverage-mapping -mtune=generic flaccid.c gasc.c gset.c load.c chunk.c fixed.c common.c peakset.c seektable.c -oflaccid -I../flac/include ../libFLAC-static.a ../libogg.a ../libcrypto.a -lm ../libomp.a -Wall -O3 -funroll-loops -Wall -Wextra -Wstrict-prototypes -Wmissing-prototypes -Waggregate-return -Wcast-align -Wnested-externs -Wshadow -Wundef -Wmissing-declarations -Winline -Wdeclaration-after-statement -fvisibility=hidden -fstack-protector-strong -DUSE_OPENSSL

./flaccid --in liberate.flac --lax --out liberate_VarSizeBruteForced.flac --peakset-window 48 --preserve-flac-metadata --queue 8192 --workers 16 --tweak 1 --merge 0 --analysis-apod 'subdivide_tukey(5);gauss(0.1);tukey(5);partial_tukey(5);punchout_tukey(5);welch;bartlett;bartlett_hann;blackman;blackman_harris_4term_92db;connes;flattop;hamming;hann;kaiser_bessel;nuttall;rectangle;triangle' --output-apod 'subdivide_tukey(5);gauss(0.1);tukey(5);partial_tukey(5);punchout_tukey(5);welch;bartlett;bartlett_hann;blackman;blackman_harris_4term_92db;connes;flattop;hamming;hann;kaiser_bessel;nuttall;rectangle;triangle' --analysis-comp mepl32r15 --output-comp mepl32r15 --mode peakset --blocksize-list 512,1024,1536,2048,2560,3072,3584,4096,4608

Finally finished my build!

So, this is a build: Linux x86-64, generic (mtune), static, optimized (O3 flto fprofile-instr-gen), stripped, UPX'd (compressed with -9 --ultra-brute)
Compiled with sources from yesterday, using clang 15.0.7

I will definitely not build a Windows version, but aarch64 build will come when I need to use it (or when someone asks for it).

Command used for compiling (PGO gen):

Code: [Select]

 clang -static -flto -s -fprofile-instr-generate -fcoverage-mapping -mtune=generic flaccid.c gasc.c gset.c load.c chunk.c fixed.c common.c peakset.c seektable.c -oflaccid -I../flac/include ../libFLAC-static.a ../libogg.a ../libcrypto.a -lm ../libomp.a -Wall -O3 -funroll-loops -Wall -Wextra -Wstrict-prototypes -Wmissing-prototypes -Waggregate-return -Wcast-align -Wnested-externs -Wshadow -Wundef -Wmissing-declarations -Winline -Wdeclaration-after-statement -fvisibility=hidden -fstack-protector-strong -DUSE_OPENSSL

I compiled everything myself (static libs like OMP too, using `clang -flto -O3 -mtune=generic`).

Command used for profiling data:

Code: [Select]

./flaccid --in liberate.flac --lax --out liberate_VarSizeBruteForced.flac --peakset-window 48 --preserve-flac-metadata --queue 8192 --workers 16 --tweak 1 --merge 0 --analysis-apod 'subdivide_tukey(5);gauss(0.1);tukey(5);partial_tukey(5);punchout_tukey(5);welch;bartlett;bartlett_hann;blackman;blackman_harris_4term_92db;connes;flattop;hamming;hann;kaiser_bessel;nuttall;rectangle;triangle' --output-apod 'subdivide_tukey(5);gauss(0.1);tukey(5);partial_tukey(5);punchout_tukey(5);welch;bartlett;bartlett_hann;blackman;blackman_harris_4term_92db;connes;flattop;hamming;hann;kaiser_bessel;nuttall;rectangle;triangle' --analysis-comp mepl32r15 --output-comp mepl32r15 --mode peakset --blocksize-list 512,1024,1536,2048,2560,3072,3584,4096,4608

on a 5 seconds FLAC file (44.1kHz stereo s16)

I'm quite happy with the output FLAC file: compared to fixed 4096-blocksize with high compression settings using `flac`, `flaccid` managed to save 0.5% of filesize!
To even save more I could have increased the numbers given to all tukey functions and add more blocksize to bruteforce, but this was just a test on a single file in order to build with PGO.

Since I wanted a slow encode (took 3000 seconds - take into account it was profiling data), I based my command on this comment https://hydrogenaud.io/index.php/topic,123248.msg1022236.html#msg1022236. I must say that it's quite unclear for me what the values --merge and --tweak do (even reading that comment and the help message from flaccid).

Even if I set workers to 16, it never reached 200% CPU usage. Perhaps because it was profiling data?

flaccid --workers 8 --preset 8p --in file.wav --out file.flac