Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: FLAC v1.4.0 (Release) (Read 30442 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: FLAC v1.4.0 (Release)

Reply #125 –
Shouldn't the exhaustive model search be trying all of them, and then picking the best one? It seems very strange to me that -e makes the compression worse; the worst case scenario should be that the file size remains the same, because the non-exhaustive search was already using the best model, but -e making the size go up is quite counter intuitive.

Re: FLAC v1.4.0 (Release)

Reply #126 –
Quote from: Aleron Ives on
-e making the size go up

Well, it doesn't. Read Porcus's message again - none of the comparisons are between [some parameters with e] and [those exact same parameters except e].

What is true is that the addition of -e does not achieve as much of a compression improvement as does the addition of another parameter, such as -A subdivide_tukey(5) or -p.

Re: FLAC v1.4.0 (Release)

Reply #127 –
Yep. -e reduces size, but not so much as -p or "going up two subdivide_tukey steps" will do (subdivide_tukey(3) is included in -8 already) - and -e spends more time achieving less.

Note that the "exhaustive" in -e doesn't cover all modelling aspects. It brute-forces part of it, whereas -p brute-forces another part. subdivide_tukey(5) only makes for the encoder to estimate more function choices before picking the guesstimated best - and it seems that trying more (without full accuracy) is more efficient than trying fewer but brute-forcing them.

Re: FLAC v1.4.0 (Release)

Reply #128 –
Scott Lawlor and Dronny Darko "Polar Night on Titan" (2015). Dark ambient, drone

100.0% - 765.928.844 bytes
34.30% - 262.710.949 bytes (484 kbps): FLAC 1.3.4 -5
33.80% - 258.918.920 bytes (477 kbps): FLAC 1.3.4 -8
33.75% - 258.472.525 bytes (476 kbps): FLAC 1.3.4 -8p
33.26% - 254.765.189 bytes (469 kbps): FLAC 1.3.4 -8pe
33.09% - 253.431.855 bytes (467 kbps): FLAC 1.4.0 -3
32.96% - 252.451.802 bytes (465 kbps): WavPack 5.5.0 -hh -x1
32.91% - 252.139.633 bytes (465 kbps): ffmpeg -compression_level 3 -frame_size 4096
32.71% - 250.531.950 bytes (462 kbps): CUETools.Flake 2.2.2 -2
32.68% - 250.301.234 bytes (461 kbps): CUETools.Flake 2.2.2 -3
32.66% - 250.157.962 bytes (461 kbps): ffmpeg -compression_level 5 -frame_size 4096
32.53% - 249.155.643 bytes (459 kbps): ffmpeg -compression_level 8 -frame_size 4096
32.51% - 248.997.530 bytes (459 kbps): ffmpeg -compression_level 10 -frame_size 4096
32.49% - 248.821.318 bytes (458 kbps): FLAC 1.4.0 -4
32.48% - 248.753.600 bytes (458 kbps): FLAC 1.4.0 -5
32.46% - 248.611.049 bytes (458 kbps): FLAC 1.4.0 -6
32.45% - 248.531.637 bytes (458 kbps): CUETools.Flake 2.2.2 -4
32.44% - 248.444.048 bytes (458 kbps): CUETools.Flake 2.2.2 -5
32.42% - 248.321.082 bytes (458 kbps): CUETools.Flake 2.2.2 -6
32.42% - 248.305.277 bytes (457 kbps): CUETools.Flake 2.2.2 -7
32.37% - 247.965.156 bytes (457 kbps): ffmpeg -compression_level 10 -frame_size 4096 -lpc_type 3 -lpc_passes 10 -exact_rice_parameters 1
32.35% - 247.791.048 bytes (457 kbps): CUETools.Flake 2.2.2 -8
32.32% - 247.585.181 bytes (456 kbps): ffmpeg -compression_level 10 -lpc_type 3 -lpc_passes 10 -exact_rice_parameters 1
32.28% - 247.269.005 bytes (456 kbps): CUETools.Flake 2.2.2 -8 -e 32 -s search --window-method search
32.28% - 247.264.261 bytes (456 kbps): FLAC 1.4.0 -7
32.27% - 247.197.583 bytes (455 kbps): FLAC 1.4.0 -8
32.23% - 246.889.401 bytes (455 kbps): FLAC 1.4.0 -8p
32.22% - 246.789.073 bytes (455 kbps): FLAC 1.4.0 -8 -A subdivide_tukey(9) -p
32.20% - 246.664.559 bytes (454 kbps): FLAC 1.4.0 -8pe
32.06% - 245.520.054 bytes (452 kbps): WavPack 5.5.0 -hh -x4
31.92% - 244.459.886 bytes (450 kbps): WavPack 5.5.0 -hh -x6
30.86% - 236.360.371 bytes (435 kbps): TAK 2.3.3 -p4m

Re: FLAC v1.4.0 (Release)

Reply #131 –
I see. I misinterpreted this:

Quote from: Porcus on
-8pe made bigger files and is five times slower than -8p -A subdivide_tukey(5) (you can increase to subdivide_tukey(7) and still save time).

as -8pe also containing subdivide_tukey(5), thus the addition of -e made things worse, but it was actually the omission of -A.

Re: FLAC v1.4.0 (Release)

Reply #132 –
-8 includes -A subdivide_tukey(3).

I see above that -e still works somewhere out in space (that album title does give you the right idea of the content yes).
Question though: is ffmpeg -frame_size 4096 (or 4608) subset until compression level 10 but not until 12?

Of course, TAK slays everything that's fast and a lot of slower codecs too. Filling some numbers into @Gravity Stupor 's list, those in italics are mine.
246.889.401 bytes (455 kbps): FLAC 1.4.0 -8p
246 867 738 bytes for -8p -r8, that helps only a a tiny bit
246.789.073 bytes (455 kbps): FLAC 1.4.0 -8 -A subdivide_tukey(9) -p
246.664.559 bytes (454 kbps): FLAC 1.4.0 -8pe
[...]
237 725 244 Monkey's "Insane" is overTAKen by something faster
236 462 439 OptimFROG at default (= -- preset 2)
236.360.371 bytes (435 kbps): TAK 2.3.3 -p4m
235 190 251 OptimFROG --preset 3

Re: FLAC v1.4.0 (Release)

Reply #133 –
Quote from: Porcus on
Question though: is ffmpeg -frame_size 4096 (or 4608) subset until compression level 10 but not until 12?
ffmpeg works a little different than FLAC.
  • ffmpeg uses a block time instead of a blocksize, which is then translated to the nearest standard blocksize. Blocktime for settings 0, 1 and 2 is 27ms, for settings 3 until 12 it is 105ms. For 44.1kHz and 48kHz this means blocksize is 4608 for settings 3 and up
  • Predictor order is 8 for levels 3, 4, 5, 6 and 7, is 12 for levels 8, 9 and 10 and 32 for levels 11 and 12
  • -e is set for levels 10 and 12. Estimation (as FLAC uses without -e) is at levels 3, 4 and 5. Levels 6, 7, 8, 9 and 11 use something in between
  • -p has no equivalent in ffmpeg

TL;DR: ffmpeg compression levels 11 and 12 for FLAC are non-subset when used for audio with a samplerate of 48kHz or lower.
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.0 (Release)

Reply #134 –
Ah, thanks. So ffmpeg selects subset-compliant block sizes up to ... found in this post, 192000. Above which, -frame_size 16384 can enforce it. Predictor order on the other hand, cannot be overridden by user (at least not by any documented option) and is what throws 11 and 12 off subset.

The testing I have done on a limited 96 kHz corpus indicates that ffmpeg's blocking strategy is quite sane: Reference FLAC could benefit from setting block size to 8192 for 96 kHz files; you suggested that windowing could do the same, and I if that holds up to testing, I guess that those who are willing to wait for creative smaller-size-than-8 combinations might be satisfied even if it takes longer time running all the functions.
Also that testing suggests that for CDDA, reference FLAC is so well tuned to its 4096 choice that 4608 is even slightly harmful. Is anyone curious enough to test 4608 vs 4096 on 48 kHz? ;-)

Re: FLAC v1.4.0 (Release)

Reply #135 –
Quote from: Porcus on
Predictor order on the other hand, cannot be overridden by user (at least not by any documented option) and is what throws 11 and 12 off subset.

I guess it is time to show you some more documentation then  :))  O:)

Code: [Select]
$ ./ffmpeg -h encoder=flac
ffmpeg version n5.0 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 11.2.0 (Rev9, Built by MSYS2 project)
  configuration: --disable-sdl2 --disable-zlib --disable-iconv --disable-bzlib --disable-lzma --disable-w32threads --disable-pthreads
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
Encoder flac [FLAC (Free Lossless Audio Codec)]:
    General capabilities: dr1 delay small
    Threading capabilities: none
    Supported sample formats: s16 s32
FLAC encoder AVOptions:
  -lpc_coeff_precision <int>        E...A...... LPC coefficient precision (from 0 to 15) (default 15)
  -lpc_type          <int>        E...A...... LPC algorithm (from -1 to 3) (default -1)
     none            0            E...A......
     fixed           1            E...A......
     levinson        2            E...A......
     cholesky        3            E...A......
  -lpc_passes        <int>        E...A...... Number of passes to use for Cholesky factorization during LPC analysis (from 1 to INT_MAX) (default 2)
  -min_partition_order <int>        E...A...... (from -1 to 8) (default -1)
  -max_partition_order <int>        E...A...... (from -1 to 8) (default -1)
  -prediction_order_method <int>        E...A...... Search method for selecting prediction order (from -1 to 5) (default -1)
     estimation      0            E...A......
     2level          1            E...A......
     4level          2            E...A......
     8level          3            E...A......
     search          4            E...A......
     log             5            E...A......
  -ch_mode           <int>        E...A...... Stereo decorrelation mode (from -1 to 3) (default auto)
     auto            -1           E...A......
     indep           0            E...A......
     left_side       1            E...A......
     right_side      2            E...A......
     mid_side        3            E...A......
  -exact_rice_parameters <boolean>    E...A...... Calculate rice parameters exactly (default false)
  -multi_dim_quant   <boolean>    E...A...... Multi-dimensional quantization (default false)
  -min_prediction_order <int>        E...A...... (from -1 to 32) (default -1)
  -max_prediction_order <int>        E...A...... (from -1 to 32) (default -1)

See the last two options.
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.0 (Release)

Reply #137 –
Quote from: Gravity Stupor on
Scott Lawlor and Dronny Darko "Polar Night on Titan" (2015). Dark ambient, drone
32.22% - 246.789.073 bytes (455 kbps): FLAC 1.4.0 -8 -A subdivide_tukey(9) -p
32.20% - 246.664.559 bytes (454 kbps): FLAC 1.4.0 -8pe
32.06% - 245.520.054 bytes (452 kbps): WavPack 5.5.0 -hh -x4
31.92% - 244.459.886 bytes (450 kbps): WavPack 5.5.0 -hh -x6
30.86% - 236.360.371 bytes (435 kbps): TAK 2.3.3 -p4m
It would be interesting to see a file with poor compression ratio as well, and unlike some of the Merzbow, this track is also poor on 7z/xz as well.
https://youtu.be/wJW9AIwHSGY

File size:
100.00% 40,388,588 Syvalion arrange MMIX.wav
83.143% 33,580,441 -8ep.flac
83.135% 33,577,007 -l 12 -b 4096 -m -r 8 -A subdivide_tukey(9) -p.flac
82.340% 33,256,002 hhx6.wv
81.436% 32,890,956 insane.ape

Decoding speed (foo_benchmark):
FLAC: 1359.678x realtime
WavPack: 315.513x realtime
Monkey's Audio: 77.892x realtime

78x decoding speed could be sane for some synth engines in foo_midi, but not very sane for CDDA.

Re: FLAC v1.4.0 (Release)

Reply #138 –
Quote from: bennetng on
File size:
100.00% 40,388,588 Syvalion arrange MMIX.wav
83.143% 33,580,441 -8ep.flac
83.135% 33,577,007 -l 12 -b 4096 -m -r 8 -A subdivide_tukey(9) -p.flac
82.340% 33,256,002 hhx6.wv
81.436% 32,890,956 insane.ape
LMAO
83.117% 33,569,736 -8p -b2048.flac

Re: FLAC v1.4.0 (Release)

Reply #139 –
So, not even that much trial and error deletions by windowing could overtake the benefit of having two smaller blocks with different predictors.
@bennetng : try CUETools.flake with and without --vbr and check the difference?

@ktf:
* On -7: Have I understood correctly that the difference between new -7 (with -A subdivide_tukey(2)) and -7 -A tukey(5e-1);partial_tukey(2) is really only the tapering applied in the two parts, apart from that they are the same and should take the same time? Or does the new -7 calculate another two "unpunched" tukeys and tests them both and should be expected to every now and then be better?
Asking because the "old -7 apodizations" takes as good as same time, but made for 35 of 38 files smaller. Not much; the overall impact is 0.002 percent (after that percentage "being halved by" those 3 of 38 that swung the other way).
* On -8: Have I understood correctly that the subdivide_tukey(3) would roughly correspond to adding another "partial_tukey(3)" to old -8 - in addition to the tapering considerations as in -7, whether I got them right or wrong?
Issue is, -8 appears slightly faster than "old -8 apodizations" and makes smaller files.

Re: FLAC v1.4.0 (Release)

Reply #140 –
Quote from: Porcus on
So, not even that much trial and error deletions by windowing could overtake the benefit of having two smaller blocks with different predictors.
@bennetng : try CUETools.flake with and without --vbr and check the difference?
I am not familiar with this encoder, could you suggest a complete commandline that you think is optimal?

Re: FLAC v1.4.0 (Release)

Reply #141 –
Optimal I don't know, but some were given in Reply 128.
Flake compression levels up to -8 are subset, so ... <path-to-cuetools>\CUETools.Flake.exe -8 --vbr 4 wavefile.wav
Replacing the "4" by "3", "2", "1" apparently gives various willingness to vary. It is not obvious what gives smallest files. "--vbr 0" is apparently synonymous to deleting the vbr thing and using constant block size.

My curiosity was rather about whether a variable block size matters, comparing Flake to Flake rather than to see if it can win against a different encoder.

Re: FLAC v1.4.0 (Release)

Reply #142 –
CUETools.Flake.exe -8 --vbr n input.wav
33,626,598 vbr0.flac
33,602,041 vbr1.flac
33,585,224 vbr2.flac
33,583,900 vbr3.flac
33,577,870 vbr4.flac

Re: FLAC v1.4.0 (Release)

Reply #143 –
https://youtu.be/iMH49ieL4es
63,014,828 Katamari on the Rocks (Main Theme).wav
45,665,993 flac140 -8pe.flac
45,648,260 flac140 -8p -b2304.flac
45,643,706 CueToolsFlake -8 --vbr 4.flac
 

Re: FLAC v1.4.0 (Release)

Reply #144 –
Quote from: Porcus on
* On -7: Have I understood correctly that the difference between new -7 (with -A subdivide_tukey(2)) and -7 -A tukey(5e-1);partial_tukey(2) is really only the tapering applied in the two parts, apart from that they are the same and should take the same time?
New should be slightly faster because it skips the zeroed part of the block, which partial_tukey does not do

Quote
Asking because the "old -7 apodizations" takes as good as same time, but made for 35 of 38 files smaller. Not much; the overall impact is 0.002 percent (after that percentage "being halved by" those 3 of 38 that swung the other way).
Yes, there might be tiny differences.

Quote
* On -8: Have I understood correctly that the subdivide_tukey(3) would roughly correspond to adding another "partial_tukey(3)" to old -8 - in addition to the tapering considerations as in -7, whether I got them right or wrong?
Yes
Quote
Issue is, -8 appears slightly faster than "old -8 apodizations" and makes smaller files.
I don't get it? Faster and smaller files, what is the issue?
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.0 (Release)

Reply #145 –
I let CUETools.Flake run all the four (plus none) variable block size settings through that 38 CD corpus, and the results are not as dramatic as in @bennetng 's file in 138 and 142.  Therein, --vbr 4 made for 0.14something percent (not percentage point). In my 38 files,
Flake -8 files were slightly bigger than reference FLAC -8 files
Flake -8 --vbr 4 files were slightly bigger than FLAC -8p files.
From FLAC -8, the -p made for 0.069 percent reduction (not percentage points)
From Flake -8, the --vbr 4 made for 0.074 percent reduction (ditto). The classical music section did disagree, there -8 made for slightly smaller. And the classical music section also made --vbr 1 worse than --vbr 0 overall. But --vbr 3 was worst overall and in two sections.

I have not bothered to check the --vbr documentation, when I have tested it earlier I've always used --vbr 4, thinking that hey if I want to check how the variable block size strategy works, I'll pick whatever seems to be labeled the most aggressive. And anything that makes for a "-p" size difference at -8, would probably be interesting to ... urrr, to anyone who might consider -8p.


Quote from: ktf on
I don't get it? Faster and smaller files, what is the issue?
:)
No, it is good - the issue is to make sense of my numbers. As -8 does additional work compared to "old -8", it should not speed up - in the very least, not so much more than -7 over "old -7".
Will have to run another overnight job and possibly blame the mysterious ways of the fan control.

Re: FLAC v1.4.0 (Release)

Reply #146 –
https://vgmdb.net/album/331
https://www.youtube.com/playlist?list=PL8woW7Kc7z-brUycWctb2nSB9P7IM5bbY

Whole image:
555,153,554 flac140 -8p.flac
554,139,415 CueToolsFlake -8 --vbr 4.flac
554,012,449 flac140 -8p -b2048.flac
553,841,524 CueToolsFlake -8 --vbr 4 -s search.flac
537,495,392 APE High.ape

Changing other CueTools.Flake settings make things much slower and therefore not tested.

Re: FLAC v1.4.0 (Release)

Reply #147 –
FLAC 1.4.1 is out.
Allegari nihil et allegatum non probare, paria sunt.

Re: FLAC v1.4.0 (Release)

Reply #149 –
Quote from: itisljar on
Quote from: ssjkakaroto on
FLAC 1.4.1 is out.
God damn it, now we have to test it all over again :)
Changes aren't that much. Mainly, the help files were wrong - so flac --help and --explain showed old obsolete information. According to https://hydrogenaud.io/index.php/topic,123014.msg1016154.html#msg1016154 , "Binary might be an unmeasurable tiny amount faster because the binary is slightly smaller."

So I guess performance tests on official 1.4.0 are still valid. But, people are pumping out new compiles faster than they can be tested.
And faster than the damn overzealous antivirus engines can get their act together. I got a ton of them new builds blocked.