Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Retune of FLAC compression levels (Read 18519 times) previous topic - next topic
0 Members and 4 Guests are viewing this topic.

Re: Retune of FLAC compression levels

Reply #50 –
Are the preset details (i.e., individual synonymous options) listed on https://xiph.org/flac/documentation_tools_flac.html still the same in FLAC's current Git revision? If so, there seems to be quite some benefit in having that automatic adaptive mid-side selection. Especially with presets 1 vs. 2, the latter seems totally useless now: preset 1 gives almost the exact same compression ratio (Man Earing Duck confirmed this above) but much faster. Again, I suggest giving preset 2 a bit more compression efficiency, e.g., by increasing the prediction order to, say, "-l 4". I'm sure the decoding speed will then still scale with the presets.

And, cid42, do you honestly consider adding 50 lines of code and a handful of extra variables questionable in a project of this scale? Even if it's useful only for preset 1, I'd say it's worth it. Though I agree that, for all higher presets, "-m" should be the one to use.

Chris

If I don't reply to your reply, it means I agree with you.

Re: Retune of FLAC compression levels

Reply #51 –
Quote from: C.R.Helmrich on
...
And, cid42, do you honestly consider adding 50 lines of code and a handful of extra variables questionable in a project of this scale? Even if it's useful only for preset 1, I'd say it's worth it. Though I agree that, for all higher presets, "-m" should be the one to use.

Chris
I do assuming the functionality was pointless. Apparently it's not.

Re: Retune of FLAC compression levels

Reply #52 –
Quote from: A_Man_Eating_Duck on
Just to make is clear what I was suggesting with only focusing on 3 presets and mapping the presets up to either 2,5 or 8 was to make is easier on ktf and other FLAC devs to refine the presets.

I've done a test comparing the tuned version against 1.4.2 stock. This test is the average bitrate over 16919 tracks. The encoding time for the tuned version at preset 8 compared to stock preset 8 was about an hour quicker, less noticeable difference as the presets lowered.

PresetTunedStock
01012 kbps1037kbps
11010 kbps1013 kbps
21009 kbps1011 kbps
3960 kbps986 kbps
4957 kbps959 kbps
5955 kbps957 kbps
6957 kbps955 kbps
7955 kbps953 kbps
8953 kbps952 kbps
Just confirming that these results are correct.
Who are you and how did you get in here ?
I'm a locksmith, I'm a locksmith.

Re: Retune of FLAC compression levels

Reply #53 –
I ran encoding timings overnight. CDDA, the 38 CD images. Representative totals, are a major WTF:
284 seconds for retune at -6 yes!
360 seconds for retune at -5. The differences to -6 vary between genres from 17 to 23 percent, not much.
359 seconds for 1.4.2 at -5. (Edit a typo, it is 359.) Faster than retune on classical (5%), slower on metal (4%).
424 seconds for 1.4.2 at -6. Time difference to -5 is ten percent on metal, twenty on classical and "other".

I should have included retune -4 ...

The above are measured as follows:
Run 1.4.2 -5 through classical music 5 times. Ditched the first for reasons to explain later.
Same setting through heavier music 5 times. Ditched the first.
Same setting through "other" music 5 times. Ditched the first.
Next setting.
Repeat the above. And again. And again. So, sixteen runs.
For each exe/setting, take median of the 16 classical timings. Median of the 16 heavier music. Etc.
Add median+median+median to get the times above.

Why I ditched the first? The following is "interesting" on this cooling-constrained computer:
* First run on heavier music - immediately after classical music - takes ~10 percent less time than the next four. Both builds. Like 90 or 93 seconds instead of 102/103.
* First run on the "others" section - immediately following the Sodom live album at bitrate > 1100 - takes ~10 percent more time, at the -5 settings. For 1.4.2 -6, the impact is halved. For retune -6 it is gone.
The impact of the first run on the classical music, after the "others", isn't much. Except for the retune -6 when it was immediately after a -5 run, which is lighter - then times are ten percent down.

Re: Retune of FLAC compression levels

Reply #54 –
I'll take all comments into consideration and will come up with a new proposal. I think I'll drop one preset (having 8 by synonymous with 7) and try to tune to a curve as suggested here. My aim is to make sure every next preset will compress better (and slower) for pretty much all material.

I'll also take a good look at the order guessing code.
Music: sounds arranged such that they construct feelings.

Re: Retune of FLAC compression levels

Reply #55 –
Would the retuned compression levels warrant a version bump?

Re: Retune of FLAC compression levels

Reply #56 –
Firing off even more loose thinking-alouds:

Quote from: ktf on
I think I'll drop one preset (having 8 by synonymous with 7)
If you are to drop one up there, then alias together -6 and -7 I think. A user who actively selects -6 or -7 likely intends to encode faster than -8.

Quote from: ktf on
and try to tune to a curve as suggested here.
I'd argue that you merely want convexity on a graph where the first axis is (unlogged!) time. We are waiting seconds to save bytes, not log-seconds. (If size locally around some CPU time t is approximately C - (log t)^2, what then? Good or bad?)

Quote from: ktf on
My aim is to make sure every next preset will compress better (and slower) for pretty much all material.
Not saying that is wrong, but it defeats the idea in your first post. (Especially concerning -2?)
Like, as you said about TAK - it has a -p0 range, a -p1 range etc, and -p2 is faster than -p1m (but more complex and heavier decoding ... in case anyone needs to care by now). Also WavPack with its four complexity levels with -x0 to -x6 on top.

It has been argued that very few will use anything but -0/default/-8 (and above), but let's just not think number of users, but what they actually select it for - presuming those are good reasons. There could be two reasons to select -0 (speed and compatibility, and you shouldn't sacrifice the latter), but apart from that?
-2: is there any other reason to select -2 than to make the most compression out of the fixed predictors? -2 could very well be -1mer6/7/8 (there are a few selected signals where you would partition up significantly) even if that makes it slower than -3.
-3 and -4? Provided default stays "-5" (whatever that is going to be alias for) - is there any reason to select them other than for speed? (You could alias them together ... and well, if -5 = default speeds up, then you could alias -4 into -5 for that matter.)
-6 and -7 ... encode faster than -8. I argued to keep -6 even if it "is bad" but in the (012), (345), (678) pattern it isn't much use in having it at "-6", hence that suggestion to let default be "new -4".
I said loose thinking-aloud, that means I don't have to stay consistent.
So therefore ...

Thinking aloud on a departure from the triplets:
-0 to 2: fixed predictors. In case that is needed, don't break anything.
-3: as light-decoding as possible with LPC. Down to -l 5 or whatever. -4: same decoding complexity, more juice in encoding
-5: sane default. -6: decoding as -5, more juice in encoding. (Bandcamp uses -6.)
-7 and -8: heavier.
Pro: closer to today, and you are already contemplating to reduce the "heavier" presets to two.
Contra: -3 and -4 are quite useless then?
Contra-contra: are -3 and -4 doing any harm?

Quote from: ktf on
I'll also take a good look at the order guessing code.
It did smart things for "-6" purposes :-)
At the risk of opening a can of worms here: it is known that a lot of high resolution material fools it big time.

Re: Retune of FLAC compression levels

Reply #57 –
Quote from: Porcus on
-0 (speed and compatibility, and you shouldn't sacrifice the latter), but apart from that?
Over the years i've read audiophiles use -0 for better sound and space is not a concern. They like the better tagging against the even better sounding wav but this has no tagging standard :)
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Retune of FLAC compression levels

Reply #58 –
Quote from: Wombat on
Over the years i've read audiophiles use -0 for better sound and space is not a concern. They like the better tagging against the even better sounding wav but this has no tagging standard :)
dBpoweramp has offered an "uncompressed" FLAC for that, *cough*, that market segment. Didn't check how - or if I did, I have forgotten it - but forcing verbatim subframes would probably do the job?
Except that spoon had to explain to users when the .flac files were actually significantly smaller. Decoded HDCD would be saved in 24-bit FLAC files with wasted bits. *points at your signature*

Re: Retune of FLAC compression levels

Reply #59 –
The syntax was like this for uncompressed: --disable-constant-subframes --disable-fixed-subframes --max-lpc-order=0 -b 4608
Not sure it took much attention. At this time also i did read real golden ears switched to aif.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Retune of FLAC compression levels

Reply #60 –
A bit late to the party since @ktf already has a revision in the works, but:

1: preset speeds I got to borrow the Ryzen-equipped laptop I also used in the 1.4.x tests. Not only is -6 faster than -5, but also than -4. And -7 is faster than -5. I didn't even notice it was that dramatic, so I went back to the Intel and ran that again, several runs each setting. It turns out on both computers that the timing of the first run is "always" contaminated by what happened immediately before - including, what musical genre was processed. Not by much, but noticeable.
Speeds relative to -5 on
Intel then Ryzen
-3: 72 resp 70
-4: 80 resp 79
-5: 100. Should be added, it is like 15 percent faster than old -6 from which it takes its nominal settings (and makes 0.01% bigger files).
-6: 77 resp 74
-7: 88 resp 85
... all CDDA.


2: -2 and variations. -2b<varying blocksizes> -r <varying>.  TL;DR: The proposal of -b4096 and -r 6 looks good indeed. 
(At this -r level, -b4096 is the best overall, and there is very little benefit in -r7.)

Write-up for CDDA. Note, these are variations on -2 and not on -0 ...
* Block size 4096 produces smallest files on total whenever the max partition order "-r" is at 3 or higher. That is among all subset-compliant multiples of 512 and of 576.
By each of the three genre sections, it is at worst second-best at -r4 and above.
(Details: Classical music marginally improves from -b4608, and 2304 does improve the "other" section. I don't know what makes 2304 beat the neighbours 2048 and 2560 at pretty much everything when we get to -r3 or above.)
* Increasing the -r of course improves, but how much?
It is the "other" genres section that benefits the most - and especially if you want to go for -b4096, then the higher -r gets -b4096 within 0.012 percent of -b2304.
At 4096, moving the four steps up  -r3 to -r7 still improve the "others" section by 0.10, 0.05, 0.03 and finally 0.009 percent.  The totals: 0.05, 0.02, 0.01 and ... 0.003.
There is quite some diveristy in the "other" section, so what makes up the 2304 vs 4096?
At -r6, the jazz benefits from 4096, the techno from 2304 and the pop/rock diverges.  Albums with largest impacts are 0.18 percent either way (Miles Davis (mono) and then Kraftwerk).  Going -7 does virtually nothing to these impacts.  -5 to -6 helps a little. 
* Timings, only for a few block sizes: For-r5 and above it is clearly better than the smaller block sizes. At -r6 the difference to 2048/2304 is above 5%.

High definition tested afterwards - not that it is the most important thing really, but "for sanity". At some -r3 or -r4 and up, 4096 and 4608 are the best. Checked 8192 afterwards, it makes bigger files as partitions are twice as large - you could of course consider -b8192 -r7 to compensate for that, but I got only 0.04 percent out of it. Timings matter surprisingly little going from -r4 to -r6 (-r7 for -b8192) and varying among the reasonable block sizes.

Re: Retune of FLAC compression levels

Reply #61 –
Quote from: Wombat on
The syntax was like this for uncompressed: --disable-constant-subframes --disable-fixed-subframes --max-lpc-order=0 -b 4608
Not sure it took much attention. At this time also i did read real golden ears switched to aif.
If the goal is to make the flac file as big as possible then a very small blocksize like 256 should be ideal.

I tried foo_benchmark on foobar2000 1.6.16 with a ~4GB aif vs wav test using RAM drive and got ~30000x decoding speed for wav but ~13100x for aif, may have something to do with endianness, but would like to see someone with a big endian processor to do such a test.

Re: Retune of FLAC compression levels

Reply #62 –
Quote from: bennetng on
Quote from: Wombat on
The syntax was like this for uncompressed: --disable-constant-subframes --disable-fixed-subframes --max-lpc-order=0 -b 4608
Not sure it took much attention. At this time also i did read real golden ears switched to aif.
If the goal is to make the flac file as big as possible then a very small blocksize like 256 should be ideal.

The idea wasn't to maximize overhead, but to store "uncompressed" for those anti-vaxxers audiophools who think compression ruins sound. Also they can distinguish out FAT from NTFS or Seagate from WD by listening, and maybe if they try careful enough, A-law from LPCM, but only if one is WAVE and the other is AIF(C) :))
On a more serious note, this was introduced when WAVE tagging was a largely unsupported. And yes AIFF was a thing in that market segment. I think it was B&W who started a limited audio "store" with selected recordings, and offered AIFF for ... demand reasons I guess.

Re: Retune of FLAC compression levels

Reply #63 –
Quote from: Porcus on
Quote from: bennetng on
Quote from: Wombat on
The syntax was like this for uncompressed: --disable-constant-subframes --disable-fixed-subframes --max-lpc-order=0 -b 4608
Not sure it took much attention. At this time also i did read real golden ears switched to aif.
If the goal is to make the flac file as big as possible then a very small blocksize like 256 should be ideal.

The idea wasn't to maximize overhead, but to store "uncompressed" for those anti-vaxxers audiophools who think compression ruins sound. Also they can distinguish out FAT from NTFS or Seagate from WD by listening, and maybe if they try careful enough, A-law from LPCM, but only if one is WAVE and the other is AIF(C) :))
On a more serious note, this was introduced when WAVE tagging was a largely unsupported. And yes AIFF was a thing in that market segment. I think it was B&W who started a limited audio "store" with selected recordings, and offered AIFF for ... demand reasons I guess.



Here's the thing, the harder the audio is compressed losslessly, the harder it becomes for the device to decode it.

Audiophiles usually do not use a computer but a media server or some other receiver which decodes their files, and these devices arent the 10th as capable as our computers, due to which, sometimes, they introduce stupid artifacts into the signal as a result of overload, be it jitter or be it anything else cuz the device has to have sufficient ram to store the file and a good processor to decode it in realtime.

that's the reason why audiophiles say that uncompressed lossless sounds better, because its making their setup act better due to the less load on resources... that is also why CDs with SHM sound better to them, because the transport reads it better in real time, rather than doing multiple scans to extract the info with some other material with less transparency.

To other people, Don't start a debate here pls, am not gonna pull needles in haystacks and prove some pretty obvious things that happen due to physics. After decoding everything is the same, because you do it in a PC, as in, decode the audio completely to uncompressed and then play it. Rather than using a real time streaming device that has to manage decoding an extremely compressed signal real time especially with its arm processors and 200mb ram whatever, maybe less... so buffer length and stuff, everything factors into the output they're getting. That's why those media servers or receivers work better with uncompressed material and why audiophiles like them.

I used to be of these opinions as well that these guys are nuts cuz lossless is lossless.. but when I factored the hardware and their setups into the equation, things started making sense... on a computer everything is decoded completely then played, and even if you try realtime, the processors and ddr5 ram is extremely fast and way over the top to be bothered with such things.. that's why it sounds the same to us... cuz it is.

Re: Retune of FLAC compression levels

Reply #64 –
That Seagate WD difference as well... WD drives are faster at 7200rpm compared to Seagate at 5400 (their common versions blue and barracuda), so the same real-time resources thing comes back again... the faster drive would obviously feed the decoder faster and consistently, and if the data rates being output by the drive are in sync with the capabilities of their server/decoder, then yes it would work better in that system

For listening, it all boils down to real-time differences. How fast and how well can the hardware cope with the material being supplied. The smoother that chain is, the better the result would be. Ofcourse that also requires the existence of a point-of-no-return concept, i.e. after you have received sufficient bandwidth and processing power to handle whatever the material needs, optimising it further would be a complete waste of resources and yet if you're hearing differences then yes, you just became an unfortunate victim of audiofoolery.

Re: Retune of FLAC compression levels

Reply #65 –
Quote from: darkalex on
Don't start a debate here pls
Don't write nonsense pls. Even 8 MHz is enough to decode FLAC in real time - https://hydrogenaud.io/index.php/topic,82125.0.html
As for audiophiles, they are just mentally ill and need medical attention.

Quote from: darkalex on
WD drives are faster at 7200rpm compared to Seagate at 5400
Wow! Captain Obvious is right here!

Re: Retune of FLAC compression levels

Reply #66 –
Quote from: darkalex on
Rather than using a real time streaming device that has to manage decoding an extremely compressed signal real time especially with its arm processors and 200mb ram whatever, maybe less... so buffer length and stuff, everything factors into the output they're getting. That's why those media servers or receivers work better with uncompressed material and why audiophiles like them.
FLAC can be decoded without much issue on the ESP8266 which has 112 kilobyte of RAM.

See for example this not specifically optimized port of libFLAC: https://github.com/earlephilhower/ESP8266Audio/blob/master/README.md It says:
Quote
On the order of 30KB heap and minimal stack required as-is.

Considering the resource load: having 'uncompressed' FLAC will indeed lower the (already very low) pressure on CPU, but not on memory and disk/network usage.
Music: sounds arranged such that they construct feelings.

Re: Retune of FLAC compression levels

Reply #67 –
Quote from: darkalex on
That Seagate WD difference as well... WD drives are faster at 7200rpm compared to Seagate at 5400 (their common versions blue and barracuda), so the same real-time resources thing comes back again... the faster drive would obviously feed the decoder faster and consistently, and if the data rates being output by the drive are in sync with the capabilities of their server/decoder, then yes it would work better in that system

RPMs alone aren't really the best way to judge HDD performance.  A 5400rpm drive with higher density platters and more heads could be faster than a 7200rpm drive. 

Re: Retune of FLAC compression levels

Reply #68 –
Quote from: Replica9000 on
Quote from: darkalex on
That Seagate WD difference as well... WD drives are faster at 7200rpm compared to Seagate at 5400 (their common versions blue and barracuda), so the same real-time resources thing comes back again... the faster drive would obviously feed the decoder faster and consistently, and if the data rates being output by the drive are in sync with the capabilities of their server/decoder, then yes it would work better in that system

RPMs alone aren't really the best way to judge HDD performance.  A 5400rpm drive with higher density platters and more heads could be faster than a 7200rpm drive. 

That was off the top of my head, there are many more things like buffer storage, the drivers, the interface in use etc. etc.

not the point I was making,

Re: Retune of FLAC compression levels

Reply #69 –
Quote from: ktf on
Quote from: darkalex on
Rather than using a real time streaming device that has to manage decoding an extremely compressed signal real time especially with its arm processors and 200mb ram whatever, maybe less... so buffer length and stuff, everything factors into the output they're getting. That's why those media servers or receivers work better with uncompressed material and why audiophiles like them.
FLAC can be decoded without much issue on the ESP8266 which has 112 kilobyte of RAM.

See for example this not specifically optimized port of libFLAC: https://github.com/earlephilhower/ESP8266Audio/blob/master/README.md It says:
Quote
On the order of 30KB heap and minimal stack required as-is.

Considering the resource load: having 'uncompressed' FLAC will indeed lower the (already very low) pressure on CPU, but not on memory and disk/network usage.

The resource load is what I am talking about, not the storage or delivery method. The amount of processing needed to decode that file into a PCM signal would significantly lower for an uncompressed flac than compressed. I agree that ESP can decode it, am not saying it can't, am talking about the realtime performance on low power systems.

It's the same argument as older CD Transports, the more properly built they were, the more stable they ran leading to less realtime jitter and etc. Ripping a cd isn't the same cuz the laser goes through that same track multiple times for error correction and stuff to get an accurate rip, that's all out of the picture in realtime.

Same thing about flac, you can decode it completely on a PC which is why you're gonna get a 1:1 lossless signal, but when you try to do it realtime, performance would matter, especially when we're trying to achieve 0ms or something in latency.

This isn't a debate or something, it's just a logical perspective that actually makes sense about why these audiophiles, especially the senior ones with older equipment, hear differences.

Re: Retune of FLAC compression levels

Reply #70 –
Quote from: darkalex on
This isn't a debate or something, it's just a logical perspective that actually makes sense about why these audiophiles, especially the senior ones with older equipment, hear differences.
It has never been demonstrated that these "differences" are real.  What people "hear" is affected by their biases.  I'd say that "senior audiophiles" are particularly susceptible to these biases, but that would be a cheap shot as everyone is affected.  How "old" their playback equipment is (or how expensive) does not matter; nor does the rotational speed of their hard drives.

"Listening is 90 percent mental. The other half is how much your gear costs."

If you claim that FLAC sounds different at "compression 0", then put forth experimental results that prove your hypothesis.  Using foobar2000's "ABX Comparator" component, a test of this sort is trivial to design. Otherwise, you are in violation of this site's 'Terms of Service":
https://hydrogenaud.io/index.php/topic,3974.html#post_tos8

Meanwhile, read this listening test result where a majority of senior "golden ears" concluded MP3 sounds better than FLAC:
http://archimago.blogspot.com/2013/02/high-bitrate-mp3-internet-blind-test_3422.html

Re: Retune of FLAC compression levels

Reply #71 –
Quote from: Porcus on
On a more serious note, this was introduced when WAVE tagging was a largely unsupported. And yes AIFF was a thing in that market segment. I think it was B&W who started a limited audio "store" with selected recordings, and offered AIFF for ... demand reasons I guess.
Of course senior audiophiles use 80-bit aif too :D

Back to the topic, while I can't explain the -b2304 phenomenon, apart from genres, I also pay some attention on the production practices of the audio files. For example this file

https://hydrogenaud.io/index.php/topic,123655.msg1022192.html#msg1022192

After adding some reverb but keeping the 16/96 dual mono format, the optimal blocksize is not too different from many other 96k contents. For example, many audiophile recordings are produced without extensive use of close miking. Of course there are also synthesized music like movie soundtracks and new age music with heavy emphasis on perceived ambience.

PS: I trimmed the file to 15 seconds otherwise I got the "maximum file size allowed is 0KB" error.

Re: Retune of FLAC compression levels

Reply #72 –
Quote from: bennetng on
Of course senior audiophiles use 80-bit aif too :D
Is that a thing? https://en.wikipedia.org/wiki/Audio_Interchange_File_Format#Common_compression_types
(Sample rate is stored as extended precision, but is there any consumer playback chain that handles non-integer rates without resampling?)

Re: Retune of FLAC compression levels

Reply #73 –
Quote from: Porcus on
Quote from: bennetng on
Of course senior audiophiles use 80-bit aif too :D
Is that a thing? https://en.wikipedia.org/wiki/Audio_Interchange_File_Format#Common_compression_types
(Sample rate is stored as extended precision, but is there any consumer playback chain that handles non-integer rates without resampling?)
Of course it is a joke with that smiley. What I really think about aif is that many earlier DAWs are Mac based and the accompanied hardware are not cheap. Not only the recording interfaces, but also the processing hardware in additional to the computer itself.

Re: Retune of FLAC compression levels

Reply #74 –
Concerning "-0" candidates: when will I/O constraints kick in, really?

I made three 4 GB FLAC files to recompress. Copy from Powershell takes like, 40 seconds (SSD to same SSD, NTFS, Win11). Using the retune build to recompress (again to the same SSD):
-0 --no-mid-side (that has an implicit -r3) takes around 170 seconds
-0 -r7 takes around 170 seconds. Maybe a percent difference. -
-1 takes 190 seconds, so brute-forcing the joint stereo / dual mono choice is enough to make time increase.
And -r7 is enough to make a difference from -2. Like, -2r7 takes 14% more time than -2r3.

I guess flac.exe must use more time to write to the files in segments (of I think 10 MB?) than built-in copy does, so ... ?