Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Retune of FLAC compression levels (Read 16953 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Retune of FLAC compression levels

Reply #75
It is really irritating to see people discuss the possibility of decoding audio beeing a relevant computational bottleneck in any way while i can render 3D images in realtime without stutter with a machine that is 12 years old by now.

Re: Retune of FLAC compression levels

Reply #76
Sure that Rockbox users would be happy to carry your old or new computer around for portable audio playback ... ?

(I'll tell you what is really irritating about this thread: over again catching myself at
* not being able to start at "0" when counting FLAC presets in a diagram
* reading colour switched, but glancing at the wrong diagram still
* forgetting for the Nth time about that log scale
... and realizing I got to either wrong conclusions or to being surprised at what should be "clear" from the initial post ...
That's annoying. ktf must have quite some patience ignoring all the pebkac.)

Re: Retune of FLAC compression levels

Reply #77
I really don't understand why people here start attacking a logical perspective with borderline mockery or hurtful language, what I have stated is clearly not my own experience or perspective, but is a logical problem that can happen in certain circumstances, but somehow y'all are relating the examples to sheer innuendo and generating mockery through that.

Even the dev didn't respond in such a trash way, I wonder why y'all are doing that. It's a fact that on computers the whole processing happens in the ram and happens so quickly that you cannot get any differences, but when you do that on some cheap arm-based processor with android 2.0 running, things become different... I don't know how to put my point further.

Add a trash harddrive to that android thingy, and boy are you gonna have trouble listening... that's not innuendo but a legitimate problem that actually exists. I wasn't taking the side of audiophiles but rather saying that some of them could actually have such a situation where the hardware is better supported to read uncompressed files easily and smoothly.

A simple demonstration of this would be to prepare any digital file and fool around with the block size, then stream it, both the files would be lossless or have the same content if they are lossy, but the one with smaller blocks would stream better and hence sound better. However, upon a full decode, they will null.

It's not rocket science. Why do you think some codecs handle this better than others? Streaming is literally the same thing as playing from a CD or real time hard disk if the internet speed is way higher than the material's rate.

The file optimised for streaming would obviously work better than the one with larger sizes. Same logic goes for interlacing images or videos too... post some uninterlaced images on a website and load it on a slow computer, you'll see nothing until the full image loads or you'll see only the parts that have completely loaded. Replace it with an interlaced image and you'll start seeing the image partially whilst the rest of it is loading, and in the end you'll get the same image as the uninterlaced one.

Ofcourse these particular techniques aren't relevant to audio, but they sure are relevant to content delivery and bandwidth and therefore help prove my point perfectly, and logically.

I personally have experienced this in-car systems or old mobile phones as well. Higher quality or bigger files struggle to play, they could be choppy or laggy or 100 other things you name it that can be caused by stuttered loading of the file.

It's not innuendo or a 1-in-a-billion type problem. It applies to real-world and happens frequently. TV's as well, the Android/Smart ones, like my Samsung from a few years ago, cries out whenever I try playing some high bandwidth media on it or a huge VBR aac, it works the best with CBR files of moderate length, like songs.

That's because the TV wasn't prepared to do this, it was made for people to causally play their mp3s or their handycam videos, not use as a proper audio system, but people still do and hence pay the price with reduced fidelity... which gets blamed to the format cuz no one will accept that their cheap equipment could be at fault..

See it from a pragmatic perspective people, rather than an all-science laboratory perspective... because that works only in labs, not in the real world. People do crazy things and form stupid conclusions about various things because they didn't know how to use them properly...

Tell me if either of the things I mentioned are textbook-type examples that can't happen in the real world... or any innuendo that I wrote.

I still gotta take my breakfast, so pardon me if there are some grammatical errors or repetitions. No intents to offend anyone.

Re: Retune of FLAC compression levels

Reply #78
I guess flac.exe must use more time to write to the files in segments (of I think 10 MB?) than built-in copy does, so ... ?
I really don't understand how you got to that conclusion. It is not that FLAC preset 0 does nothing, it still calculates an MD5sum and the bitwriter code also takes quite a bit of time. Could very well be that is the best explanation for the difference between simply copying and recompressing.
Music: sounds arranged such that they construct feelings.

Re: Retune of FLAC compression levels

Reply #79
I wonder why y'all are doing that.
You stepped over the line when you started talking about "senior audiophiles hearing differences".  If the file decodes, it will sound the same regardless of FLAC compression level.  That kind of woo is typical on other audio forums, but prohibited here.

Long ago your point about CPU power being a bottleneck might have been valid ("Android 2.0" has been gone for 13 years), but modern ARM and x86 chips have no issue with FLAC at any compression.

Re: Retune of FLAC compression levels

Reply #80
A simple demonstration of this would be to prepare any digital file and fool around with the block size, then stream it, both the files would be lossless or have the same content if they are lossy, but the one with smaller blocks would stream better and hence sound better. However, upon a full decode, they will null.
Exactly this sounds like the typical senior audiophile drivel.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: Retune of FLAC compression levels

Reply #81
I guess flac.exe must use more time to write to the files in segments (of I think 10 MB?) than built-in copy does, so ... ?
I really don't understand how you got to that conclusion.
I might be completely wrong of course, but: When an application passes 10 MiB to the write queue 400 times over, is that really going to be handled in the same overall time as when the operating system calls (the write part of) a copy command?

It doesn't look that way. On this SSD I got three .flac files of around 4 GB each.
Initially encoded with fixed predictors, and all the following are done with the retune build, some "sync d:" commands in between for good measure:
151 seconds: Measure-Command {flac -d -ss --force-rf64-format  Image*.flac} Deleting the flac files and sync and then:
147 seconds: Measure-Command {flac -0r3 --no-mid-side -ss Image*.rf64}
143 seconds: Measure-Command {flac -0fr7 --no-mid-side -ss Image*.rf64} this with -f rather than deleting the .flac files
122 seconds: Measure-Command {flac -0fr7 -ss Image*.rf64} (-M is good here!)
186 seconds for flac to flac: Measure-Command {flac -0fr7 -ss Image*.flac} <------ r7
187 seconds for flac to flac: Measure-Command {flac -0fr3 -ss Image*.flac} <------ r3
109 seconds: Measure-Command {flac -t  Image*.flac}
107 seconds: Measure-Command {flac -ss -t Image*.flac} this time using -ss just for sanity check
Measure-Command {copy Image*.flac copies}, which copies the flac files to a "copies" directory on the same SDD, takes ... <checks again> not too consistent duration, but the "forty" seconds isn't the worst estimate.
Measure-Command {copy Image*.RF64 copies} takes 55 to 57 seconds.

So ... what to make out of this, when -r7 doesn't hurt compression time? (It does when the baseline is -2 ...)
Hunch: minor bottleneck at every write?
Other suggestions?

Are you actually reaching the point where nobody who will use the encoded data (rather than discarding it and recording the timing figure) will need the extra speed? Encoding speed, I mean?
If so, that is ... awesome. But sure employ an ancient CPU with a fast SSD to crack that hypothesis  ;)

Re: Retune of FLAC compression levels

Reply #82
So ... what to make out of this, when -r7 doesn't hurt compression time? (It does when the baseline is -2 ...)
That MD5summing and bitreading and/or bitwriting takes 70 seconds? flac -t takes about 70 seconds longer than copying, and reencoding (which is decoding and then encoding again) takes about 70 seconds longer than flac -t? Makes perfect sense to me?
Music: sounds arranged such that they construct feelings.

Re: Retune of FLAC compression levels

Reply #83
@darkalex

I don't think the type of person who typically uses lossless for audio quality is using ancient or inadequate hardware.  A media server or receiver may be a 10th as powerful, but the hardware and OS/firmware are designed specifically for the task.

My 23 year old K6-III with 256M RAM plays my FLAC files compressed with "-m -b 4096 -l 12 -r 7 -p -A subdivide_tukey(13/10e-1);welch;hann;flattop" just fine with a USB 1.1 flash drive.  I also have a 13 year old Android 2.1 device that doesn't break a sweat playing these files.

I think the type of people who think uncompressed FLACs sound better have decent hardware, and they're probably also the people buying "audiophile grade" routers and RAM sticks.

Re: Retune of FLAC compression levels

Reply #84
The last time I experienced stuttering/dropouts (when I believed I have correctly configured my system) was a Pentium II 400, 192MB RAM PC, playing APE files with "Extra High" compression mode in foobar2000, with SSRC turned on.

And yes, the OS/firmware does matter, just an example, and read the reply right below it:
https://www.audiosciencereview.com/forum/index.php?threads/about-flac.14760/post-460345

Speaking of Samsung, the old smart TV in my home has a habit of suddenly setting the volume to max when turned on. This happens every 2 or 3 months or so. No real damage happened as the TV is not connected to any high power audio system, it just uses the built-in speakers, but the annoyance is real, especially when someone turn on the TV during midnight.

Re: Retune of FLAC compression levels

Reply #85
So ... what to make out of this, when -r7 doesn't hurt compression time? (It does when the baseline is -2 ...)
That MD5summing and bitreading and/or bitwriting takes 70 seconds? flac -t takes about 70 seconds longer than copying, and reencoding (which is decoding and then encoding again) takes about 70 seconds longer than flac -t? Makes perfect sense to me?
Not depending on -r, is that really reasonable? If so, give -0 the -r6 treatment because why not?
But as --no-md5-sum does make a difference, I just tested that: we are not "hitting hard" a constraint, and faster bitwriting is possible indeed. (Oops pardon my ignorance: "bitwriting" is the process of writing the bit stream (in the format's order of bits and bytes, including endianness) to RAM before sending it off to file?)
But OTOH, from numbers @cid42 posted at https://github.com/xiph/flac/issues/585 it seems that buffering does matter.


Thinking aloud (and again at the risk of invoking the last line of Reply #76), here is a "possible" but likely "not uncontroversial" retuning principle: retune "even number" presets and simply drop (= alias together) the odd ones.
-0 for "as fast as possible" (within suitable constraints, including: with MD5).
-1: aliased together with -0. Admittedly this makes not so much sense if -0 must stay dual mono for compatibility reasons.
-2: make the smallest out of fixed predictors (within subset, and likely sticking to -b 4096 unless also elsewhere adopting a proposal of -b8192 for say rates > 144kHz)
-4 and -3 and -5 aliased together with -4: default. Fast. -l limited to 8 to be sure not to break anything.
Because -4 and -5 will now be the same, traditionalists who want default to be -5 are still satisfied.
-6, and -7 aliased together with -6: reasonably fast, but allowing -l 12 (so not selected by default)
-8 is -8, can be tuned but is for those who think they want more than -7.
Now you can consider the following two heresies:
--FAST (capitalized) for -0 --no-md5-sum (I read from the IETF draft that you agree to accept "0" if the MD5 sum is not known, and it leaves room for the following interpretation: it is perfectly OK not to calculate it, for then you don't know it)
--BEST (capitalized) for where I was thinking aloud of a "-9" to avoid the nasty surprises. Say if for a high-resolution file the algorithm proposes a prediction order of 18, then this setting can try 16 and 20 without going full -e on it.

Completely different from {0 1 2} {3 4 5} {6 7 8}, but ... possibly worth thinking over before rejecting.

Re: Retune of FLAC compression levels

Reply #86
I really don't understand why people here start attacking a logical perspective with borderline mockery or hurtful language
Maybe you have a point, but your perspective is not "logical" and you shouldn't confuse it by logic. That jittery problem you hint at is resolved by what we in everyday speech call "buffering".
Digital dropouts easily get so ugly that you don't need to be, ahem, "senior audiophile" to catch it. Indeed, glossing over such network issues that you describe - making for audibly graceful failure - that is more of an engineering art.

A simple demonstration of this would be to prepare any digital file and fool around with the block size, then stream it, both the files would be lossless or have the same content if they are lossy, but the one with smaller blocks would stream better and hence sound better. However, upon a full decode, they will null.
Here you presume graceful failure - that is, sound that is not perfect but so good you need to concentrate to hear the imperfections.
Sure you can interpolate over very short holes that way, but lossless codecs easily have block sizes of 0.09 seconds and up.

Ironically, ffplay still chokes at small FLAC blocks (under 128 samples). It isn't because 127 samples in a block is inherently bad (although it isn't as good as you think), it is because ... well, bad code. Sure Slim Devices had trouble with 4608. Why trouble with 4608 when 4096 plays fine? Not because "4096" is an upper bound on fidelity, but because ... well, bad code.


It's not innuendo or a 1-in-a-billion type problem. It applies to real-world and happens frequently. TV's as well, the Android/Smart ones, like my Samsung from a few years ago, cries out whenever I try playing some high bandwidth media on it or a huge VBR aac, it works the best with CBR files of moderate length, like songs.
Bad decoders refusing VBR is an old and known problem, but don't confuse that with "golden ears" falling for placebo.
And don't confuse that false equivalence by anything "logical".

Old Android? Android was released in 2008. Have a look at https://hydrogenaud.io/index.php/topic,123374 .



PS @Replica9000 :
my FLAC files compressed with "-m -b 4096 -l 12 -r 7 -p -A subdivide_tukey(13/10e-1);welch;hann;flattop"
That doesn't make for much of an extra decoding load. Playback-wise it is like -8r7. "-m" will do joint stereo optimization (but even -2 does that). -b4096 is default. -l 12 is what you get in -7 or -8. Then -r7 means that it might change an encoding&decoding parameter every 32 samples (compared to every 64 samples for standard -7 or -8), and "might" means it only does so in the frames it thinks it is worth it (it means you have to store twice as many of that parameter, so ...). Everything from "-p" and on will only brute-force check several predictor vectors (including, how many bits precision they have) and shouldn't matter for decoding workload.
Several aspects of FLAC work that way: instead of nesting up successive rounds of number-crunching that have to be "undone one by one", it "tries several light ones" and picks the one that gives smaller files and throws away the rest. If it tries a thousand different ways upon encoding a block, then that takes more time - but the decoder doesn't know it tried so many and it doesn't care. You made 9 worse attempts? Oh you made 999 worse attempts? Decoder doesn't know, decoder doesn't care.

Re: Retune of FLAC compression levels

Reply #87
Why don't leave open the pull request? Is the idea abandoned, or you want another re-tune of the presets?
https://github.com/xiph/flac/pull/576


Re: Retune of FLAC compression levels

Reply #89
I suspect this is what happens when git-ignorants (like myself) see a PR "closed" with no comment as to whether it means "closing this particular one, which won't happen, in order to propose something new" or "abandoning the whole idea".

Two words then would have saved you more words now ;-)

Re: Retune of FLAC compression levels

Reply #90
I am an AMD Ryzen user and have noticed that certain software when compiled with optimizations for AMD, result in huge performance jumps on my PC.

I came across this today:

https://www.amd.com/en/developer/aocc.html which I believe is the official Ryzen optimised compiler by AMD.

I came across this CLANG 16 binary by Netranger, and it was a day and night difference in performance from the rareware libraries I was using to this day. 24x speed with Rareware x64, whilst 36x with CLANG16 x64 on 24 bit 96khz material. Blew me away.

Hence my question about a FLAC binary that's optimized for AMD Ryzen. It would be a huge help.

Re: Retune of FLAC compression levels

Reply #91
I do agree that this should - at least ideally - be part of this discussion. We (or at least I) have been doing some head-scratching over the performance of flac level "-3" in ktf's lossless test - apparently that is an AMD thing. (But official compile ran on AMD. Differences across CPUs ... but I guess with the same build - posted at https://hydrogenaud.io/index.php/topic,122508.msg1024512.html#msg1024512 )


Anyway:
At least before deciding on re-assigning the presets, performance tests across platforms should be welcomed ... so, can you stay tuned for when the next version arrives? (Are you able to compile?)

You could of course try a few builds already, but ktf is tweaking one behind-the-scenes algorithm that does affect speed, namely which prediction order (i.e. "how long history") the encoder should use. Anyway, there were quite a few builds posted in https://hydrogenaud.io/index.php/topic,123234.0.html
Various posts where users tested them against each other on different CPUs, including here: https://hydrogenaud.io/index.php/topic,123025.0.html

Re: Retune of FLAC compression levels

Reply #92
OK, tried some flac v1.4.2 encodes on an old Nokia Lumia 520 smartphone with LineageOS and foobar2000 mobile. The three trials are all non-buffered and read from storage, arranged in different order just in case if test order does matter. Also, the phone was rebooted after each set of trial to avoid caching.

X

Tested album, 2 discs combined into a single image:
https://www.discogs.com/release/2836867-Andrew-Lloyd-Webber-The-Phantom-Of-The-Opera
Code: [Select]
trial 1
0b128    92.283x  
0b4096  142.455x
0       140.347x
0r7     136.920x
0b2304  145.963x

trial 2
0b2304  118.903x
0r7     132.194x
0       127.061x
0b4096  129.846x
0b128    92.663x

trial 3
0r7     125.965x
0b2304  140.878x
0b128    92.064x
0b4096  137.545x
0       138.527x

file sizes:
0b128   618463524
0       582240567
0r7     581839033
0b2304  580637454
0b4096  580333909

I also noticed on desktop both my old i3-4160 and the newer i3-12100 decode AAC much faster than opus with similar bitrate, would like to see if it is the case with this smartphone.

Code: [Select]
trial 1
AAC-CBR256    84.851x
AAC-VBR249    86.096x
Opus-CBR257   30.964x
Opus-VBR258   30.909x

trial 2
AAC-VBR249    85.572x
Opus-VBR258   30.650x
AAC-CBR256    86.318x
Opus-CBR257   30.856x

For those who think the bitrates are too high or too low, or thinking about Opus' mandatory 48kHz treatment I think they can do their own benchmark with what they what to test.

I have some other benchmarks on this smartphone as well, but with full RAM buffering and much shorter duration, so different test conditions:
https://hydrogenaud.io/index.php/topic,123025.msg1016437.html#msg1016437

Re: Retune of FLAC compression levels

Reply #93
I do agree that this should - at least ideally - be part of this discussion. We (or at least I) have been doing some head-scratching over the performance of flac level "-3" in ktf's lossless test - apparently that is an AMD thing. (But official compile ran on AMD. Differences across CPUs ... but I guess with the same build - posted at https://hydrogenaud.io/index.php/topic,122508.msg1024512.html#msg1024512 )


Anyway:
At least before deciding on re-assigning the presets, performance tests across platforms should be welcomed ... so, can you stay tuned for when the next version arrives? (Are you able to compile?)

You could of course try a few builds already, but ktf is tweaking one behind-the-scenes algorithm that does affect speed, namely which prediction order (i.e. "how long history") the encoder should use. Anyway, there were quite a few builds posted in https://hydrogenaud.io/index.php/topic,123234.0.html
Various posts where users tested them against each other on different CPUs, including here: https://hydrogenaud.io/index.php/topic,123025.0.html


That's the thing I'm talking about, the guys were talking about AOCC (that compiler from AMD) which wasn't supporting Zen4 yet, however, with a newer update released later last year, AOCC now supports Zen4

Hence, a developer with these resources can indeed build a compile that should be optimised for AMD Ryzen

Thankfully I got the same results as you, I was surprised to see that the rareware-x64 compile was performing so poorly on ryzen, which is probably why the CLANG compile by Net Ranger blew me away when I ran it today. Indeed porcus, the results you're getting are in the same ratio for my system as well, the CLANG16 compile was 1.5 times faster than the x64 version I procured from Rarewares. There was absolutely no change in compression except the huge reduction in encoding time. from 23-24x to straight 37-38x on 24 bit 96khz files.

Even this seem slow but compared to the past results, it gives me hope that a properly optimized build for AMD is indeed possible which could further boost the performance of FLAC.

The new release is definitely worth waiting for, I'll indeed stick around for it, but in the meantime, could anyone here please build a 64 bit AMD-optimised version for FLAC 1.4.2? That would be a huge help.

Re: Retune of FLAC compression levels

Reply #94
by the way, geniuses, with senior audiophiles, I wasn't referring to people with ears made up of gold... I was talking about people who are senior in age, the 50-60 year old crowd which isn't as computer savvy as us and are possibly using their 2002 marantz receiver because it's dac sounds warmer to them.. pun intended.. equipment like that is not made with intel processors smh... those Motorola or other Chinese chips could indeed mess up if you give them an input or a task too complex. The same goes for car receivers which are manufactured even today, except the extreme top of the line, none of them are using the latest and greatest in processing power or their operating system. A few lines of unoptimised code in their decoding mechanism and there you have a stuttering player that cannot decode a 256mb flac file without choking thrice every minute.. I have seen this happen, literally in my brand new car or in my friend's cars... just connecting them with AirPlay through a lightning cable sometimes makes the devices hang.. which is a fact.

Maybe start using a pragmatic lens and view the reality rather than throwing slurs and bookish know-how at me. No one is siding with anti-vaxxer style audiophiles that use gold-plated ram, but rather with actual human beings who seem to be stuck with inferiorly manufactured equipment. Those equipment would still be used by these people irrespective of what you tell them because they love the sound that the device is creating from other mediums and they'd rather not replace their beloved DAC just because they cannot get it to play some huge file properly. Same goes for car receivers, the airplay may very well not work in my car, but it is integrated really well with all the sensors present inside, due to which, no matter what anyone tells me, I'm not gonna change that device just because it performs poor in one area. There are other things to consider as well. That's what a practical view is. Tell me which part of this does not strike true in the real world. Rather than slurring or mocking a genuine perspective, maybe develop some patience and humility. Certain devices like these which are integrated have various other limitations as well, for instance, only supporting SATA1.0 hard drivers or the older IDE hard drives. Try doing benchmarks on such trash equipment. Seeking a file would crash you out in seconds. Yet these machines remain heavy in use because barring the particular problematic aspect, they perform reliably and excellently in all other facets.

No one is saying that your work is trash or that the audiophiles that behave like karens are right, rather all I'm saying is treat some of them with respect because even they have genuine points at times. Going all psycho on them is neither a pragmatic nor a humble way to treat anyone.

All of the above is not some fairytale or highly specific text book style example that has no relevance to the real world, but is literally a bunch of incidents that I keep striking with every now and then.. which is why some points do seem valid to me, and by now, hopefully to you as well.

I'm done posting on this particular facet, irrespective of whatever I just said about humility, I know that talking this further would only escalate it to hostile levels, which is clearly not my goal here. So I'm done talking about this topic.

Peace

Re: Retune of FLAC compression levels

Reply #95
I came across this today:

https://www.amd.com/en/developer/aocc.html which I believe is the official Ryzen optimised compiler by AMD.

I came across this CLANG 16 binary by Netranger, and it was a day and night difference in performance from the rareware libraries I was using to this day. 24x speed with Rareware x64, whilst 36x with CLANG16 x64 on 24 bit 96khz material. Blew me away.

Hence my question about a FLAC binary that's optimized for AMD Ryzen. It would be a huge help.

could anyone here please build a 64 bit AMD-optimised version for FLAC 1.4.2? That would be a huge help.

@Wombat , does any of your compiles qualify?



PS:

by the way, geniuses, with senior audiophiles, I wasn't referring to people with ears made up of gold... I was talking about people who are senior in age, the 50-60 year old crowd which isn't as computer savvy as us and are possibly using their 2002 marantz receiver
That would be my gear. DAC/amp/speakers, I think "2002" is spot-on!
Handling 96kHz/24, but 32kHz needs to be resampled.

Re: Retune of FLAC compression levels

Reply #96
My GCC compile of 1.42 in the thread you already linked to is still faster as a recent Clang 16 compile of mine on my Ryzen Zen 3 5900x.
After haswell the additional CPU features doesn't seem to do much for the flac code.
I may build one for Zen 4 later if wanted.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

 

Re: Retune of FLAC compression levels

Reply #97
On my Ryzen 5850U, building with AOCC 4.0 is the same performance as regular Clang 16.  Building with -march=znver3 or the generic x86-64-v3 doesn't make a difference either.  For 44.1/16 audio, GCC has about a 4 second advantage with my 1.2GB file over Clang with ASM optimizations disabled, and about a 4 second disadvantage compared to Clang when enabled.




Re: Retune of FLAC compression levels

Reply #98
This x86-64-v3 option seems even cleaner as the haswell one for a wider range of CPUs, interesting. I later may add a GCC and Clang 1.42 version with that.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!