Skip to main content

Topic: Wondering about non-subset FLAC and non-standard-level options (Read 882 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • radorn
  • [*][*]
Wondering about non-subset FLAC and non-standard-level options
Before I spend a ton of time trying this I just wanted to see if someone already looked into it.

First of all, is anyone familiarized with how compatible are different programs with FLAC files encoded with --lax and crazy options outside the bounds of standard levels (1-8)? Software like foobar2000, MusicBrainz Picard, CUETools, Audacity, GoldWave...
I'm not really interested in streaming these files or necesarily share them, so compatibility would be limited to the programs I use. I'll find about that when necesary, but these avobe would probably be the main ones, so if anyone has experience in that direction, please, share.

Secondly, and no less important. All this trouble isn't worth it if there isn't a substantial improvement in compression from straying away from regular config.
FLAC is very fast compressing, even at level 8, so I wouldn't mind even a 50% speed penalty for, let's say, a 10% or 5% improvement in compressiond, if that can be achieved. So, what's the reality on this issue, and what tweaks are advisable?

  • pdq
  • [*][*][*][*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #1
A five to ten percent improvement over flac -8? Highly unlikely.

  • arkhh
  • [*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #2
I don't know about compatibility. But FLAC encoder doesn't really do much better than standard level 8... if time is no concern -e and -p switches for about 1%(at best) improvement can be used, but they are really slow, like 10 times more time to encode.

If you need more than that you have to switch to another encoder FLAKE or CUEToolsFLAKE compress a little better, while still producing subset files... and can compress another little bit more at non subset settings.

If you absolutely need more compression, it's time to switch format, the best performers that are quite usable are TAK and Monkey's Audio at the higher settings.

My conclusion is this: just don't bother. There is no magic trick that makes FLAC encoder do much better, and then the saved disk space is minimal. If you are running out of storage it's just better to switch to a larger hard drive.

For reference:
Lossless codec comparison https://hydrogenaud.io/index.php/topic,107990.0.html
And here are some results that include flake https://hydrogenaud.io/index.php/topic,97310.msg812229.html#msg812229
  • Last Edit: 17 May, 2017, 05:17:01 PM by arkhh

  • saratoga
  • [*][*][*][*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #3
http://wiki.hydrogenaud.io/index.php?title=Lossless_comparison

You can get about 4% improvement in compression for about a factor of 10-100 reduction in encoding speed, mostly independent of format.  If you have a GPU you can get slightly better compression out of FLAC at reasonable encoding time by using OpenCL as well. 


  • pdq
  • [*][*][*][*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #4
You might consider lossyFLAC.

  • radorn
  • [*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #5
I see. Thanks to all. I guess there's not much point, as I feared.

Now to specific answers:

I do not wish to stray from FLAC, really. A competing format would need to offer a considerable compression advantage to give up the conveniences of licensing, compatibility, and speed that FLAC offers.

About the alternative encoders: Are they safe/stable?

And about lossyFLAC: No, thanks, I prefer to stay lossless with my lossless. If I go lossy, then I go lossy for good. Sacrificing noise-floor while staying lossless otherwise doesn't seem like an appealing idea to me, even if it may be an acoustically solid concept.

OFF-TOPIC: This may be stupid, and may well be entirely misguided, but my idea of near-losslessness would imply bandlimitation and a coding scheme based on frequency analysis instead of waveform replication. Something like a lossy format, but without the psychoacoustics part, I guess. I wonder how well that would work, and how good it'd be for transcoding from already lossy material. Perhaps, such an encoder could be made for an already existing lossy format, like Opus. "LLOpus"?
  • Last Edit: 18 May, 2017, 04:02:49 AM by radorn

  • Porcus
  • [*][*][*][*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #6
There is one thing that can possibly save you space, and that is if you have one track per file and embed large album art in every file. But I don't know any "automated" software that compares all embedded art per folder, extracts it, puts it in the folder, deletes it from the files and reclaims the space.

All this trouble isn't worth it if there isn't a substantial improvement in compression from straying away from regular config.
FLAC is very fast compressing, even at level 8, so I wouldn't mind even a 50% speed penalty for, let's say, a 10% or 5% improvement in compressiond, if that can be achieved. So, what's the reality on this issue, and what tweaks are advisable?

I suppose that by "10% or 5%" you mean further reduction in file size. Consider https://xiph.org/flac/comparison.html , the first graph. Going from FLAC -8 to TAK could save you four percent it seems. Going from FLAC -8 to most badass OptimFrog gives you seven. OptimFrog also decodes slowly. Have also a look at this: https://hydrogenaud.io/index.php/topic,97310.msg812720.html#msg812720
FLAC -8: 66.49%
FLAKE r264 -12: 66.28% (that's non-subset)
OPTIMFROG BESTNEW: 63.56%

Here, nothing could improve five percent over FLAC -8. Safe to say that if you have a music collection close to the ones tested, then there is no lossless codec setting that can give you ten percent.  And you want to stay with FLAC ...

  • radorn
  • [*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #7
If I were to change FLAC for something else, my next preferred codec would be WavPack, I think. Licensing is on par with FLAC, I believe, and it has features FLAC doesn't have, but is less supported all arround and it seems to be slower.
I'm not really interested in changing FLAC for something else, especially slower, sparsely supported, closed sourced alternatives, just for a small advantage in compression. The compression advantage to overcome these objections would have to be HUGE... 15-20% over FLAC to even start considering.
I just wanted to see how much FLAC can be currently pushed and how worth it was to go that route... Not much, it turns out  :)

As for embedded art... I pretty much avoid it like the plague, so not much to improve in that department :D

  • PoisonDan
  • [*][*][*][*][*]
  • Members (Donating)
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #8
If you want better compression than -8 out of FLAC, you could try --super-secret-totally-impractical-compression-level (not a joke!). However, it's called "impractical" for a reason: it's extremely slow, and probably won't save a whole lot of space.
Over thinking, over analyzing separates the body from the mind.

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #9
If you want better compression than -8 out of FLAC, you could try --super-secret-totally-impractical-compression-level (not a joke!).
AFAICS this switch was removed in FLAC 1.1.3 (more than 10 years ago).

  • saratoga
  • [*][*][*][*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #10
OFF-TOPIC: This may be stupid, and may well be entirely misguided, but my idea of near-losslessness would imply bandlimitation and a coding scheme based on frequency analysis instead of waveform replication. Something like a lossy format, but without the psychoacoustics part, I guess. I wonder how well that would work, and how good it'd be for transcoding from already lossy material.

Not stupid, actually various codecs like this have been developed:

https://books.google.com/books?id=a1RULRErhOYC&lpg=PA354&ots=_z5y7ERJuM&dq=LTAC%20compression%20format%20DCT&pg=PA343#v=snippet&q=LTAC&f=false

  Look up LTAC for instance:

https://en.wikipedia.org/wiki/Lossless_Transform_Audio_Compression

It transforms using the DCT like a lossy codec, then codes the residual.  It performs worse than time domain prediction however, so its no longer used. 

Re: Wondering about non-subset FLAC and non-standard-level options
Reply #11
Why do you need such high space savings? Of course, having smaller files is good, but is it just that, or do you have a specific use-case that requires it?

I used to encode to FLAC using FLACUDA (now FLACCL) at level 12 (non-subset) and the files were about 1-2% smaller than subset @-8 from what I remember. They also played quite well on foobar2000, but my rockboxed sansa clip+ hiccuped on them now and then. It would basically randomly fail to decode a file, then skip to the next. I was still getting the best battery life out of the clip+ with it, though (followed by musepack). So FLAC was great.

I switched to using TAK after that, transcoding to mpc for lossy use, because I couldn't afford getting a bigger HDD, being a poor student that I was at the time. But in the end I ditched TAK, because of bad support outside Windows and lack of unicode filename support in the encoder.

I ended up going with WavPack hybrid.

Yes, it's quite slower at encoding than FLAC (22x vs 80x on an old i5-2540m). My library is also 6.78% larger than TAK (only about 1% larger than FLAC!) now. But I encode and tag only once and I have both a lossy and a lossless library in one. Which also means I don't have to worry about any linux converters screwing up/losing the tags when converting to lossy, or having to re-scan RG tags after conversion. That's more than worth it to me.

The support is good, too, though worse than FLAC (no out-of-the-box support on Android, for example). The decoding is acceptably fast and doesn't measurably impact battery life compared to other codecs from what I saw.

  • Wombat
  • [*][*][*][*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #12
I think CUEtools flake out of the 2.16 CUEtools package makes most sense. All older flacCL and flake versions were not further optimized with non subset settings.
CUEtools flake -8 compresses as good as official flac -8 -ep but at a nicely usable speed. The last compression improvements of CUEtools flake are not part of flacCL.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

  • Porcus
  • [*][*][*][*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #13
Wasn't Reference FLAC updated (improving compression) after the the CUETools Flake improvements, thus obsoleting the comparisons one made back then?
  • Last Edit: 18 May, 2017, 01:52:59 PM by Porcus

  • Wombat
  • [*][*][*][*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #14
Before lvqcl pimped flac CUEtools flake was clearly compressing superior. Today it is like written above, -ep is barely usable.
  • Last Edit: 18 May, 2017, 01:57:26 PM by Wombat
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #15
Wasn't Reference FLAC updated (improving compression) after the the CUETools Flake improvements
IIRC it was the other way around.

Before lvqcl pimped flac CUEtools flake was clearly compressing superior.
It was ktf (again IIRC), not me.

  • Wombat
  • [*][*][*][*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #16
  • Last Edit: 18 May, 2017, 02:31:43 PM by Wombat
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

  • Porcus
  • [*][*][*][*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #17
Thanks. Anyway, the OP can still forget the "substantial improvement in compression" within the FLAC format I guess.

  • radorn
  • [*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #18
--super-secret-totally-impractical-compression-level (not a joke!).
nice xD
Look up LTAC for instance:
But that IS lossless. I was talking about a different approach to "near-losslessness" than lossyWAV.
Not lossless if we look at the bits, but not much more lossy than the frequency based reconstruction a 0-level equalizer would do, I guess. Just like any lossy codec, but without all the artifact-inducing psychoacoustic evaluation and quantization.
I wonder how well that kind of approach could work. I guess someone would have done it already if it was good for something. I'd find it interesting if it were to be incorporated as a special mode in an existing lossy codec (preferably Opus, I guess). Perhaps with an optional decreasing response near 20K to save some bits. I guess my concept of near-losslessness could be defined as high fidelity reproduction of the full range of human hearing and zero risk of artifacting with no-psychoacoustics.

Why do you need such high space savings?
[...]
I ended up going with WavPack hybrid.
Yes, it's quite slower at encoding than FLAC (22x vs 80x on an old i5-2540m).
It's not so much "need" as it is "can I and how worthy is it?"
Also, I'm not interested in hybrid coding. It's interesting but seems like a hassle to me.
My use case is like this: Whatever I have that is lossless, if it's worth it for me to keep it so, I FLAC it. If it's not, but I still want to have it in some form, I Opus it to whatever quality I deem it worth for the particular case.
For portable use, as I just have an android phone with shitty earbuds, not a dedicated player with lots of storage, I just transcode to Opus on demand at very low bitrate (currently 50-60), which seems to be pretty good for outdoors listening, and it also encodes really fast.
If I were to use a hybrid codec, I would have to choose a particular level of lossy quality that would be "set on stone" and then require me to manage the pairs of files. Too rigid and too much trouble to me, really. Admittedly, I haven't even bothered to try it. I find the concept to be really neat and I like it, but it doesn't seem to be practical for what I want and need.

Thanks. Anyway, the OP can still forget the "substantial improvement in compression" within the FLAC format I guess.
Yep :)

Anyway, I ran some tests while typing this.
All CUETools 2.1.6, just one album.
libwavpack   @high+   ~122X   1014kbps
flac.exe 1.3.2   @8      ~132X   1024kbps
libFLAC      @8      ~102X   1025kbps
FLACCL      @11      ~139X   1024kbps
cuetools flac   @11      ~ 49X   1022kbps

FLACCL seems to take the cake by a small margin. Is it still subset at 11 though? If it isn't, then I'll stick to regular 1.3.2

  • Porcus
  • [*][*][*][*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #19
Anyway, I ran some tests while typing this.
All CUETools 2.1.6, just one album.
libwavpack   @high+   ~122X   1014kbps
flac.exe 1.3.2   @8      ~132X   1024kbps
libFLAC      @8      ~102X   1025kbps
FLACCL      @11      ~139X   1024kbps
cuetools flac   @11      ~ 49X   1022kbps

FLACCL seems to take the cake by a small margin. Is it still subset at 11 though? If it isn't, then I'll stick to regular 1.3.2

CUETools FLAC, you mean? That's the smallest of the FLAC files. Also, using libFLAC ... why does it matter? Different version?

And, you can try
flac -8 -e -p
and see what happens. (Warning: it happens much slower. But so does CUETools @11). But encoding is done only once. FLAC has the beauty of decoding extremely fast anyway.

Cogito: New "-8" does not imply neither -e nor -p, right?

  • radorn
  • [*][*]
Re: Wondering about non-subset FLAC and non-standard-level options
Reply #20
Anyway, I ran some tests while typing this.
All CUETools 2.1.6, just one album.
libwavpack   @high+   ~122X   1014kbps
flac.exe 1.3.2   @8      ~132X   1024kbps
libFLAC      @8      ~102X   1025kbps
FLACCL      @11      ~139X   1024kbps
cuetools flac   @11      ~ 49X   1022kbps

FLACCL seems to take the cake by a small margin. Is it still subset at 11 though? If it isn't, then I'll stick to regular 1.3.2

CUETools FLAC, you mean? That's the smallest of the FLAC files. Also, using libFLAC ... why does it matter? Different version?

And, you can try
flac -8 -e -p
and see what happens. (Warning: it happens much slower. But so does CUETools @11). But encoding is done only once. FLAC has the beauty of decoding extremely fast anyway.

Cogito: New "-8" does not imply neither -e nor -p, right?

Well, CUETools FLAC does achieve the lowest bitrate, indeed, but that's just a measly 2kbps at the cost of almost 3 times the encoding time than the external v1.3.2 exe. Doesn't seem worth it to me.
FLACCL @11 gives the same bitrate than external 1.3.2 with slighty faster encoding. Not worth it for me IF non-subset (haven't been able to find confirmation on this). If it was MUCH faster and/or compressed MUCH MORE, then it would be worth for me to consider, even if files were non-subset. But to give up subset-ness, I need a wider difference.
All things wheighted, I think I still find the current "official" version the best compromise overall, as far as I'm concerned. It's almost as fast as the fastest one, with very little of a difference, and compresses the same, it's guaranteed to be subset and ¡¡it's "official"!!
Alternatives to it should give me a very clear advantage to make me switch.
Use cases different than mine may find clear advantages in the alternatives, but there doesn't seem to be much in them for me.
I wanted to try flake, but CUETools complained it couldn't find the exe, and I couldn't be bothered.