Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: what flac compression level should I use for 'archival'? (Read 5566 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

what flac compression level should I use for 'archival'?


I am encoding my .wav files to flac for 'archival' purposes.  By 'archival', I mean these will be my 'master' copies, metadata will be added and then I will transcode/re-encode to mp3 or apple alac etc for various players.

What level compression should I use? I am not especially concerned about space or speed of this encoding. I looked at the CUETools page on flac encoders - http://cue.tools/wiki/CUETools_FLAC_encoders_comparison and read that it means different things for the different flac encoders supported by cuetools....

"libFLAC has compression levels 0—8, where 0 is the fastest and 8 provides the best compression ratio. The levels are actually aliases for certain combinations of technical command-line options.

libFlake and FlaCuda are tuned differently, so libFlake -5 might in fact compress better than libFLAC -8. They also support additional compression levels 9–11, however their use is not recommended, because those levels produce "non-subset" files which might not be supported by certain decoders."

so which encoder and which level?

Thanks

Re: what flac compression level should I use for 'archival'?

Reply #1 –
First of all, quality for lossless isn't an issue as long as (1) you stick to signals that the codec/encoder can handle, and (2) you avoid the buggy encoders.
(2) first: due to a bug, you should avoid the variable block sizes settings in CUETools' encoders. Also there is a rare bug in reference FLAC 1.3.4. No reason to panic, but no reason to use that one either.
(If you have non-CDDA signals you might want to avoid ffmpeg too.)

Apart from that, everything up to "-8" plays well in modern devices; there have been some old examples of old Squeezebox devices that would sometimes get in trouble, and then at https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench there are some in-car models that would err out when you get to even more exotic options.

That encoder comparison is a bit outdated - FLAC has improved by 1.4, and FlaCuda is now called FLACCL; it uses your GPU, so if you have a powerful graphics card it might be very fast indeed.
But if you are not concerned about space or speed, then why not use reference FLAC at default level? If you want more compression, "-7" gives smaller files for relatively cheap; if you want faster encode, -4 performs not far from -5. FLAC decoding speeds are already faster than anything. See http://audiograaf.nl/losslesstest/revision%205/Average%20of%20all%20CDDA%20sources.pdf . If you want to make .mp3 for portable use, don't worry about the decoding of the FLAC at any level.

And use verification. Like flac -7V (capital V!) Actually, before you even think of deleting your .wav files, reboot your computer and run a bit-compare round. (foobar2000 does that.)

As for (1):
* FLAC cannot handle floating-point WAVE (those sometimes show up when artists upload from their DAWs without noticing how big they are). If you are using reference FLAC, it will reject them. (This is one reason not to use ffmpeg, it will decimate them and they could be clipping real bad.)
* If you have any 32-bit integer, you should use newest FLAC - and be prepared for issues if your player doesn't support it.
* If you have tagged your WAVE files, --keep-foreign-metadata will import what it knows.  But it will not preserve the non-audio chunks bit for bit: for that, use a "WAVE file compressing format" like WavPack. FLAC is an audio compression format.

Re: what flac compression level should I use for 'archival'?

Reply #2 –

Thanks for the quick reply. I'll clarify / provide more context:

- All my WAV (not WAVE)  files are straight from EAC rips of CDs and are split into tracks using CUETools...So I assume no issues with 32bit integer?
- A few CDs are HDCDs
- the only metadata is in the cue file and obviously just the basics which will from the barebones of the meta data in the flacs...augmented by Picard/MB.
- i currently put mp3s into itunes to get them to my iphone. Would prefer apple lossless though.
-I have a graphics card: an ancient NVIDIA GeForce GTX 750i :-)  (not a PC gamer!)  If I can use it then great. (Have around 800 CDs to encode), but more interested in getting a solid encode that is useful to encode to other formats or play.
- target flac player is likely to be a raspberryPi using LMS/Squeezebox... and yes have one old Logitech squeezebox touch but I was more focused on getting from the WAV files to Flac to get the metadata sorted first... and also to Apple for the phone which is also used for the car.

So what flac encoder FLACCL? or play it safe with the reference flac - libflac? what it is the default compression?
and what for Apple lossless? reflac?


Re: what flac compression level should I use for 'archival'?

Reply #3 –
You can use CUETools to convert. I would have used reference FLAC with it, and made sure to update to 1.4.x (you can make it call flac.exe; if that is in your PATH then it will work, otherwise you go to Settings -> Encoders and scroll to flac.exe and give the location there. If you have foobar2000 with the free encoder pack (which is updated to flac.exe 1.4.1), you have updated encoders at C:\Program Files (x86)\foobar2000\encoders (or C:\Program Files\ ... if you are trying the x64 beta).
Note that you have to be in the same profile ("convert" up in the top-left corner) as the one you want to use, when you tweak settings.
Also you probably want it to verify against AccurateRip when you are encoding, and you might as well write the AccurateRip tags (easy to delete later on).
CUETools can look up MusicBrainz tags for you as well.

Other things as per your questions:
* CDDA resolution ... no issues here! There might be some 24-bit files around if you have decoded the HDCD. (If you haven't ... don't! Especially not for archival. Make sure you tell CUETools not to decode HDCD.)
* CUETools can also encode to ALAC. refalac can also convert FLAC to ALAC directly if you supply it the libflac.dll
* Squeezebox quirks mentioned here: https://hydrogenaud.io/index.php/topic,123569.msg1021323.html#msg1021323 . That version does not make trouble with CDDA resolution. You can test that for yourself I guess; take some metal or something else that would lead to quite high bitrate, encode it with flac --lax -7l32 -r10,12 (which would likely be much more taxing to decode than plain -7 or -8) and see if your Squeezebox objects. If it doesn't, then plain -7 shouldn't be any problem at all.
* Default is -5 yes.
 

Re: what flac compression level should I use for 'archival'?

Reply #4 –
For CUETools and 16bit CDDA a recent flac compile with disabeld asm optimization can be up to 30% faster.
https://hydrogenaud.io/index.php/topic,123234.msg1018713.html#msg1018713
If the CPU is to old i can compile a generic one.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: what flac compression level should I use for 'archival'?

Reply #6 –
Thanks korth for the recent version!
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: what flac compression level should I use for 'archival'?

Reply #7 –
OP:
In case you wonder about third-party compiles, they are safe: Wombat's compile is still "reference FLAC", but built with another compiler. Not a different implementation, so nothing to worry about in terms of "producing buggy .flac" - it won't: it is the "same code, translated to different machine code by a different compiler", and safe to use except that some old CPUs will refuse to run it. (I think the one in fb2k's free encoder pack is compiled by HA user Case.)
Indeed, FLAC 1.3.3 never had an "official Windows binary" released, so all the flac.exe you got from that version, where "such compiles".

@korth : Thanks from me as well - and a question on github internals: is there any way to find things like these current builds? (I.e. when I don't have the URL already)?
Going to https://github.com/gchudov/cuetools.net/find/master and searching for .zip wasn't fruitful.

Re: what flac compression level should I use for 'archival'?

Reply #8 –
They're a bit hidden under the closed pull requests
They're actually test builds that were created before merging into the project.
korth

Re: what flac compression level should I use for 'archival'?

Reply #9 –
For archiving make sure you create recovery records, e.g. using par2.

FLAC is sensitive to bit errors, you wouldn't want silent skips in your music.

Re: what flac compression level should I use for 'archival'?

Reply #10 –
Quote from: birdie on
For archiving make sure you create recovery records, e.g. using par2.

FLAC is sensitive to bit errors, you wouldn't want silent skips in your music.

What’s recommended way of doing that?

Re: what flac compression level should I use for 'archival'?

Reply #11 –
Quote from: Wombat on
For CUETools and 16bit CDDA a recent flac compile with disabeld asm optimization can be up to 30% faster.
https://hydrogenaud.io/index.php/topic,123234.msg1018713.html#msg1018713
If the CPU is to old i can compile a generic one.

An Intel core duo E4300… not old, ancient! :-)

Quote from: korth on
Latest git of CUETools includes libFLAC 1.4.2 CUETools_2.2.2_2023-01-06_git_eedea80_ALACEnc_check_for_StereoMethod.zip
The built-in ALAC encoder ('cuetools' under Lossless > m4a) is based on ffdshow ALAC audio encoder
Thank you!

Re: what flac compression level should I use for 'archival'?

Reply #12 –
Quote from: dpr on
Quote from: birdie on
For archiving make sure you create recovery records, e.g. using par2.

FLAC is sensitive to bit errors, you wouldn't want silent skips in your music.

What’s recommended way of doing that?

Keeping a backup. And every now and then run a verification on your working set.

Parity files don't hurt, but in the cases you destroy data you quite a bit of luck for them to help. They won't guard against a failing drive or file system.
Anyway, here is more about how parchive works: http://www.quickpar.org.uk/VerifyingAndRepairing.htm

Note that you cannot update tags in .par-protected files. But if you use foobar2000 then you can use https://www.foobar2000.org/components/view/foo_external_tags that will keep tags (and thus tag updates) external to the .flac files. That is a good idea as long as you use nothing but foobar2000 for tagging, as it reduces the number of times you write a .flac anew, and it is easy to update your backup with only the tags (and newly added .flac of course).
foo_external_tags will not commit you to staying at it: In case you need to use other taggers (that will tag to the files) you can "give up on" the external tags for the files in question by committing the external tags to those files, tagging them with whatever application and in the end write new external tags. But then the .flac is changed compared to your backup drive and also if you use parity files you need to update them.


As for audio formats' sensitivity to corruption, I tested that: https://hydrogenaud.io/index.php/topic,122094.0.html
Best: TAK at low blocksize. Will mute the offending block and keep the length.
Second-best: WavPack (works as TAK but has longer blocksize).
Good: FLAC. Dropouts are short, but length may not be correct.
In case of disaster, all these could improve by trying both the official decoder and ffmpeg.
Avoid: ALAC and TTA. (Who needs to be told to avoid TTA anyway?)

Re: what flac compression level should I use for 'archival'?

Reply #13 –
Quote from: Porcus on
Good: FLAC. Dropouts are short, but length may not be correct.
Those results are out-of-date. This has been addressed in FLAC 1.4.0. I see I forgot to add that to the changelog though. The change commit is this one: https://github.com/xiph/flac/commit/1793632ee6988deb933ff7551fee92134622b558
Music: sounds arranged such that they construct feelings.

Re: what flac compression level should I use for 'archival'?

Reply #14 –
Quote from: Porcus on
Avoid ALAC

The target is iphone, so… its ALAC or AIFF…

is there a AIFF encoder?

Re: what flac compression level should I use for 'archival'?

Reply #15 –
A case for ALAC is if you have a 512 GB iPhone - then you could very well fit all the 800 albums on it. No, you don't need lossless for mobile use, but all for sudden you have another full backup.
(Not likely on a 256 GB. My classical music CD rips have a FLAC bitrate that would translate to 230 GB for 800 CDs. Add fifty percent for rawk ... except there are more CD singles in that genre, YMMV big time.  Add a percent for ALAC even if you use CUETools' encoder, and another percent for refalac.)

AIFF? That is uncompressed. Yes a version of the AIFF container can accommodate compressed audio as well, as is the case for WAVE, but those details are not relevant. But FLAC can decompress to AIFF if you need it.

Re: what flac compression level should I use for 'archival'?

Reply #16 –
Quote from: dpr on
An Intel core duo E4300… not old, ancient! :-)
A classic Core 2! If you run a x64 Sysem i can try a generic or core2 compile with disabled asm optimizations but i doubt it does help much. flacCL really may be an option here.
The good thing is CUETools shows the speed for easy comparisons.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: what flac compression level should I use for 'archival'?

Reply #17 –
Quote from: Wombat on
Quote from: dpr on
An Intel core duo E4300… not old, ancient! :-)
A classic Core 2! If you run a x64 Sysem i can try a generic or core2 compile with disabled asm optimizations but i doubt it does help much. flacCL really may be an option here.
The good thing is CUETools shows the speed for easy comparisons.
according to the web page, my
“  ancient NVIDIA GeForce GTX 750i ” is too new! flacCLL only supports upto the 500x. I will give it a try and hope the docs are out of date!

Re: what flac compression level should I use for 'archival'?

Reply #18 –
If you were serious about speed not being a big concern, then even newest reference FLAC works with ancient gear. Here is a discussion about, in ktf's own words, "the build of FLAC nobody has been waiting for: FLAC for MS-DOS!"
https://hydrogenaud.io/index.php/topic,123374.0.html
And some discussion about how old a CPU would encode or decode ...

Re: what flac compression level should I use for 'archival'?

Reply #19 –
Quote from: Porcus on
First of all, quality for lossless isn't an issue as long as (1) you stick to signals that the codec/encoder can handle, and (2) you avoid the buggy encoders.
(2) first: due to a bug, you should avoid the variable block sizes settings in CUETools' encoders. Also there is a rare bug in reference FLAC 1.3.4. No reason to panic, but no reason to use that one either.
(If you have non-CDDA signals you might want to avoid ffmpeg too.)

Apart from that, everything up to "-8" plays well in modern devices; there have been some old examples of old Squeezebox devices that would sometimes get in trouble, and then at https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench there are some in-car models that would err out when you get to even more exotic options.

That encoder comparison is a bit outdated - FLAC has improved by 1.4, and FlaCuda is now called FLACCL; it uses your GPU, so if you have a powerful graphics card it might be very fast indeed.
But if you are not concerned about space or speed, then why not use reference FLAC at default level? If you want more compression, "-7" gives smaller files for relatively cheap; if you want faster encode, -4 performs not far from -5. FLAC decoding speeds are already faster than anything. See http://audiograaf.nl/losslesstest/revision%205/Average%20of%20all%20CDDA%20sources.pdf . If you want to make .mp3 for portable use, don't worry about the decoding of the FLAC at any level.

And use verification. Like flac -7V (capital V!) Actually, before you even think of deleting your .wav files, reboot your computer and run a bit-compare round. (foobar2000 does that.)

As for (1):
* FLAC cannot handle floating-point WAVE (those sometimes show up when artists upload from their DAWs without noticing how big they are). If you are using reference FLAC, it will reject them. (This is one reason not to use ffmpeg, it will decimate them and they could be clipping real bad.)
* If you have any 32-bit integer, you should use newest FLAC - and be prepared for issues if your player doesn't support it.
* If you have tagged your WAVE files, --keep-foreign-metadata will import what it knows.  But it will not preserve the non-audio chunks bit for bit: for that, use a "WAVE file compressing format" like WavPack. FLAC is an audio compression format.

Again you post shit about ffmpeg, this is getting really boring.

Really get something else to brag about.

Re: what flac compression level should I use for 'archival'?

Reply #20 –
Quote from: dpr on
“  ancient NVIDIA GeForce GTX 750i ” is too new! flacCLL only supports upto the 500x. I will give it a try and hope the docs are out of date!
Starting CUETools and switch flacCL on to encode some for testing should be faster as even posting another post...
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: what flac compression level should I use for 'archival'?

Reply #21 –
Quote from: mycroft on
Again you post shit about ffmpeg, this is getting really boring.
Wouldn't it be better if you contributed to enlightenment rather than holding these grudges?

* What is the ffmpeg command to read out the actual number b of bits flag from a WAVE file or an AIFF file, and encode a "b-bit signal in c-bit container" FLAC or WavPack file? Is that anywhere as simple as using the reference implementations which do it automatically?
* What is the ffmpeg command to decode a b-bit signal in c-bit container .flac or .wv file, to a b-bit signal in c-bit container WAVE or AIFF? Without the user having to find b and c first?
* For .wv containing float: same question. Detect it and get it right, can you do that? As simple as using wvunpack?
* How do I get ffmpeg to read and decode the high-channel .wv files that ffmpeg produces? (See attachment produced by 5.1.2-essentials_build-www.gyan.dev)

Heck novice users don't even know that ffmpeg picks output format without warning and happily decimates. Best is to avoid ffmpeg for lossless until you know precisely what it is doing, and damn that isn't easy - especially when you (who probably know it better than most) don't care to get the facts straight, instead you typically spew forth something toxic and then leave. Look at https://hydrogenaud.io/index.php/topic,122094.msg1008187.html#msg1008187 - you made claims, you got a file, how to get it better? You didn't answer. It isn't your fault that TTA is a bad codec (indeed ffmpeg does TTA better than the more crippled reference implementation) but it isn't necessary to go denial when you should know better. And in the same thread, let me quote you:
Quote from: mycroft on
Nowhere mentioning ffmpeg version used.
Aha, and so over to https://hydrogenaud.io/index.php/topic,121478.msg1015609.html#msg1015609 where you cannot even be bothered to cite the ffplay version you used even when asked to - which is because you just made up a false claim. Then in the end you discard the entire ffplay application.
Discarding it that way and complaining that others post shit about ffmpeg? That was shit talking indeed. Not saying it wasn't well-deserved - but don't shoot the messenger when the shit hits the fan.

Re: what flac compression level should I use for 'archival'?

Reply #22 –
Also ffmpeg truncates 24 bits to 16 bits by default when output format is wav.

Re: what flac compression level should I use for 'archival'?

Reply #23 –
Doesn't matter whether the input is 24 bits or anything else, actually. Output to uncompressed LPCM in WAVE or WAVE64 or AIFF or CAF or AU/SND, the default is not lossless, but 16 bit integer. (Oh, and if you try a ".pcm" extension, you will get IMA ADPCM, which is lossy ...)

16-bit integer yes - hm, how to handle float input you ask? Clip without warning, am I right?
ffmpeg is an invaluable tool, but you just need to know that when handling "lossless" formats, ffmpeg does not default to lossless operation. Default output is "by coincidence lossless" if you got the right input.

You can force ffmpeg to output 24 bits; those who don't know how, can (1) look it up, and (2) ask themselves "do I want to look this up rather than let the reference FLAC/WavPack/TAK/Monkey's decoder get it right for me automatically?".
For all that I know, it isn't unlikely that somewhere there is a way to probe the input file to find the bit depth and pass that to the output format and thus - in a single line I guess - force the decoded output to have the same bith depth as the input ...

... at least if it is a 16 or 24 bit file and nothing more exotic like a 20-bit signal off a DVD ... That is the "b-bit" part of my question above.

Re: what flac compression level should I use for 'archival'?

Reply #24 –
I recently started working with Blu-ray.

Input: 24-bit DTS-HD MA / Dolby TrueHD
Expected output: 24-bit WAV (PCM)
Default output: 16-bit WAV (PCM)



I agree with Porcus that FFmpeg is an invaluable tool. I use it every single day.
But you gotta admit that some design choices are retarded.
gold plated toslink fan