Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Does it make sense to reencode to the new FLAC version (Read 3199 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Does it make sense to reencode to the new FLAC version

Hi,

does it make sense to reencode FLAC files with a new libFLAC version? I encoded my files some years ago with FLAC 1.3.2 (2017) and now I see that 1.4.2 is out (2022). A few questions arise:

  • Is it "more compliant" with the CELLAR specs? More long-term archive ready?
  • Is the file format any differnt or just the way the lib works internally?
  • Does it have a better lossless compression?
  • Will it have the same md5 checksum as it is the same audio data after all?
  • How to do it practically? Just run a recent flac executable on the existing .flac files?

Re: Does it make sense to reencode to the new FLAC version

Reply #1
Likely it doesn't make much sense - except possibly on high resolution material. I did re-encode a hard drive from 1.3.1 because (1) it was down to 2 GB free space, and (2) out of curiosity on the impact. Edit: heck, I a fb2k portable has the bitrates in the backup and hence in the old set. Filling them in at the very end.

With reservations stated at the end:

1. No ... surely not if you have CD material encoded. My impression is the CELLAR spec largely clarifies corner cases in a way that the reference FLAC has already done things.  Some of the "new" requirements amount to forbidding things that wouldn't even make sense to do.
2. Same.
3. Very little on CDDA, but of course it depends on whether you use a heavier compression setting than you did on the originals. Potentially more on high sample rates (96k and above) if you use a heavy compression setting.
4. Yes.
5. Yes, with -f and your options. But you can also have e.g. foobar2000 do it for you.
If you only want to re-encode high resolution, then there is a trick and a problem with it. Can come back to you if ...


Then the reservations:
If you have files with corruption, you should as long as you can refrain from touching them, as re-encoding will hide the corruption rather than repair it. There have been cases where applications have "abused" libFLAC to writing streams that cannot be contained in the "format" (like, a so-called 16-bit stream which when decoded, will exceed the bounds of 16 bits), and FLAC version 1.4 has become better at detecting those and will reject them.
As for 4: yes, if the original file had a correct MD5. At least one music store supplies FLAC files without MD5 at all.


Impact, imprecisely measured:
CDDA: file sizes actually decreased by around 0.1 percent, but some of it was likely reclaimed padding.
high resolution decreased a lot - although in the end I re-encoded them with a higher setting than -8. They are now at bitrate 2701 and they were 2524
And that figure excludes some of the more stupid files in a test folder. ( The one at https://hydrogenaud.io/index.php/topic,123025.msg1025239.html#msg1025239 replies 283 to 287 is not in here.)

Re: Does it make sense to reencode to the new FLAC version

Reply #2
IMO nothing to gain. I wouldn't do it even if it was flac from 2003.
There are chances of user / hardware errors going unnoticed.
For very large collections doing it a lot will cause increased wear on
your disk(s), & possibly the CPU if its old due to more heat if using high levels like -8
wavpack -b2.9cs.5
lame --preset cd -f

Re: Does it make sense to reencode to the new FLAC version

Reply #3
The FLAC format is very stable. If you encode CD audio with one of the presets of current FLAC 1.4.2, you can decode it with FLAC 0.9 from 22 years ago. For hi-res material (24-bit audio) that is encoded with FLAC 1.4.2, you can decode it with FLAC 1.2.0 or newer, from 15 years ago.

Anyway, there has been some improvement between versions, but not a lot. We're talking about 0.1% to 0.5%, depending on the kind of music you're compressing. For 24-bit material this can be a little more. This means that is you have a collection of 1000 CDs encoded in FLAC 1.3.2, recompressing with FLAC 1.4.2 might save you space for 1 to 5 extra CDs, so not a lot really.

As has been said, re-encoding brings a certain risk, so if you decide to go through with this, you should have a back-up ready. Having a back-up is always a good idea by the way.
Music: sounds arranged such that they construct feelings.

Re: Does it make sense to reencode to the new FLAC version

Reply #4
For 24-bit material this can be a little more.
If I remember correctly, the big difference was not higher bit depth, but higher sampling rates? (The figures I quote above are for files with largely both ...)
 
have a back-up ready. Having a back-up is always a good idea by the way.
This!

Re: Does it make sense to reencode to the new FLAC version

Reply #5
Thanks to all of you for the replies! I am not interested in per-mil level compression gains. The stability of the format is really a great feature. And, yes, I have a backup (or several).

My conclusion is that I keep my files as they are and use the recent version for new files. I can probably reevaluate that question in about 10 years from now, again. Thank you!

Re: Does it make sense to reencode to the new FLAC version

Reply #6
high resolution decreased a lot - although in the end I re-encoded them with a higher setting than -8. They are now at bitrate 2701 and they were 2524
Opposite! They are now smaller.
And now I've spent more than an hour editing the nonsense I was writing, so I have to correct it this way.

Re: Does it make sense to reencode to the new FLAC version

Reply #7
It’s not totally pointless. I occasionally do this using F2K but I haven’t attempted to do this to my library en-masse. Though I have considered it!
I like the idea of my files being compressed optimally but the tradeoff with how long it would take to go through everything, just doesn’t make it worth it for me as of this point in time. Might be a future project, far off into the future when SSDs finally match spinning rust for price and capacity.

F2K has been pretty good with detecting any corrupt files and in my experience it will NOT re-encode them with a new “valid” MD5 - that would be pretty bad if it were the case…


One positive aspect of doing this is that the process of re-encoding will by nature check all the older files for corruption, and in some cases this is actually faster than just verifying the integrity of said files without re-encoding (i/o bottleneck when reading from disk? Possibly, but I’m not 100% sure - reading and writing from disk as opposed to just reading seems to go faster for whatever reason.)


There’s also the fact that older files may not be compressed optimally - not that I’m concerned about files compressed with older versions of FLAC so much as older files compressed with sub-optimal settings. Your FLAC from 2003 might have been compressed at -5 instead of -8, or something along those lines.

Re: Does it make sense to reencode to the new FLAC version

Reply #8
Who knows if FB2k will tell you of corruption in 100% of cases.
Recently I encountered some errors with conversions leaving some .tmp files
Forward to 2 weeks ago, I lost the entire 3TB seagate.  Can't say it was the culprit but
I suspect it. Uncorrectable sectors was playing up on and off (no bad) + random bsod once or twice
a month. I also suspect bsod was HDD even though the dumpfile point to my webcam.

Without previous versioned backup(s), theres so many ways to fall that it leaves me astonished.
With newer hardware the risk is smaller , but watch it esp on HDD's that have racked up Hours=several
years.
wavpack -b2.9cs.5
lame --preset cd -f

Re: Does it make sense to reencode to the new FLAC version

Reply #9

Even going from flac 2003 -5 to flac 1.42 -8 will give me extra 1 % on cd audio ?  That is 3MB per album. 600MB for
a 200 cd collection. Even if it amounted to 1000MB its not worth it .
wavpack -b2.9cs.5
lame --preset cd -f

Re: Does it make sense to reencode to the new FLAC version

Reply #10
Yeah for 200 albums the gains might be pretty minimal. On the other hand, would also be very quick to burn through. For thousands of albums that might add up, but even then the vast majority of my library comes from my own CD rips and I already know they were always done at -8 at minimum (nowadays, -8ep as my processor can handle it and CD ripping is more of a time bottleneck than the compression itself)

FLAC integrity is pretty straightforward IMO. If there is any corruption, it can’t decode back to a WAV with the original stored MD5. If it fails this, there is a very clear error presented at the end of the whole conversion process.

Reasons like this why I can’t imagine anyone would ever advocate for the use of ALAC or WAV. It’s like flying with a blindfold. I recently experienced a nightmare situation in what is probably the biggest studio on the planet. Album masters on their server… one of them corrupt BIG time. Not like a sample here or there devolved into clicks or something. Half of the file completely devolved into nothing but white noise. Nothing but a plain jane 24/48 WAV (with some metadata/cue marks from the ME’s DAW)

Of course had to go through the process of having the files resent to them, all while being billed for studio time. All could’ve been avoided with FLAC or routine MD5 checks.

Re: Does it make sense to reencode to the new FLAC version

Reply #11
I've been updating my FLAC files over the past couple weeks.  Not to save space, but to update the album artwork.  Even with more aggressive compression settings (-m -b 4096 -l 12 -r 7 -p -A subdivide_tukey(13/10e-1);welch;hann;flattop), the files will be slightly larger with the new artwork.  I've been simply using -8 until recently.  Without updating the artwork, I'd be lucky to gain 1GB out of 213GB of space using the more aggressive settings.

I starting ripping my CDs to FLAC back in 2011/2012, when FLAC 1.2.1 was around.  I wrote a script the uses cdparanoia to rip the CD and encode the WAV files to FLAC, storing the md5sum of the WAV files in the vorbis comment (along with other metadata).  Most of the files have seen multiple storage devices and multiple file systems (none being FAT32). 

I'm gone through almost 2000 files in my collection so far.  Only one file had a MD5 mismatch to what's stored in the vorbis comment, but no complaint about the md5sum stored in the STREAMINFO block.  (For those who don't know, the md5sum in the STREAMINFO block is for signed little-endian PCM).  I can only assume my mismatch was due to a scripting error, thinking md5sum ran on the WAV file before it was complete.

Good to know that over a decade later, my files are still good.  I've had plenty of mp3s in the past get randomly corrupted with no obvious reasons.

On a side note, cdparanoia is a Xiph project like FLAC, any plans for updates, or is it considered complete? @ktf

Re: Does it make sense to reencode to the new FLAC version

Reply #12
On a side note, cdparanoia is a Xiph project like FLAC, any plans for updates, or is it considered complete? @ktf
I have no clue about other Xiph projects to be honest. I occasionally have contact with one other member of Xiph, it is not like 'they' have regular meetings or something. Or maybe they have, but I'm not included :D

If you'd like to ask, try their IRC: https://xiph.org/contact/
Music: sounds arranged such that they construct feelings.

Re: Does it make sense to reencode to the new FLAC version

Reply #13
I've been updating my FLAC files over the past couple weeks.  Not to save space, but to update the album artwork.  Even with more aggressive compression settings (-m -b 4096 -l 12 -r 7 -p -A subdivide_tukey(13/10e-1);welch;hann;flattop), the files will be slightly larger with the new artwork.

I've always considered embedding artwork in music files redundant and wasteful. Makes no sense to embed the exact same data in a dozen (or however many) files. AFAICR, every media player software and firmware I've used can recognise images in the same directory as the music files named "cover.{ext}", "folder.{ext}" or "album.{ext}" and show those as the album artwork when playing the files. This is a much better and less wasteful solution. This way, you can have larger artwork while wasting less space on storing it.

Re: Does it make sense to reencode to the new FLAC version

Reply #14
To be extra anal about space efficiency there's probably crunch programs to losslessly make jpegs smaller, they definitely exist for png. In a few years if players start supporting jpeg-xl then embedded jpegs could be losslessly transcoded to jpeg-xl for ~20% size reduction.

Re: Does it make sense to reencode to the new FLAC version

Reply #15
On my PC, I could have something like folder.jpg, and symlink half a dozen file names to make media libraries happy.  I haven't really used a media library in some time though.  I usually navigate the folders and open individual files or playlist in mpv or mocp.

For something like my car stereo which needs FAT formatted storage, symlinks won't work.  I'd rather just embed the image in each file rather than have the same image under different names hoping the media player picks one up.  This is also useful for folders with mixed albums/compilations.

As far as the image type, I wish I could find the artwork in high quality PNG format.  Just about any source I've used only offers jpg and sometimes webp.  Converting the jpegs to PNG only increases file size with no quality gained.  optipng is a good program for compressing PNGs as it will go though 1080 combinations to find the best compression. 

Re: Does it make sense to reencode to the new FLAC version

Reply #16
Jpeg-XL does lossless images like png much better than png too, although currently it's a moot point as jpeg-xl adoption is in its infancy so things like foobar are unlikely to have support for years if ever. There's three modes; lossless, lossless-jpeg-transcode, lossy, all of which are good so jpeg-xl is a one size fits all solution for most input.

webp can be lossless and in fact it's pretty good at lossless compared to png, if you have large webp files they may be lossless.

Re: Does it make sense to reencode to the new FLAC version

Reply #17
On my PC, I could have something like folder.jpg, and symlink half a dozen file names to make media libraries happy.

...I don't understand why you need symlinks? Do you store the same album on several locations?

I would just have a copy of folder.jpg as well in that case - that is still less wasteful than embedding it in each individual file.

As for compilations of songs from various albums, I agree that in that specific case embedding the artwork is the only viable solution, unless you're willing to settle for the same cover image for the entire compilation.

Re: Does it make sense to reencode to the new FLAC version

Reply #18
As far as the image type, I wish I could find the artwork in high quality PNG format.  Just about any source I've used only offers jpg and sometimes webp.

Have you tried this? https://bendodson.com/projects/itunes-artwork-finder/
Not perfect but still higher quality than most other sources.
gold plated toslink fan

Re: Does it make sense to reencode to the new FLAC version

Reply #19
...I don't understand why you need symlinks? Do you store the same album on several locations?

Not albums.  Using symlinks so cover.jpg, thumb.jpg, something.jpg, etc.jpg all link to folder.jpg

Have you tried this? https://bendodson.com/projects/itunes-artwork-finder/
Not perfect but still higher quality than most other sources.

I've been using itunes as one of my sources.  I've been editing the urls to get the original images.  Bandcamp has the best quality images most it the time, but their selection is limited and mostly for recent albums.  AlbumArtworkExchange and fanart.tv are also good.  Amazon is hit or miss and YouTube music occasionally has a quality image when all else fails.

Re: Does it make sense to reencode to the new FLAC version

Reply #20
Not albums.  Using symlinks so cover.jpg, thumb.jpg, something.jpg, etc.jpg all link to folder.jpg

That's so unnecessarily hypercautious and overengineered that it's ridiculous. A single cover.jpg will do. It has yet to fail anywhere for me.

Re: Does it make sense to reencode to the new FLAC version

Reply #21
I just tried it on a classical album encoded in 2011. very little improvement if at all. <1%

Re: Does it make sense to reencode to the new FLAC version

Reply #22
That's so unnecessarily hypercautious and overengineered that it's ridiculous. A single cover.jpg will do. It has yet to fail anywhere for me.

My car stereo for example has no issues reading embedded artwork, but doesn't display folder.jpg, cover.jpg, artwork.jpg, folder-name.jpg or any other typical filename I've tried.  Unless storage space is really limited, I don't think the extra kilobytes per track will be an issue.

Re: Does it make sense to reencode to the new FLAC version

Reply #23
I just tried it on a classical album encoded in 2011. very little improvement if at all. <1%
"<1%" doesn't rule out "0.1%" ... and if you got anywhere near a full percent on CDDA, you must have used a different preset?
Going from -5 to -8p or so, that is when I think you can speak about a full percent - at least, http://audiograaf.nl/losslesstest/revision%205/Average%20of%20all%20CDDA%20sources.pdf suggests that "eyeballpark".

Yes it is very little in absolute terms and hard drive cost - except when the drive is about full of course.
That is a perspective often lost in the performance measurement threads, which might be dominated by those of us who cherish the 0.1 percent improvement for the sports more than for the utility value. At https://hydrogenaud.io/index.php/topic,123025.msg1016625.html#msg1016625 I got nearly 0.1 percent improvement (over -8) by spending nearly tenfold the time.