Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: re-encoding flac files with a new encoder (Read 12816 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: re-encoding flac files with a new encoder

Reply #25 –
Quote from: Porcus on
Still it is interesting to know whether codecs originally developed for the Windows platform does any more sanity-checking of the written file on (an NTFS) drive. @bryant, what about WavPack? And @TBeck , even if Tak/Takc cannot (AFAIK) re-compress .tak files in-place?
For WavPack’s verify mode, the output file (which has a temporary name if the original is being overwritten) is closed, reopened, and reread in its entirety. Only if that passes the md5sum check (for lossless) is it renamed to the correct name (which deletes the original).

I think this is about as safe as you can easily get in a portable way. Yes, in some cases the entire file would be in cache (either in the OS or the drive), but there’s little chance that the file might not actually be being written at all (or be truncated).

On Linux this uses rename() which is reasonably safe (it does the rename and remove atomically and guarantees that if the operation fails, the old version will still be there). I wasn’t able to find anything equivalent on Windows that I could use, so it’s a separate remove and rename, which in theory leaves open the possibility of the original being deleted and the temp file not renamed (although it would still be there, and the name is obvious rather than gobbledygook).

As was mentioned, FLAC simply decodes the stream on its way out without even pretending to reread the file. This has the advantage that it works even when writing to pipes, and I seem to remember documentation somewhere stating that the purpose of this mode was intended more to verify the integrity of the algorithm on the given processor than the file integrity itself.

Re: re-encoding flac files with a new encoder

Reply #26 –
Coming back to this thread after I started looking into this again. Is there any way to re-compress a file in Foobar while retaining the original filename? (And only deleting the original after a verification of course.)

Re: re-encoding flac files with a new encoder

Reply #27 –
Quote from: MrRom92 on
Coming back to this thread after I started looking into this again. Is there any way to re-compress a file in Foobar while retaining the original filename? (And only deleting the original after a verification of course.)
There is no way to do this in the source folder since both files would be named filename.flac . You'd need to do this in a seperate folder. %filename% in the name format will give you the source filename. If you are interested i have a foobar formatting script that will copy the whole relative source directory structure up to a specified root folder.

Re: re-encoding flac files with a new encoder

Reply #28 –
Quote from: Kartoffelbrei on
Quote from: MrRom92 on
Coming back to this thread after I started looking into this again. Is there any way to re-compress a file in Foobar while retaining the original filename? (And only deleting the original after a verification of course.)
There is no way to do this in the source folder since both files would be named filename.flac . You'd need to do this in a seperate folder. %filename% in the name format will give you the source filename. If you are interested i have a foobar formatting script that will copy the whole relative source directory structure up to a specified root folder.

That could totally be a workable solution, I’d love to hear more about it! If the conversion creates new folders somewhere with the same directory structure, and I just batch delete the original FLAC files after conversion, and then merge the new folders with the old, the new files should theoretically end up where they are supposed to.

Re: re-encoding flac files with a new encoder

Reply #29 –
Quote from: MrRom92 on
Quote from: Kartoffelbrei on
Quote from: MrRom92 on
Coming back to this thread after I started looking into this again. Is there any way to re-compress a file in Foobar while retaining the original filename? (And only deleting the original after a verification of course.)
There is no way to do this in the source folder since both files would be named filename.flac . You'd need to do this in a seperate folder. %filename% in the name format will give you the source filename. If you are interested i have a foobar formatting script that will copy the whole relative source directory structure up to a specified root folder.

That could totally be a workable solution, I’d love to hear more about it! If the conversion creates new folders somewhere with the same directory structure, and I just batch delete the original FLAC files after conversion, and then merge the new folders with the old, the new files should theoretically end up where they are supposed to.

$puts(nameroot,Music) $puts(newpath,$directory(%path%,13)\$directory(%path%,12)\$directory(%path%,11)\$directory(%path%,10)\$directory(%path%,9)\$directory(%path%,8)\$directory(%path%,7)\$directory(%path%,6)\$directory(%path%,5)\$directory(%path%,4)\$directory(%path%,3)\$directory(%path%,2)\$directory(%path%,1)) $puts(posroot,$sub($strstr($get(newpath),$get(nameroot)),1)) $right($get(newpath),$sub($len($get(newpath)),$get(posroot)))\%filename%

Yea :| Dont look at it too long it ugly af. i did my best. In case your root folder is NOT "Music" you have to change that to what ever root you want to copy. You can watch how the resulting files will look like with the converter window.

Re: re-encoding flac files with a new encoder

Reply #30 –
Quote from: MrRom92 on
Coming back to this thread after I started looking into this again. Is there any way to re-compress a file in Foobar while retaining the original filename? (And only deleting the original after a verification of course.)
What about the following:
* Keep a source files playlist.
* Naming pattern for the conversion: directory_path(%path%)\%filename% (@Kartoffelbrei : I think you are looking for $directory_path(%path%)?)
This will replace the ":" after the drive letter by "-". Say, you want to convert from a directory tree under L: to some new structure on the same drive: Then selecting "L:" as target folder and the above naming will copy things under L:\Music to L:\L-\Music. Make sure that there is no "L:\L-" already then, as that will be populated.
You need enough space for both source and target. Make sure that under "When done" you not only transfer everything you want tansferred, but also "Show full status report".
Upon done, you can select all the new files in the popup window, and get them into a new playlist.
* If you want to ensure that no "delayed write" or anything is fooling you, cf. ktf's reply 18, then reboot.
* Now you have a source playlist and a target playlist. Verify that they have the same number of elements (that takes a few seconds to do, could save you from having to wait a day for the obvious conclusion), dump them into the same same playlist, sort by path (get a path column if you don't already have that in your display!) - and run a foo_bitcompare. If everything matches, then highlight the source and File Operations -> Delete.
* Then you can move the first directory under "+" up to "L:\" and Windows will ask if you want to merge, and ... voilà - except, that you might get into trouble with say file/folder names that start or end with whitespace. Alternatively you can use foobar2000 and delete the appropriate characters from the $directory_path

This assumes that you never have simultaneously selected a filename.flac and a filename.wv from the same directory (with same filename for both) - that would make a conflict when both target files would be say filename.whateversuffix


Now, if you have only FLAC and/or WavPack files, and you want to recompress FLAC to FLAC and WavPack to WavPack, then you can do something else with foobar2000, assuming source and target is on same drive L:
* Instead of using foobar2000 to convert, link to the L:\L- structure. (Either shift-rightclick File Operations -> Link, or select copy and choose the "Link" radio button). Linking will create an additional name for the same file. Deleting what appears to be a "new file" will only delete the new name-link; the file is lost only when the last filename is deleted.
* Now you have what looks to be a structure copied, but they are not copied - they don't take more space. Yet.
* As flac.exe and wavpack.exe can recompress in-place, you can traverse the "L-"  hierarchy that way. The exe will create new files and delete the "newly generated filename" - taking more space!
* Dump into foobar2000, bitcompare and delete. But before doing that: check for temp files!
* Alternatively, after linking: (1) reboot, (2) make sure you have as many FLAC in "L-" as in source, (3) use flac.exe to recompress source (leaving the "L-" as backup"), (4) check against temp files (actually, during (3) it might be a good idea to close fb2k or not have it monitor the folders, as that can prevent other applications from deleting and renaming files ...) (5) bitcompare, (6) delete the "L-".

Re: re-encoding flac files with a new encoder

Reply #31 –
You asked over at https://hydrogenaud.io/index.php/topic,122750.0.html too, but some more considerations that are not so foobar2000-specific:
* You have a backup, of course? You might run flac -f on the working set and foo_bitcompare against the backup afterwards.
* Before you do any recompression, verify your flacs. Recompressing will "write valid files with corrupted audio", and the information that there was ever an error will be lost from the .flac files. foo_verifier, or drag and drop into audiotester.exe found at http://www.vuplayer.com/other.php
* You might want to wait for a new FLAC version that seems to improve speed ... or compression for given speed.
* In that thread you were asking for maximum FLAC compression. Actually, you can have FLAC running for ages to squeeze out just a little bit more, and you don't want to do that, it isn't worth it. If you want heavier compression, then change codec. If you go to samefilenameasflac.tak, it is easy to run a loop that will copy each album.cue into a album.tak.cue and then you can batch edit all the album.tak.cue. (TAK isn't open source, but it has an open source decoder. If you want free software ... WavPack.)
* CUETools isn't that dependent upon filenames in .cue files being correct. But it does depend upon everything being CDDA.
* You might want to ask yourself if it all is worth the hassle (and the risk) - storage is not that expensive anymore. But if your drive is nearly full ... sure it might be a point.

Re: re-encoding flac files with a new encoder

Reply #32 –
Thanks all for the tips! I will continue experimenting with this a bit though I am starting to reconsider the immediacy with which I plan to carry this out across my entire library.

I tested recompression with a single FLAC file I have that I know is corrupt. Running the foobar verifier threw an error message as expected, but so did attempting to re-compress the file, and nothing was lost in the process.
Since the verification process seems to take nearly as long as just complete recompression, and gives the same error result when appropriate, I am not sure I see the utility in running a verification first.

I did a batch test by doing a set of files added in my library within a certain date range, and the total size of the selection prior to recompression was 238GB.
Recompressing this set of files took somewhere between 1-2 hours (I walked away from the PC and foobar doesn’t give a readout of the total elapsed time once complete, so I don’t know exactly how long it took for certain) and when it was done the new set of files was only 228GB.


I know hard drive space is cheap these days - and I’m not at a particular loss for space either - but 10GB is 10GB. That’s nothing to sneeze at! If nothing else, it feels better to me to not have drive space wasted on “hot air”, and managing this data in the future will itself be more efficient once compressed more efficiently. ie. Drive migrations, backups and other transfers of this data will not take as long now that it is 10GB smaller than it previously was.

Since this was also only a 238GB selection out of my music library, I estimate a full recompression could net me as much as 100GB of reclaimed space. Not insignificant by any means.

With my current hardware I think such a recompression effort could be carried out across the full library within a 24 hour timespan.


At some point in the future I may move everything to a BTRFS based NAS. I’ll probably wait to recompress everything as part of that migration process, otherwise I am in no particular rush but I would like to do this at some point going forward.

Re: re-encoding flac files with a new encoder

Reply #33 –
10 GB of 238 is in the 4 percent ballpark, and that is quite a lot: In ktf's latest lossless test - CDDA at http://www.audiograaf.nl/losslesstest/revision%205/Average%20of%20all%20CDDA%20sources.pdf - it amounts to around the difference between FLAC and TAK. (Part of it could potentially be tags padding. If you have removed a big picture from tags, fb2k will by default not reclaim the "saved" space as that would trigger a full file rewrite.)

If you migrate to NAS, then the transfer speed is often throttling down so much that you can consider even slower FLAC options than -8.
-8p and -8e take about equal time, and it seems to me that the "p" gives more savings on CDDA material, the "e" gives more savings on high resolution material; -8ep is awfully slow.

But expect the next (?) official FLAC release to improve. ktf has teased improvements here.


Quote from: MrRom92 on
I tested recompression with a single FLAC file I have that I know is corrupt. Running the foobar verifier threw an error message as expected, but so did attempting to re-compress the file, and nothing was lost in the process.
I think that depends on the tool. Be cautious.


Quote from: MrRom92 on
Since the verification process seems to take nearly as long as just complete recompression
Ah, the joys of spinning drives :-)

Quote from: MrRom92 on
and gives the same error result when appropriate, I am not sure I see the utility in running a verification first.
If you have a second set (commonly known as "backup"), then what about (1) recompress, (2) bitcompare to backup, (3) if there is anything lost (say information about corruption), copy from backup to working drive?

Re: re-encoding flac files with a new encoder

Reply #34 –
Quote from: Porcus on
10 GB of 238 is in the 4 percent ballpark, and that is quite a lot: In ktf's latest lossless test - CDDA at http://www.audiograaf.nl/losslesstest/revision%205/Average%20of%20all%20CDDA%20sources.pdf - it amounts to around the difference between FLAC and TAK. (Part of it could potentially be tags padding. If you have removed a big picture from tags, fb2k will by default not reclaim the "saved" space as that would trigger a full file rewrite.)

If you migrate to NAS, then the transfer speed is often throttling down so much that you can consider even slower FLAC options than -8.
-8p and -8e take about equal time, and it seems to me that the "p" gives more savings on CDDA material, the "e" gives more savings on high resolution material; -8ep is awfully slow.

But expect the next (?) official FLAC release to improve. ktf has teased improvements here.


Quote from: MrRom92 on
I tested recompression with a single FLAC file I have that I know is corrupt. Running the foobar verifier threw an error message as expected, but so did attempting to re-compress the file, and nothing was lost in the process.
I think that depends on the tool. Be cautious.


Quote from: MrRom92 on
Since the verification process seems to take nearly as long as just complete recompression
Ah, the joys of spinning drives :-)

Quote from: MrRom92 on
and gives the same error result when appropriate, I am not sure I see the utility in running a verification first.
If you have a second set (commonly known as "backup"), then what about (1) recompress, (2) bitcompare to backup, (3) if there is anything lost (say information about corruption), copy from backup to working drive?



My apologies for not being able to address some of these points in a more organized form by quoting it in parts (formatting when posting from mobile is not the easiest!) but I will try my best…

• I did strip any embedded album art tags before running the converter, knowing that recompressing would then remove the leftover padding… so that did account for SOME of that 10GB savings. But I’m not sure just how much of that can be attributed to the total 10GB savings as the image tags definitely weren’t present in every file (typically only attached to stuff from download providers) and they tend to be pretty small anyhow.
Even if I made a crazy assumption that half of that space was reclaimed just from removing image tags… 5GB is still quite a bit of shrinkage!

• in my string of commands, -8 -e -p were all separately present (I’m not sure if the -8ep you mentioned is a more “proper” way of using those 3 commands together or if that was just your way of referring to them being used as I did - my apologies if that’s a stupid question!) but as far as I am aware (again, please correct me if mistaken) using -8 -e -p should provide the highest compression, while still being fully compliant, at the expense of overall processing time/intensity.


• I did notice that in the process of running the standalone verification against the first set of files, the disks were pegged at 100% usage, which is what I attributed the overall slow speed to. However it seemed that running the actual compression - in which verification would be a part of that process as well - disk usage was very low, maybe hovering around 5-15%?
This is interesting to me since the compression involves reading AND writing to the same disk (or disks, as they were from a RAID array) so I figured there would be heavier disk activity there, not less.

Maybe it has something to do with the rate at which data is actually being read from the disk within those processes. I am thinking the compressor simply reads less data in at a time and that keeps things manageable, whereas the verifier tries to read the data all at once causing the disk usage to go haywire. Just a guess though.

I can experiment with the settings... I’m wondering if lowering the simultaneous thread count might put less stress on the disks when it’s reading the files? In theory, less threads might result in overall faster verification if the disks don’t get hit so hard. In which case, I’d be less hesitant to run extra verification checks!

Re: re-encoding flac files with a new encoder

Reply #35 –
-8 -e -p is synonymous to -8ep is synonymous to -p8e is synonymous to -ep8 etc.  Also the "-f" for force overwrite can be merged into there.

It is possible to get smaller files by stacking up with windowing functions (FLAC will then run through them all).
Say, take a test file of a minute music or so, and give the command
flac -8f -A "partial_tukey(1);partial_tukey(2);punchout_tukey(3);welch;partial_tukey(2/0/999e-3);punchout_tukey(3/0/8e-2);partial_tukey(9);punchout_tukey(9)" testfile.flac
... which tests like 30 functions instead of standard -8's six. With either -e or -p (but not both!) it is still faster than -8ep.  But if you throw both -e and -p into this ...

Read the output file sizes and see, it is hardly worth it.

Then you can try instead a git build (warning: should not be considered production ready, have backup!) posted at https://hydrogenaud.io/index.php/topic,122179.msg1013309.html#msg1013309 (post 119).
It is a bit slower than 1.3.4, but you will easily find files where this new -8 outdoes what that monster-long thing spends ... here it is 50 times as much time on.
How? This is (I think!) due to some parameters calculated with double precision, which makes for better accuracy (and smaller residual hence smaller file) but all those calculations take a bit longer time. Now the monster-long -8pef -A line with this build is not that much more time-consuming than with the old build.


You still got your 228 GB to experiment with? ;-)

Re: re-encoding flac files with a new encoder

Reply #36 –
Quote from: Porcus on
-8 -e -p is synonymous to -8ep is synonymous to -p8e is synonymous to -ep8 etc.  Also the "-f" for force overwrite can be merged into there.

It is possible to get smaller files by stacking up with windowing functions (FLAC will then run through them all).
Say, take a test file of a minute music or so, and give the command
flac -8f -A "partial_tukey(1);partial_tukey(2);punchout_tukey(3);welch;partial_tukey(2/0/999e-3);punchout_tukey(3/0/8e-2);partial_tukey(9);punchout_tukey(9)" testfile.flac
... which tests like 30 functions instead of standard -8's six. With either -e or -p (but not both!) it is still faster than -8ep.  But if you throw both -e and -p into this ...

Read the output file sizes and see, it is hardly worth it.

Then you can try instead a git build (warning: should not be considered production ready, have backup!) posted at https://hydrogenaud.io/index.php/topic,122179.msg1013309.html#msg1013309 (post 119).
It is a bit slower than 1.3.4, but you will easily find files where this new -8 outdoes what that monster-long thing spends ... here it is 50 times as much time on.
How? This is (I think!) due to some parameters calculated with double precision, which makes for better accuracy (and smaller residual hence smaller file) but all those calculations take a bit longer time. Now the monster-long -8pef -A line with this build is not that much more time-consuming than with the old build.


You still got your 228 GB to experiment with? ;-)


I see, thank you for the clarification!
Still have those files of course, not to mention PLENTY more where they came from… total size of the FLAC library is about 2.5TB, and growing all the time…

I should mention I also commonly install the new git builds posted here by NetRanger (I like to live on the edge) so the version I just used in this short test is probably closer to an eventual 1.4.0 than it is to 1.3.4 and should include these new compression improvements


On my new PC build (AMD Ryzen 5950x) compressing a file with -8 -e -p takes about as long as it took my old PC to compress a file at a regular old -8. So right off the bat I already use -8 -e -p as a standard compression routine in EAC. It’s already done compressing by the time the CD is ripped, so the time spent is all the same either way.

 I did occasionally use -8 -e -p on on the old PC, but it did take an ungodly amount of time… and I let it run anyway, not fun waiting though! Thankfully those days are behind me, but maybe I’ll try those new commands you posted and see what it’s like to re-live that experience…

Re: re-encoding flac files with a new encoder

Reply #37 –
If you use bleeding-edge git builds for recompressing your working set, and an older flac in foobar2000, then foo_bitcompare will verify that an official build decodes them to what they are supposed to be. (This is "better" than using -V, which will decode using the same executable, and if code is recycled between encoding and decoding, errors might be mirrored. I have never seen that happen though, but it is an easy fix that is kinda free.)


If you are really adamant about size, you can go for a different codec than FLAC. WavPack -hx4m ("m" for MD5) is exciting, but I don't know if it can still beat the new FLAC builds using -8pe -A tonsofthem. (You can specify up to 31 functions, but partial_tukey(n) and punchout_tukey(n) count as n each.) For high resolution material, ktf's most recent test measures quite some difference (CDDA here). But it depends on the kind of music as well.

WavPack is FOSS and has third-party implementations. If you want to sacrifice the FOSS principle, ffmpeg has FOSS decoders for TAK (including multichannel) and Monkey's (stereo only). Beyond that, you are in frog territory.

Re: re-encoding flac files with a new encoder

Reply #38 –
For me it’s FLAC or bust. Extreme compression isn’t the ultimate priority enough to drive me towards another format, just more a side goal of trying to have a well maintained FLAC library in its most optimized form. To me, no more different than trying to make sure tags are correct, folders are well organized, etc.

Re: re-encoding flac files with a new encoder

Reply #39 –
I kinda married FLAC myself, and at a time when storage was more expensive. Slightly unfaithful when WavPack serves special purposes. I like WavPack (and I'm impressed at TAK) - but not enough to migrate.
 

Re: re-encoding flac files with a new encoder

Reply #40 –
Oh, and ...

Quote from: MrRom92 on
I did occasionally use -8 -e -p on on the old PC, but it did take an ungodly amount of time…

I'll raise you ungodly amount of time: https://hydrogenaud.io/index.php/topic,120158.msg1001334.html#msg1001334
13.6 GB of (high resolution) PCM. Just for the hell of it to see FLAC kinda "maxing out", I gave a command where it took ... five days or so, to compress three hours and a half of 96 (mostly) / 24, while I was away from the computer.
I think it was done sequentially on one core, and the CPU isn't exactly new.