Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: New HDD and Mass lossless conversion (Read 6408 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

New HDD and Mass lossless conversion

Hi,

I intend to buy soon a new 3 TB HDD, and I will use it among other things to stock all my music (about 1 TB).
My music is mainly lossless : lots of CDs painfully ripped one by one by myself with EAC, etc..
As of today I have about 34.000 lossless files.
All this lossless music is encoded in FLAC. But FLAC has evolved over time, so I've ended up with lots of different FLAC versions, from 1.1.3 up to 1.3.1.
This of course doesn't bother me (lossless is lossless, all my files work perfectly).

BUT since I'm going to transfer all my music to my new hard drive anyway, I figured it would be a good occasion to convert all my lossless files to the latest FLAC version (with compression 8 and verify option of course O:) ).

What would be the easiest/100% secure way to do this ?
1. Chosse a root "starting" folder (e.g. "X:\MUSIC\LOSSLESS")
2. Choose a "destination" folder (e.g. "Y:\MUSIC\LOSSLESS")
3. Batch convert everything from X: to Y: while preserving EVERYTHING :
- All my tags created by foobar (including custom tags, tags with multiple values, etc.).
- All my folder hierarchy (X:\MUSIC\LOSSLESS\Artist 1\Album 1\Title1.flac becomes Y:\MUSIC\LOSSLESS\Artist 1\Album 1\Title1.flac, and so on)

I must emphasize on the fact that I absolutely need to preserve EVERYTHING : for example in the past I've already used converters that "lost" information in the process (e.g. for multivalue tag fields, only the last value was preserved !).

Thanks in advance. :)

Re: New HDD and Mass lossless conversion

Reply #1 –
If you care about FLAC's Application ID stuff that will most likely disappear when you mass convert files. Apart from that regular tags are just text and don't include any hidden surprises like custom fields familiar from Apple-created MP4 files.

You can retain your directory structure by using "$replace(%path%,X:\MUSIC\LOSSLESS,)" as the name format. To also copy all tags and extra files in the directories make sure the appropriate settings are enabled in Converter's Other-section. Note that you can't use "*.*" for "Copy other files to the destination folder" as that would also copy the source FLAC files. You need to specify non-audio format files there, something like *.txt;*.log;*.cue;*.pdf;*.jpg;*.png;*.gif.

Using verify while encoding doesn't ensure that the conversion worked correctly. It only ensures the encoder encoded the data it saw correctly. Bad memory for example can cause bit flipping in the data and encode/verify pass can't detect that. Running binary verification between the new and old files will spot all differences. Potentially false positives too with bad hardware.

There's a component for the file comparison between two sets, but this latest version with offset checking seems to run much slower than it should. Until Peter fixes things it might be best to use an older version, if one can be found.

Re: New HDD and Mass lossless conversion

Reply #2 –
Quote from: Case on
Note that you can't use "*.*" for "Copy other files to the destination folder" as that would also copy the source FLAC files. You need to specify non-audio format files there, something like *.txt;*.log;*.cue;*.pdf;*.jpg;*.png;*.gif.
For separated flac tracks i'd use just *. Source flac files are copied only if they were added to playlist by cue. Also it worth to mention that subfloldes (for example with pictures) cannot be copied using ''Copy...'' function in converter.

Re: New HDD and Mass lossless conversion

Reply #3 –
Thanks for your replies.

Case, from your reply, I take it that you think that foobar's built-in converter would be the best option for the job ?
The "$replace(%path%,X:\MUSIC\LOSSLESS,)" is a great idea, I can definitely use that.
I'll make sure all appropriate settings are checked in the "Other" section, for transferring tags and RG info.

In "Copy other files to destination folder", I'm worried that I can't use "*.*". I have many other types of files and I don't remember them all : .jpg, .log, .txt, .accurip, and so on. So here's a converter feature request : why not add a converter parameter called "-" which would allow to NOT copy a particular type ? (example: "*.* -*.flac". You get the idea).
I guess I can run verify while encoding AND binary verification afterwards. Thanks for the tip.
About the binary comparator, I guess that in my case no offset will be introduced by the conversion process. So wouldn't there be a way to just disable the offset checking ? That would be a nice option, to be suggested to Peter eventually. So that makes two feature requests, lol.

THanks again !

Re: New HDD and Mass lossless conversion

Reply #4 –
Actually I was mistaken about the "Copy other files" feature. Rollin was right to correct me. Its name does say "other" files and you can use * or *.* without worry. It won't copy the files that were ran through the converter.

Re: New HDD and Mass lossless conversion

Reply #5 –
Awesome. I'll give it a try. I'll just have to hope that I don't have any non-music subfolders : I don't remember and I have thousands of folders... Would there be a smart way to check that beforehand ? Any ideas ?

Re: New HDD and Mass lossless conversion

Reply #6 –
In Ed Sheeran's words, "Just a small bump". Any ideas guys ? ;)

Re: New HDD and Mass lossless conversion

Reply #7 –
I don't see any reason at all to convert the older FLAC files. As you put it yourself, "lossless is lossless, all my files work perfectly" and "I absolutely need to preserve EVERYTHING."

Copy the files as they are to the new drive and keep the old drive as a backup. That's as close to "100% secure" as you're going to get.

Re: New HDD and Mass lossless conversion

Reply #8 –
I think the easiest way would be to copy full directory structure to the new drive and just copy re-encoded files over the existing ones. That requires the use of a temp dir to re-encode the files you want though.

But if you prefer you could try this simple batch file I made. It reports names of subdirs that don't contain FLAC files but only one level beyond the FLAC-containing directory.

Save it with appropriate name, I used findextras.cmd, and call it from your music dir or just give it the dir as parameter. You can redirect the output to a file for easier reading.
For example: findextras "X:\MUSIC\LOSSLESS" > extras,txt

Code: [Select]
@echo off
setlocal enableextensions disabledelayedexpansion
:start
if ."%~1"==."" (
  call :walkdir .
) else (
  call :walkdir "%~1"
)
shift
if not .%1==. goto start
endlocal
goto :EOF

:walkdir
if ."%~1"==."" exit /B
pushd "%~1"
if errorlevel 1 exit /B
if not exist *.flac if exist ..\*.flac echo "%cd%"
for /D %%d in (*) do call :walkdir "%%d"
popd
exit /B

Edit: forgot to mention. The Binary Comparator component was updated recently and it's now faster than ever before. Safe to use for verifying your libraries no matter how large they are.

Re: New HDD and Mass lossless conversion

Reply #9 –
You can use a copy utility where you exclude the FLAC files. Use one which can handle too long file names.
- If you have a corrupted file, conversion may convert into a compliant file with corrupted content (and you will no longer know you had corruption). So, this is the occasion to verify that your audio files are good.  You may save this to the end though (remember to verify the SOURCE then). I would use audiotester.exe from http://www.vuplayer.com/other.php , because fb2k have earlier on failed to identify audio files that are too corrupted. (They have not shown up, they have not been tested, they have not been reported as corrupted. Would not be converted either, then.) The "file count" step should catch those as well, but I still use audiotester.
- Copy with directory structure, excluding *.flac (and *.fla if you ever used that)
- Then I would use the following intermediate step to detect possible old mistakes: dump everything that is by now in the target, to a media player which ignores suffixes. That way you can find FLAC files which by mistake you have given a different suffix.
- Then convert, preserving structure.
- Do a file count! If the number is different, then watch out. (If you let Windows index, you may have a few pesky Thumbs.db ...)
- Then, foo_bitcompare is your friend.

Notice that converting causes fb2k to write file segment by file segment, causing disk fragmentation, does it not?



Quote from: Jailhouse on
I don't see any reason at all to convert the older FLAC files.

When I let fb2k select a set of files in my library, then I don't have to read through a long line of FLAC versions to see whether they are all FLAC or some are other formats - since I only have one FLAC version. And I will continue using 1.3.1 for years. 
 

Re: New HDD and Mass lossless conversion

Reply #10 –
Some news report : my batch conversion is finally taking place as we speak !

As told previously, I'm using foobar's built-in converter to copy/convert my 37.000 lossless files to FLAC 1.3.2 in a new HDD, including copying all non-FLAC files to the destination folders (*.* trick). Preserving of course all tags, ReplayGain, etc.. I ran two previous tests on a small number of files that seemed to work, so Fingers Crossed now.

Once the conversion is over, I will use the latest version of the binary comparator to see if everything's been copied as expected.

Now I just have a question about it (for @Case since he suggested it ;) ) : the binary comparator manual says you need to create a single playlist containing all the "old" files, immediately followed by all the "new" files. Then the binary comparator magically "cuts" that playlist in two perfect halves and compares all the "old" files to the "new" files. Does this mean that I need to create a single playlist with 37.000 + 37.000 = 74.000 files ?  :o  Is that really the way to go ? I was expecting something a bit more intuitive, like a dialog asking us to enter a "left directory" and a "right directory" in order to compare their contents, lol

Re: New HDD and Mass lossless conversion

Reply #11 –
Quote from: wcs13 on
Does this mean that I need to create a single playlist with 37.000 + 37.000 = 74.000 files ?  :o  Is that really the way to go ?
That's right. Note that the files need to be sorted the same way, so don't just drop random albums in the playlist one at a time.
You could for example drag your source file directory to a temporary playlist, do the conversion, then drag the new target dir over the playlist tab. That gives you a set that is ready to be compared.

Re: New HDD and Mass lossless conversion

Reply #12 –
In addition to my reply #9: I would not only check the file count, but also the total time, or total number of samples.
And then I would sort by path - given that you have preserved file structure, that will give you the 37k files on (say) E:\ in path order followed by the 37k files on (say) F:\ in path order.

Re: New HDD and Mass lossless conversion

Reply #13 –
OK, now the problems begin...

After 25 hours of 100% CPU work, the converter status shows this :
" 46 out of 37171 tracks converted with major problems. "
Let it be said that there were exactly 34.290 audio tracks to convert. The rest (2881) are other files (.txt, .log, etc.). Funny that foobar calls everything "tracks".

I have created a Word document with the full status, then I have performed some searches.
The "Track converted successfully" string appears 34.244 times.
34.290 - 34.244 = 46, so indeed there should be 46 tracks without that mention.
However, I can only find 2 times the "major problems" string. How about the 44 other tracks ?

How can I find all the 46 tracks ? And why doesn't the status simply make them stand out in some way ?
Thanks.

EDIT : I found them. 2 tracks with the "major problems" string and 44 tracks with the "conversion failed" string. I still don't understand why the status doesn't make them stand out. This made me lose half an hour.

Re: New HDD and Mass lossless conversion

Reply #14 –
Turns out most of the remaining problems have been caused by tracks with too long paths.
Some of them can't even by renamed by Windows ! (F2 doesn't work on them).
Yet they exist.
foobar can even access/play them, so I'm trying to convert/rename them at the moment.

Re: New HDD and Mass lossless conversion

Reply #15 –
Everything's fine at the moment.
FYI the whole conversion process took about 24 hours for all the 37.000 files (24.290 audio files + the remaining non-audio files).
I have handled each and everyone of the 46 faulty tracks. It's all good now.
I have also verified that every single non-audio file was copied, with the very useful WinMerge (highly recommended).
Now I have as expected, 34.290 starting FLAC files, and 34.290 converted FLAC 1.3.2 files.
I have added all the files to a big 68.580 files playlist, and sorted everything by path.
Finally, I have launched foo_bitcompare. It's running at the moment.

So far I have only one small complaint/feature request for @Peter : foo_bitcompare uses only one of my CPU cores, while foobar's Converter used all of them. On 68.580 files, it makes a huge difference ! This operation may take 12 hours or so. If my CPU had been used to 100%, it would have taken 3 hours...

Also, there is no "Pause" button (which would have been useful to give my CPU a break eventually), no "Processed files" counter, and no "Elapsed time / Estimated time" indicator, like in the Converter. I have only the progress bar. It's a bit too "basic"for my taste. Maybe foo_bitcompare wasn't initially thought for comparing so many tracks at once, but hey, that happens from time to time. ;)

Re: New HDD and Mass lossless conversion

Reply #17 –
OK, here's the Bit Compare result : "4 out of 34290 track pairs could not be compared."

How can I find those 4 track pairs in that huge text log ?... What kind of text string am I supposed to search for ?
And what is wrong with them ? Why couldn't they be compared ? @Peter , are you listening ?
foo_bitcompare's log needs some work in order to be really useful, IMHO.

Please help. Thank you.

Re: New HDD and Mass lossless conversion

Reply #18 –
Hopefully the log gets more helptul and gathers all problems into easy to find location. The dll lists at least these strings for non-matching situations:
Access denied
Audio data format mismatch
Compared tracks contain apparent errors reported by the decoder; use File Integrity Verifier component to get more information.
Comparing failed
Decoder produced no data
Decoder returned invalid data
Differences found
Invalid audio data format
Invalid data returned
Neither track could be opened for decoding
Non-zero offset detected in     Zero offset detected.   Zero offset detected in
Object is not seekable
Operation aborted, results are incomplete.
Premature EOF in referenced audio file at
Problematic files:
Sample format change in mid stream
Seek offset out of range
There were problems reported
Unsupported format or corrupted file
compared track pairs contain apparent errors reported by the decoder; use File Integrity Verifier component to get more information.
could not be opened for decoding
length mismatch
non-identical track pairs.
track pairs could not be compared.

Re: New HDD and Mass lossless conversion

Reply #20 –
There should be a "Comparing failed" in each decode error notice.
Microsoft Windows: We can't script here, this is bat country.

Re: New HDD and Mass lossless conversion

Reply #21 –
Thanks guys for your help !

Thanks to Peter's valuable info I could immediately find the 4 culprits. They were 4 "residual files" (.fla.flac - don't ask me where they came from, lol) that I had found and erased on each side before the comparison. I have double-checked, everything is fine now.

In other words, all my lossless files have been PERFECTLY copied and converted to FLAC 1.3.2 in my new hard drive ! Yay !

I can now backup everything and finally get some rest.

Again, thanks to everybody for your great help. I really appreciate. :)

Re: New HDD and Mass lossless conversion

Reply #22 –
If you used the most recent foo_bitcompare, it should report if there were errors in any of the files.
(Why bother? Converting would take a corrupted FLAC file, decode what can be decoded, write to a compliant FLAC with the same corrupted audio - and the warning would be lost.)

Quote from: wcs13 on
They were 4 "residual files" (.fla.flac - don't ask me where they came from, lol)

You checked that they match something else in your library?

You intend to keep the old one as backup?

Re: New HDD and Mass lossless conversion

Reply #23 –
Hey Porcus :) Yes, I used the most recent foo_bitcompare.
The 4 errors were indeed reported, I just had to search for the "Comparing failed" string within the huge log.

The errors had nothing to do with corrupted files. The "FLA.flac" files were most probably residues of a former operation which I don't even remember. Most importantly, they were duplicates (e.g. a given Track1.flac would have a Track1.FLA.flac duplicate), so simply removing them was the solution.

It fhere had been any corrupted files, I would have re-ripped the corresponding CDs, etc..

Rather than keeping the old library as a backup, I have backuped the new library (900 GB, including 34K FLAC 1.3.2 files and 13K lossy files) to a backup HDD.

I can report that all my tracks are safe now. I am happy. :)

Re: New HDD and Mass lossless conversion

Reply #24 –
Quote from: wcs13 on
Rather than keeping the old library as a backup, I have backuped the new library (900 GB, including 34K FLAC 1.3.2 files and 13K lossy files) to a backup HDD.

If you are going to delete the old - and, assuming you have not already done so - I would have used a utility that deletes duplicates by name and/or by content, to see if there is anything left uncopied / unconverted.
If you do that a clever way, you could possibly detect if there were any files too corrupted for fb2k to even recognize as FLAC.  I took a lot of such measures when I got rid of 1.1.3, 1.1.4, 1.2.0 and 1.3.0. Paranoid? Well yeah ...