Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: dBpoweramp + ALAC + CRC32 + foobar2000 verification (Read 5725 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

dBpoweramp + ALAC + CRC32 + foobar2000 verification

Hi.

I've been reading about the robustness of FLAC against ALAC, because of the MD5 checksum it stores in itself. Problem is that nothing beats the experience I'm having with Apple Music and iTunes, and I've been converting CDs to ALAC, using dBpoweramp.

Based on that, is it safe to assume that if one once has registered all CRC32 from the tracks in the dBpoweramp logs (or whatever ripper for that matter), if an ALAC file gets corrupt... is it still possible to verify that the CRCs won't mach upon a simple foobar2000 integrity verification?

Thanks a lot!

-- krafty

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #1 –
Quote from: krafty on
is it safe to assume that if one once has registered all CRC32 from the tracks in the dBpoweramp logs (or whatever ripper for that matter), if an ALAC file gets corrupt... is it still possible to verify that the CRCs won't mach upon a simple foobar2000 integrity verification?
fb2k doesn't read those logs?

Possible: Keep the cuesheets and try to verify with CUETools. If it matches AccurateRip or the CUETools database, it matches a known rip. Unlikely to happen for a corrupted file.

Those that do not enter the AR nor CUETools databases ... oh well. it is still kinda possible to automate it. Set CUETools to write AccurateRip tags upon verify, and it will then write an ACCURATERIPCRC tag. dBpoweramp will (check your settings!) write an ACCURATERIPRESULT tag. Then you can use fb2k to check up those which do not match.
Example tags from a file:
dBpoweramp wrote <ACCURATERIPRESULT>: AccurateRip: Not in database   Secure: Yes   [AABD20DD]
CUETools post verification: <ACCURATERIPCRC>: aabd20dd
You see that the latter is a substring of a lower-case conversion of the former.

Of course, then the CUETools verification procedure will write to file every time you verify. Maybe that is not what you want - not even annually; but you can of course do the following:
* Run this immediately after conversion to ALAC. Will write the tags.
* Upon verification, single out the "not yet verified" by tags (see also the other tags CUETools writes). Those which have not yet been verified to match an AR, use fb2k to move them temporarily to another music folder, keeping structure; like, just use E:\%path% or something.
* Those unmoved: they verified last time. Let's say you don't want tag overwrites on everything; then switch off AR tag writing and dump the album folders into CUETools. After it is done, copy the report window and check that each line confirms they still verify. (Copy the output window with Ctrl-a, paste into a text editor like Notepad++, remove all lines with the string "AR: rip accurate" etc.)
* Those which were moved: Switch on AR tag writing. Dump the folders into CUETools. Run verify. Check by tags. Next year, those which match will be among those "unmoved".



... phew, had I suspected I would write this much I wouldn't have started.

But by now it struck me: https://www.dbpoweramp.com/Help/perfecttunes/accuraterip.htm . I have used that as well ...

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #2 –
https://foobar.hyv.fi/?view=foo_audiomd5
That will do. Writes the crc of the audio (is codec dependent) to a tag, so you can at any time check the file against the stored crc.

Obviously you can simply use the standard audio verification which will decode the audio first and verify it outputs no error. But there is no CRC to compare against.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #4 –
Thanks Porcus. Thanks regor. This Audio MD5 is a lifesaver!

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #5 –
Quote from: regor on
https://foobar.hyv.fi/?view=foo_audiomd5
That will do. Writes the crc of the audio (is codec dependent) to a tag, so you can at any time check the file against the stored crc.
Nice. Just to reiterate, this checksums the encoded/compressed audio data, so it's not equivalent to FLAC's built-in MD5 checksum.
But it's still useful and I'm going to use it for old lossy files that don't have a lossless alternative.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #6 –
Quote from: Brand on
Nice. Just to reiterate, this checksums the encoded/compressed audio data, so it's not equivalent to FLAC's built-in MD5 checksum.
But it's still useful and I'm going to use it for old lossy files that don't have a lossless alternative.

Indeed. The MD5 from the FLAC files differs -- it parses the file differently. After that I don't know whether I will use or not.
But dbPoweramp registers each file CRC to its log file.
If there is any kind of corruption, one can write a script to verify these CRCs against a TXT file, right?
Because this is basically it was all I would need.
The convenience of having FLAC verification built-in is that one could just use foobar2000 to verify integrity and leave it running.
Is this thought correct (specially the CRC registered by DBPA and one that's being got by a script against the audio data)?

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #7 –
First, I have not used this, but since you are a dBpoweramp user:
https://www.dbpoweramp.com/codec-central-utility.htm , the Calculate Audio CRC; does that help you?
dBpoweramp can write a CRC to a tag, but will it verify it smoothly?

Quote from: krafty on
Indeed. The MD5 from the FLAC files differs -- it parses the file differently. After that I don't know whether I will use or not.
As far as I understand - I tested only on a .flac - the foo_audiomd5 on a .wav file (but not an .aiff, endianness matters!) matches the built-in MD5 from the FLAC.

Which means, you can take FLAC files, make some AUDIOMD5FROMFLAC tag from its stored audio checksum, copy that over to the MP4 files containing your ALAC, and every once upon a blue moon you can check integrity by
* transcoding your ALAC to FLAC and see if the built-in MD5 matches the tag
or
* transcoding your ALAC to WAV and then install this component and run it, and check whether the AUDIOMD5 it creates, matches your AUDIOMD5FROMFLAC tag from back then.

But again, if you have cuesheets or ACCURATERIPDISCID tags from dBpoweramp, you can also let CUETools run a verification.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #8 –
Hi Porcus,

I have been trying for the second time this dBpoweramp [Calculate Audio CRC] Utility Codec but I can't find -- for the life of me -- where are the options to it or how to activate it after installation. I installed the utility codec running the executable, but I can't find this on the DSP list at all: should it be listed in this DSP list? How can one access the options CRC32/MD5? If it does write a MD5 tag alone to the file and it is the same MD5 sum as FLAC that would be a killer. But, I don't know. I am doing everything right here, or is there any secret to this?

Thanks again.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #10 –
I think that paragraph must have been written with another version of DBPA or as you said for MacOS alone. As of DBPA R17.4 on Windows 10, I have on my dashboard:

Audio Codecs [ List/Options]

Configuration [ Advanced Settings ].

The CRC Codec appears as installed in Advanced Settings, but just like it informs FLAC is installed, in the same way, with no options. I can't seem to find the Enable Codec Utility.

This is too foggy. Some threads are responded with a line, and by reading other threads, it seems to me the whole thing works only with existing files - I quite didn't get it.

When I read about this codec I thought it would do the following sequence:

1) You'd install the codec.
2) You'd select it from the DSP tab, just as you would with ReplayGain, for example.
3) You'd start ripping your CD and when the track would be ripped, this codec would put a MD5 tag into the file, with a MD5 hash in it.
4) You'd have an option to select whether it was going to be a CRC32 tag or a MD5 tag.
4) And that would be the end of it.

That was my thought about this utility codec. Now, what I am reading from sparks here and there is that it doesn't actually do what it leads us to think it does.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #11 –
You don't use it upon ripping but as if you were trying to "convert". Instead of (hm, maybe also "in addition to"?) converting to a different format, you apply the utility.

IIRC:
* You need to go to the Codec Central to download it. It is an exe file that will install and activate it.
* Enter dBpoweramp Control Center. Check the settings for the codecs. You should find it there.
* Try to "convert to" this.

There is an option to write the CRC to tag upon ripping as well ... wherever that setting is hidden.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #12 –
I don't understand why you make this so complicated. Just create AudioMD5 tag when you encode your files and you can quickly verify them later from the same tag at any time.

And if you ever switch to another format, like FLAC, you can use foobar's bit compare tool to verify your transcoded files are identical to the source files.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #13 –
Can adding a custom tag like that to an MP4 file cause any problems? (with iDevices or in general)

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #14 –
Quote from: Case on
I don't understand why you make this so complicated. Just create AudioMD5 tag when you encode your files and you can quickly verify them later from the same tag at any time.

And if you ever switch to another format, like FLAC, you can use foobar's bit compare tool to verify your transcoded files are identical to the source files.


Sorry, but you're right. I was making a hell of an issue. I think I got AudioMD5 component working now -- the first time around was a bit weird, I didn't see the Update Tag button.

Porcus, I got CRC thing working now. It works like this:

Select a file in Explorer, Convert To. There will be only this option. FLAC will be brought as the output codec. But IT IS RIGHT HERE where you change to the CRC calculator (and here will have the CRC32 and MD5 options), INSTEAD of an audio codec such as FLAC or other lossless. I think I now got it right but AudioMD5 will do.

The CRC Codec Utility reads identical MD5 from ALAC, the same that is stored in another same audio file in FLAC. Same hash.
The AudioMD5 is not creating the same hash. I wonder why.

Thank you all for the input.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #15 –
After doing a test, it seems that the Audio MD5 will only render the same MD5 hash (as a FLAC file, for instance), if it is only WAV. If it's a normal ALAC or FLAC file, it won't render the same MD5... even if you wipe out all tags. So, one could think FLAC actually gets this hash from the WAV/PCM data only and store it into the FLAC tag, but not the FLAC file itself; whereas the AudioMD5 constructs the file said to be analyzing audio only -- but in fact -- something else is happening under the hood.

Late minute update:

According to this source, the component might not be working as expected, because it may be still parsing more than just audio.

I ran the following command in Power Shell:

Code: [Select]
ffmpeg.exe -i losslessfile.m4a -vn -f md5 -

...and it indeed returned the same MD5 hash as a FLAC would do to itself.

In foobar2000 it will only return the right MD5 hash if it is a WAV file. If it's a FLAC file or ALAC, it will return a different MD5.

I tried to reach the developer, user Janne Hyvärinen here, because his web form to e-mail only marks things as spam, so I could not make this go through. It would be interesting if he could look into this.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #16 –
The hash is different because AudioMD5 generates it from the encoded binary data. The checksum in lossless codecs like FLAC and WavPack is for the uncompressed PCM data.

The reason it works this way is that AudioMD5 component was made for lossy codecs. Generating checksum from the decoded output makes no sense in this scenario because different decoders can produce different outputs and it's waste of CPU cycles to even decode the data. I kept the same behavior for lossless codecs as from my point of view all the codecs worth using already had this stuff built-in natively. Also checking just the compressed data is faster than decoding and checking the decoded output.

The checksum of the component is meant for checking that audio bits of your files haven't gotten corrupted. It's not meant for checking that two different encodes are the same. Bit Compare works best for that - it can even tell what is different.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #17 –
Ok, thanks, i see that you are the developer I tried to reach. Thanks for the explanation, I was totally getting the purpose wrong.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #18 –
I uploaded a new version with advanced config option to use decoded PCM as the hash source for lossless formats. With that option enabled FFmpeg isn't needed or used for lossless files.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #19 –
Quote from: Case on
Also checking just the compressed data is faster than decoding and checking the decoded output.
That's a good point actually, I might end up using it for FLACs too, for the speed.
I ran a few tests and it's definitely faster on an SSD and on a "warmed up" HDD (second run, cached). On a first run HDD, however, it's a bit slower than the regular integrity check for FLACs (which decodes the audio data)...I wonder why.

And I think there's a small a bug: the "Track naming in results dialog" setting doesn't do anything, it always ends up showing the default (artist - title).

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #20 –
Quote from: Case on
I uploaded a new version with advanced config option to use decoded PCM as the hash source for lossless formats. With that option enabled FFmpeg isn't needed or used for lossless files.

Case, you're the man. :-)

Would it be correct to say that this tool, with foorbar2000, updates ALAC "weak" state, that it has no MD5 checksumming, to a new level? Because it is working just as a FLAC verification would work. Of course, you need the tool to do this -- it is not a built-in thing in the codec, but after this lossless option being added, this is really something to consider, even a Wiki Update. You do have error robustness at some level. In fact, following the thread pointed in the Wiki, @ktf talks about being able to play with errors, more like a feature than a weakness, whereas other lossless codecs would not play at all. And with this tool, coming from that point of view from @ktf, not just you are able to decode through errors but also a way to check the audio integrity. 

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #21 –
Quote from: Brand on
On a first run HDD, however, it's a bit slower than the regular integrity check for FLACs (which decodes the audio data)...I wonder why.
I believe foobar uses better input buffering than ffmpeg. Peter has tuned the file reader buffering behavior for optimal multi-threaded speed during all these years. When disk reading is the bottleneck a fast codec like FLAC won't be a problem to decode faster than the checksumming needs. Another possible problem might be anti-virus. If a virus scanner intercepts the ffmpeg process creation it can certainly cause slowdowns.

Quote from: Brand on
And I think there's a small a bug: the "Track naming in results dialog" setting doesn't do anything, it always ends up showing the default (artist - title).
Thanks! It seems I forgot to actually use the configured value in the dialog. Fixed version uploaded.

Quote from: krafty on
Would it be correct to say that this tool, with foorbar2000, updates ALAC "weak" state, that it has no MD5 checksumming, to a new level?
This is close to built-in checksumming but not equal. In FLAC the checksum gets used by all tools that deal with the file and they are able to warn about corruption. With this third party solution only this special tool is able to utilize the checksum and detect errors. This is of course enough if all you use is foobar2000.

Re: dBpoweramp + ALAC + CRC32 + foobar2000 verification

Reply #22 –
Thanks, Case!
Not to derail this thread too much (frankly, this component could have its own thread), but I was thinking that it would be nice to have this functionality for video files too. It does work for video files now, but only the audio data is checksummed.
So maybe it could produce a videomd5 checksum too (and/or a combined audiovideomd5). This would of course require a small change in the output display, since there would be more values for one file. But it could come handy, if someone converts just the audio or just the video stream and leaves the other intact.

(BTW, I tested a mkv video with two separate opus tracks and it seems to checksum both of them — individual tracks produce a different MD5. However, when combining the two back into a video file, the combined MD5 changes...)

Anyway, I understand if this is too much for a f2k component. Maybe a standalone tool would make more sense. Regardless, I think there are some movie hoarders out there who would find it useful.