CRC32 mismatches in CUETools log when gaps prepended to next track

Topic: CRC32 mismatches in CUETools log when gaps prepended to next track (Read 2171 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

CRC32 mismatches in CUETools log when gaps prepended to next track

2018-09-29 04:46:10

I am using Exact Audio Copy v1.3 to rip albums with gaps prepended to the following track ("Appended to next track" setting). Any write operation I try to perform with EAC fails with my DVD writer, so I am using CUETools v2.1.6 (though I see the same behavior described below with v2.1.7 as well) to modify the .wav files so as to negate my writer's write offset before burning them to disc with ImgBurn. This works fine; ripping a burned disc yields .wav files identical to those ripped from the original pressed disc. My rips are also confirmed by AccurateRip with high confidence, so there are no concerns there.

What I did find curious is that when I run an operation in CUETools it performs a verification of the CRC32 values from the EAC log, and the values CUETools calculates don't always match what EAC calculated. Sometimes it is just a few tracks that don't match, sometimes it is most tracks. Here's an example from one album:

CUETools log EAC-style info (gaps prepended):

Code: [Select]

Track Peak [ CRC32  ] [W/O NULL] [  LOG   ]
 --  100.0 [788A15CF] [BA31310E]           
 01  100.0 [B0A55C46] [61CE6A93]   CRC32   
 02   99.7 [F12B0F4C] [3B860B61]   CRC32   
 03  100.0 [E43BBB08] [4F8E8196] [20976537]
 04  100.0 [7C89ECCC] [EE3D7F4C] [A18395BE]
 05   99.3 [BBDA580F] [33E3E970]   CRC32   
 06   99.2 [F7FCFC87] [89BF5A5F]   CRC32   
 07   99.3 [D68D9665] [7A3BFD7F]   CRC32   
 08  100.0 [9B34AC01] [0AE1A531]   CRC32   
 09  100.0 [E577BA2B] [EC3CA8B6] [24EA7D15]
 10  100.0 [E20851E6] [5BFA3319] [66A78D20]
 11  100.0 [892F3A30] [8448FD67]   CRC32   
 12   98.8 [B18E55A0] [FF7FBC51]   CRC32   
 13   99.0 [2D53534F] [5693CE1E]   CRC32

I was going to post asking, for the sake of curiosity, what would cause this result and what it means, but after much searching and not finding anything I finally got the idea to see what happens if I rip with gaps appended. Here's the same album ripped that way:

CUETools log EAC-style info (gaps appended):

Code: [Select]

Track Peak [ CRC32  ] [W/O NULL] [  LOG   ]
 --  100.0 [788A15CF] [BA31310E]           
 01  100.0 [B0A55C46] [61CE6A93]   CRC32   
 02   99.7 [F12B0F4C] [3B860B61]   CRC32   
 03  100.0 [E43BBB08] [4F8E8196]   CRC32   
 04  100.0 [7C89ECCC] [EE3D7F4C]   CRC32   
 05   99.3 [BBDA580F] [33E3E970]   CRC32   
 06   99.2 [F7FCFC87] [89BF5A5F]   CRC32   
 07   99.3 [D68D9665] [7A3BFD7F]   CRC32   
 08  100.0 [9B34AC01] [0AE1A531]   CRC32   
 09  100.0 [E577BA2B] [EC3CA8B6]   CRC32   
 10  100.0 [E20851E6] [5BFA3319]   CRC32   
 11  100.0 [892F3A30] [8448FD67]   CRC32   
 12   98.8 [B18E55A0] [FF7FBC51]   CRC32   
 13   99.0 [2D53534F] [5693CE1E]   CRC32

As you can see, the CRC32 and W/O NULL columns are identical to the other log, but now there are no mismatches in the LOG column. If needed I can post the CUE sheet, but, in short, there are only two gaps in the entire album: between tracks 3/4 and between 9/10, which happens to correspond to the four tracks with CRC32 mismatches in the gaps-prepended log.

To try to figure out how CUETools is arriving at those CRC32 values, I wrote my own code to calculate the CRC32 of various permutations of all or some of file 3 concatenated with all or some of file 4, and compared those to the values calculated by EAC and CUETools:

Track	Extraction gaps	Calculation method	CRC32
3	Appended to previous	EAC	E43BBB08
4	Appended to previous	EAC	7C89ECCC
3	Prepended to next	EAC	20976537
4	Prepended to next	EAC	A18395BE
3	Appended to previous	CUETools log	E43BBB08
4	Appended to previous	CUETools log	7C89ECCC
3	Prepended to next	CUETools log	E43BBB08
4	Prepended to next	CUETools log	7C89ECCC
3	Appended to previous	File3:Main	20976537
4	Appended to previous	File3:PostGap + File4:Main	A18395BE
3	Prepended to next	File3:All	20976537
4	Prepended to next	File4:All	A18395BE
3	Appended to previous	File3:All	E43BBB08
4	Appended to previous	File4:All	7C89ECCC
3	Prepended to next	File3:All + File4:PreGap	E43BBB08
4	Prepended to next	File4:Main	7C89ECCC

Hopefully that's not information overload, but the important takeaways are:

CUETools calculates the same CRC32 values regardless of the gap handling used when extracting the tracks (rows 5-8), and...
When CUETools calculates the CRC32 values for tracks with gaps prepended (rows 7-8) it yields the same results as when tracks with gaps prepended are reassembled with gaps appended (last two rows)

Therefore, it appears that, when processing EAC logs, CUETools always calculates CRC32 values with gaps appended, regardless of the gap handling used by EAC during ripping. I also took a cursory look through the source code and do not see anywhere that the Gap handling line from the beginning of the EAC log is being read in so CUETools can use the same handling. Of course, it is to be expected that a stored hash will not match a verification of that hash if the input data is not arranged in the same way.

I am inclined to suggest that this is a bug, but I suppose that depends on what the purpose is of the CRC32 calculations performed by CUETools. If, as I assume, they are meant to verify that the .wav files have remained unchanged since they were ripped, then CUETools would need to ensure it uses the same gap handling as EAC did. If, however, it's intentional that CUETools always calculates CRC32 values with gaps appended regardless of the original gap handling (perhaps to allow for consistent, gap handling-agnostic comparisons of multiple rips of the same disc? just spitballing ideas...), then I think it would be helpful if the wiki were updated to make that technical detail clear.

Re: CRC32 mismatches in CUETools log when gaps prepended to next track

Reply #1 – 2018-09-29 15:03:49

Try verifying the gaps prepended rip without the CUE sheet (track files only). You shouldn't get any database matches but the CRC32 values should match your extraction log.
CUETools is using the CUE information to restructure the audio data to match the TOC from the original CD so it can verify the rip against the databases. Everything in the CUETools accurip log is calculated from that TOC including the CRC32 values.
I can put a note on the wiki.

Re: CRC32 mismatches in CUETools log when gaps prepended to next track

Reply #2 – 2018-09-29 21:43:51

Ah, yes, I was verifying against the CUE sheet since I had it, and didn't think to try without it. So, if I understand, without the CUE sheet CUETools will have no choice but to hash entire .wav files as-is because it has no way of knowing where the gaps are (and, possibly, which tracks are even adjacent).

Here's the log for the same album ripped with gaps prepended but without the CUE sheet:

Code: [Select]

[CUETools log; Date: 9/29/2018 1:49:26 PM; Version: 2.1.6]
[CTDB TOCID: drsojIvwTTKXE_eNfziw_H7NZu8-] disk not present in database.
[AccurateRip ID: 001eca53-01359172-ab0e940d] disk not present in database.

Track Peak [ CRC32  ] [W/O NULL] [  LOG   ]
 --  100.0 [5CD1BDBE] [52CC154E]           
 01  100.0 [B0A55C46] [61CE6A93]   CRC32   
 02   99.2 [F7FCFC87] [89BF5A5F] [F12B0F4C]: CRC32 for track 7
 03  100.0 [9B34AC01] [0AE1A531] [20976537]: CRC32 for track 13
 04  100.0 [A18395BE] [B69E715A]   CRC32   
 05   99.0 [2D53534F] [5693CE1E] [BBDA580F]: CRC32 for track 8
 06  100.0 [24EA7D15] [AE57F005] [F7FCFC87]: CRC32 for track 2
 07   99.7 [F12B0F4C] [3B860B61] [D68D9665]: CRC32 for track 10
 08   99.3 [BBDA580F] [33E3E970] [9B34AC01]: CRC32 for track 3
 09  100.0 [892F3A30] [8448FD67] [24EA7D15]: CRC32 for track 6
 10   99.3 [D68D9665] [7A3BFD7F] [66A78D20]: CRC32 for track 11
 11  100.0 [66A78D20] [432D878E] [892F3A30]: CRC32 for track 9
 12   98.8 [B18E55A0] [FF7FBC51]   CRC32   
 13  100.0 [20976537] [C9B3171F] [2D53534F]: CRC32 for track 5

As it pertains to my issue I see that CUETools is calculating the same CRC32 values as EAC did for a gaps-prepended rip. I'm a little confused by this output, though. Every value in the CRC32 column does also appear in the LOG column, just not necessarily in the same row. I was expecting to see CRC32 all the way down the LOG column, just like the CUE sheet-based logs. Do I have that right? Or this is how it matches the files up to the log, not by reordering the rows but by the for track X reference at the end?

I see that the for track X text represents what that track is according to the EAC log, but I'm curious where the values in the Track column come from? Is that just based on the lexical ordering of the file names in the directory? (I should note my .wav file names are not prefixed with the track number, so, for me, lexical ordering ≠ track ordering) I notice that the first 1st, 4th, and 12th files lexically happen to be the 1st, 4th, and 12th tracks on the album, and those are the only three tracks where the log shows a CRC32 match (in the same row).

Re: CRC32 mismatches in CUETools log when gaps prepended to next track

Reply #3 – 2018-09-30 00:43:08

The for track X reference is telling you the CRC32 value was found at a different position in the EAC extraction log.

Yes without the CUE, the WAV track files are sorted by file name. If your files aren't named so they can be sorted by track number, you could use a simple .m3u playlist file to order the track files correctly.

Styx - The Grand Illusion.m3u

Code: [Select]

The Grand Illusion.wav
Fooling Yourself (The Angry Young Man).wav
Superstars.wav
Come Sail Away.wav
Miss America.wav
Man In The Wilderness.wav
Castle Walls.wav
The Grand Finale.wav

Code: [Select]

Track Peak [ CRC32  ] [W/O NULL] [  LOG   ]
 --  100.0 [A8F0139A] [03688182]           
 01  100.0 [494B7503] [5C045495]   CRC32   
 02  100.0 [B771D6D2] [04DD3973]   CRC32   
 03  100.0 [8C82B2B6] [06DD5471]   CRC32   
 04  100.0 [1F8E9B48] [05409DD4]   CRC32   
 05   99.5 [C9710FD1] [A150F355]   CRC32   
 06  100.0 [228F2AF9] [3005ABC5]   CRC32   
 07  100.0 [BBA22C22] [AFEF9C52]   CRC32   
 08  100.0 [9CDD59E5] [B0FDF6B6]   CRC32

Note: CUETools sorts lossless compressed track files (FLAC, ALAC, APE) by metadata (if exists) then the fallback is by file name.

Re: CRC32 mismatches in CUETools log when gaps prepended to next track

Reply #4 – 2018-09-30 02:31:07

I see. That works. I was thinking the tracks would be treated as unordered and unnumbered in the absence of any CUE/playlist/metadata information and, therefore, subject to the rows being reordered.

Now that I take a second look at your comment...

Quote from: korth

CUETools is using the CUE information to restructure the audio data to match the TOC from the original CD so it can verify the rip against the databases. Everything in the CUETools accurip log is calculated from that TOC including the CRC32 values.

...I think I now understand that to mean that CUETools has to normalize all rips to gaps-appended for submission/storage/comparison with the database, and the EAC log verification, therefore, is also restricted to gaps-appended? If I have that right that makes sense.

I think I can conclude that the CRC32 mismatches I was originally seeing are not mismatches after all, and, anyways, those values would only be worth comparing for files ripped, say, two years prior, not two minutes prior. It was puzzling, though, when I first saw the logs because I couldn't figure how both CTDB and AccurateRip could deem my rip accurate and yet still show CRC32 mismatches. Until I dug into the .wav files myself I was starting to wonder if CUETools was even calculating the CRC32 values from the local files or possibly pulling them from an online database.

Seeing as how the wiki is...a wiki, is that something to which I might contribute to clarify those details? Or is that something you prefer to manage yourself?

Re: CRC32 mismatches in CUETools log when gaps prepended to next track

Reply #5 – 2018-09-30 03:25:33

Quote from: BACON. on 2018-09-30 02:31:07

...I think I now understand that to mean that CUETools has to normalize all rips to gaps-appended for submission/storage/comparison with the database, and the EAC log verification, therefore, is also restricted to gaps-appended? If I have that right that makes sense.

Close enough.

Quote

Seeing as how the wiki is...a wiki, is that something to which I might contribute to clarify those details? Or is that something you prefer to manage yourself?

Feel free to contribute.

Re: CRC32 mismatches in CUETools log when gaps prepended to next track

Reply #6 – 2018-09-30 06:50:12

Great. I'll take a crack at that.

Thanks for your help and insights.

Notice