Skip to main content

Topic: CUETools versions 1.9.5 through 2.1.5 (current) (Read 1311937 times) previous topic - next topic

0 Members and 3 Guests are viewing this topic.
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1350
Just to clarify my position: i think AR does it's job just fine. The exact properties of it's CRC are almost irrelevant, it doesn't need a very strong or very fast CRC to function. My objections to V2 are mostly theoretical. I even could probably implement in some form cross-pressing verification, it's just very annoying that i will have to waste time on it when it could have been much easier with a normal CRC, and the verification could be much faster. I have no enmity towards Spoon or AccurateRip.
  • Last Edit: 18 May, 2011, 08:15:06 AM by Gregory S. Chudov
CUETools 2.1.4

CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1351
And just to demonstrate the point about collisions in ARv2.

ARv2 takes the 32-bit sample value, multiplies it by the sample number, resulting in 64-bit number, and then adds the higher 32 bits to lower bits.

Now consider for example the 15th sample in a track, and it's possible values.
Code: [Select]
ARv2(15,0x11111111) = 0xFFFFFFFF
ARv2(15,0x22222222) = hi32(15 * 0x22222222) + lo32(15 * 0x22222222) = hi32(0x1FFFFFFFE) + lo32(0x1FFFFFFFE) = 0x1 + 0xFFFFFFFE = 0xFFFFFFFF
ARv2(15,0x33333333) = hi32(0x2FFFFFFFD) + lo32(0x2FFFFFFFD) = 0xFFFFFFFF
...
ARv2(15,0xFFFFFFFF) = hi32(0xEFFFFFFF1) + lo32(0xEFFFFFFF1) = 0xFFFFFFFF

ARv2(15,0x00000001) = hi32(0x00000000F) + lo32(0x00000000F) = 0x0000000F
ARv2(15,0x11111112) = hi32(0x10000000E) + lo32(0x10000000E) = 0x0000000F
...
ARv2(15,0xEEEEEEEF) = hi32(0xE00000001) + lo32(0xE00000001) = 0x0000000F

That is, at least 15 sample values in this location produce the same CRC. At least 14 sample values produce another CRC. I'm too lazy to explore further, but this might be equivalent to loosing 3-4 bits of of this sample.

UPD: Not so lazy:) It gets worse for sample 255...
Code: [Select]
ARv2(255,0x01010101) = hi32(0x0FFFFFFFF) + lo32(0x0FFFFFFFF) = 0xFFFFFFFF
ARv2(255,0x02020202) = hi32(0x1FFFFFFFE) + lo32(0x1FFFFFFFE) = 0xFFFFFFFF
...
ARv2(255,0x0F0F0F0F) = hi32(0xEFFFFFFF1) + lo32(0xEFFFFFFF1) = 0xFFFFFFFF
ARv2(255,0x10101010) = hi32(0xFFFFFFFF0) + lo32(0xFFFFFFFF0) = 0xFFFFFFFF
ARv2(255,0x11111111) = hi32(0x10FFFFFFEF) + lo32(0x10FFFFFFEF) = 0xFFFFFFFF
...
ARv2(255,0x1F1F1F1F) = hi32(0x1EFFFFFFE1) + lo32(0x1EFFFFFFE1) = 0xFFFFFFFF
...
ARv2(255,0xFFFFFFFF) = hi32(0xFEFFFFFF01) + lo32(0xFEFFFFFF01) = 0xFFFFFFFF
That's 255 samples having the same CRC... Basically, the same problem as with ARv1.
  • Last Edit: 18 May, 2011, 06:21:39 PM by Gregory S. Chudov
CUETools 2.1.4

  • greynol
  • [*][*][*][*][*]
  • Global Moderator
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1352
So you are saying I have an obligation to do this?

No, I believe I was quite clear in what I said.  Anyhow, seeing that neither of us has changed our stance since the last time we butted heads, I think I'll just go with the "if you don't have anything nice to say, then don't say anything at all" approach.
13 February 2016: The world was blessed with the passing of a truly vile and wretched person.

Your eyes cannot hear.

CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1353
Hello there,
a rather simple question for a complicated case:

is it possible to 'verify' against AR/CTDB a Playstation-style cuesheet referencing TWO pre-audio data tracks ? i.e. the first playable audio CD track starts at #3.

All my attempts at it have failed so far, since only the first referenced data track is detected and thus the generated AR record has no match.
Out of despair I'm starting to experiment with alternative cuesheet designs, file based vs. track based, compliant or not, messing with pregaps and stuff...

Overview of the CD TOC, first few lines:
Code: [Select]
Track |   Start  |  Length  | Start sector | End sector 
    ---------------------------------------------------------
        1  |  0:00.00 |  7:06.02 |         0    |    31951    <= that's data
        2  |  7:06.02 | 21:19.06 |     31952    |   127882   <= still data
        3  | 28:25.08 |  2:12.33 |    127883    |   137815   <= here we go with audio!



(This 2 pre-audio track thing is such a glorious idea...)
  • Last Edit: 20 May, 2011, 05:40:30 AM by HotShotFR

  • drfr
  • [*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1354
OK, so now you are getting the same errors I reported earlier on the thread.
I dont know the reason for those errors, they are probably related to .NET 4.0.
Try this code instead, it works for me:

Code: [Select]
if (processor.PreGapLength != 0)
return processor.WriteReport();;
foreach (uint gap in new uint[3] {32, 33, 37})
{
processor.PreGapLength = gap;
processor.ArVerify.ContactAccurateRip(AccurateRipVerify.CalculateAccurateRipId(processor.TOC));
}
return processor.WriteReport();



Does this really work for you?
When testing on a rip with known 37 pregap and present in AR database, first it shows greyed AR icon for a while, then for just a fraction of second
I can see it has found match but then stops with batch log result: AR: disk not present in database.
So it seems to be close to doing what it should but something is missing.
Any idea?
  • Last Edit: 22 May, 2011, 03:43:20 AM by drfr

  • korth
  • [*][*][*][*][*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1355
So it seems to be close to doing what it should but something is missing.
Any idea?


See [a href='index.php?act=findpost&pid=707559']this message[/a]

Perhaps the missing parts?
korth

  • korth
  • [*][*][*][*][*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1356
I saw some discussion about this somewhere else. How can all the tracks return "(0/0) No match but offset" ?

Code: [Select]
[CUETools log; Date: 4/29/2011 1:59:20 PM; Version: 2.1.1]
[CTDB TOCID: gFCqM0DjKCCUX0sr_Hee.snk.mE-] found.
       [ CTDBID ] Status
       [1998766c] (4/4) Accurately ripped
[AccurateRip ID: 001c6798-011d7713-ac0d620d] found.
Track  [ CRC    ] Status
 01    [68f1f565] (0/0) No match but offset
 02    [3dc5d93d] (0/0) No match but offset
 03    [559a9dcc] (0/0) No match but offset
 04    [a4a4ba22] (0/0) No match but offset
 05    [83f4349b] (0/0) No match but offset
 06    [b5a142ff] (0/0) No match but offset
 07    [74e00357] (0/0) No match but offset
 08    [54efd8f2] (0/0) No match but offset
 09    [eb152ab6] (0/0) No match but offset
 10    [7372a99c] (0/0) No match but offset
 11    [8c120f5b] (0/0) No match but offset
 12    [aee4e5a8] (0/0) No match but offset
 13    [65b0e952] (0/0) No match but offset

Track Peak [ CRC32  ] [W/O NULL]
 --  99.8 [C01A9145] [49C4FCD1]         
 01  99.8 [CBD678A6] [A9EEAD84]         
 02  99.8 [3545F77C] [17F572CA]         
 03  93.3 [45584278] [E5EC2DFD]         
 04  98.5 [36AB01C5] [477BC49D]         
 05  97.7 [5D4E7C2B] [B783D705]         
 06  99.2 [CBDD28B0] [6ADC932A]         
 07  98.8 [67DA1B68] [355E3F04]         
 08  99.8 [741C13E7] [A0FCA76B]         
 09  98.1 [28046873] [74213A74]         
 10  99.8 [BBCB1F02] [D323A234]         
 11  97.7 [8B74D3EC] [B4F94215]         
 12  99.4 [AD9FD8B2] [E6175469]         
 13  99.8 [C9E0F028] [971B67E5]         
korth

  • drfr
  • [*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1357
So it seems to be close to doing what it should but something is missing.
Any idea?


See [a href='index.php?act=findpost&pid=707559']this message[/a]

Perhaps the missing parts?


Thanks. One day I must find the time to read through the whole thread.

CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1358
I saw some discussion about this somewhere else. How can all the tracks return "(0/0) No match but offset" ?


IIRC that is either due to past aggressive cleanup of the AR database (?) or, more likely, when only two conflicting rips have been submitted to the DB and thus await resolution by a third one.
I have a small number of rips that return such a "(0/0) No match but offset" or "(0/0) Rip not accurate"... which sounds a bit counterintuitive indeed.
  • Last Edit: 23 May, 2011, 11:40:05 AM by HotShotFR

CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1359
is it possible to 'verify' against AR/CTDB a Playstation-style cuesheet referencing TWO pre-audio data tracks ? i.e. the first playable audio CD track starts at #3.

I never saw such strange discs, so it's untested. How is cuesheet supposed to look like in this case? Can you send me the cuesheet and eac log?
CUETools 2.1.4

  • Goratrix
  • [*][*][*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1360
when only two conflicting rips have been submitted to the DB and thus await resolution by a third one.


But in that case, it returns just "(0/0) No match" as there is no checksum data available to compare to.

A "(0/0) No match but offset" result indicates there is some data for comparison, but why the 0/0 then? Mind boggles 

  • Marc27
  • [*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1361
I apologize if this is completely nonsense... but I don't know... with all the discussion about AR/CueTools databases. Why can't the internal checksums be used for verification. Since this info is already stored there would be no need to process the audio again (lot faster)
and it should be more accurate (md5 128 bit vs 32 bit).

CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1362
32 bit checksums are quite enough. But you do have a good point about not having to process the audio again. Unfortunately, this won't work, and there is a number of reasons for that.

1) This would require a CD drive to support overread into leadin/leadout, else some leading or trailing samples are lost and hash changes.
2) If the database was track based, you'd have to keep your rip as separate tracks for this to work, and if it was disc image based, you'd have to keep your rip as an image.
3) You'd have to store your files in a format that does have an md5 hash, i.e. no WAV, no ALAC, probably no WMAL... And even in FLAC it's optional.
4) You wouldn't be able to compare your rip with a different pressing.
5) You'd still want to be sure that audio does indeed match the hash.

Frankly, i don't like the fact that FLAC uses md5 hash and i am sometimes tempted to propose to get rid of it (you can, because it's optional) and replace it by some tag containing a good checksum like CRC32. The reason for that is that md5 is the only part of FLAC encoding process that cannot be parallelized. In the age of GPUs and multicore processors that is unacceptable. For example, FLACCL's encoding speed is already limited by md5 calculation speed.

Hash functions should be used for cryptography, not for data integrity checks. They are slow and non-parallelizable. They are meant to protect against deliberate tampering, not hardware/software failures. They have their uses in DRM, but are out of place in open formats like FLAC. md5 as a checksum is not stronger than CRC128, but is slower. And it's an overkill. I would expect FLAC to use Fletcher-16 for frame CRCs and CRC32/64 for file CRC.
  • Last Edit: 25 May, 2011, 05:51:23 PM by Gregory S. Chudov
CUETools 2.1.4

  • sauvage78
  • [*][*][*][*][*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1363
Quote
I would expect FLAC to use Fletcher-16 for frame CRCs and CRC32/64 for file CRC.


Stupid question, (maybe going a little offtopic) would it be noticeably faster on single core CPU too ?

I ask cauz I think I recall that I compared TAK with Flac in the past & I noticed that (IIRW) TAK encoding was mainly much faster than Flac due to the fact that TAK doesn't output MD5 sum by default while Flac does it by default. Once I enabled MD5 in TAK I think I recall the encoding speed difference wasn't so important anymore. (TAK was likely still a little faster, but not as incredibly faster as some test which compare TAK vs. Flac at default setting without caring for MD5. Indeed TAK compression was still 1or1.5% smaller/better.)

I ask cauz as a flac user I would enjoy that the encoding speed gap between Flac & Tak at default [so Flac(With MD5) vs. TAK(Without MD5)] would be reduced ... TAK would still win by a good margin on compression, but the encoding speed comparison would be more fair IMHO.

I mean if it would really work as you say, & more if it's faster on single core too (A slow Sempron 3000+ user like me cares a lot about speed!!!), it would be an interesting improvement to Flac ... well if Josh still knows how to code for Flac  & if he reads CT topics which I doubt anyway ;(

In the end, I think MD5 audio sums should still be available in Flac (just not as the default sum if there is a more efficient alternative as you think), because the main other lossless codecs offer it, because it works & just in case your solution wouldn't work
  • Last Edit: 25 May, 2011, 07:32:10 PM by sauvage78
CDImage+CUE
Secure [Low/C2/AR(2)]
Flac -4

  • Marc27
  • [*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1364
Thanks for the answer and all the information. It's very clear now. I wasn't aware as well of the md5 performance issues...

Edit: about the checksum used, should be "proof of" incorrect ripping settings? IIRC I have read in some threads that some errors may go unnoticed due to this (like C2 correction). Personally I plan to re-rip many albums based on this (though they are reported as accurate by AR), but "in a more perfect world" if they pass AR/Cuetools verification it shouldn't be necessary.
  • Last Edit: 25 May, 2011, 07:49:57 PM by Marc27

  • Porcus
  • [*][*][*][*][*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1365
Edit: about the checksum used, should be "proof of" incorrect ripping settings? IIRC I have read in some threads that some errors may go unnoticed due to this (like C2 correction). Personally I plan to re-rip many albums based on this (though they are reported as accurate by AR), but "in a more perfect world" if they pass AR/Cuetools verification it shouldn't be necessary.


I wouldn't worry too much. Yes there are chances for collisions due to weak CRC, but as Gregory points out, there is no deliberate tampering of data -- the errors are drawn random (subject to passing the drive's error corrections). Yes a few samples at the ends are cropped off too. (You might want to notice that e.g. EAC runs through the ripping procedure first, and checks against AccurateRip afterwards. dBpoweramp, on the other hand, invokes no secure reading mode if a burst rip is reported Accurate.)

If you still worry, then run them through the Cuetools database, which is disc based.

If you still want to re-rip a few discs after this, I would assume that you need to use a different drive in order to detect differences. And «different drive» -- you could probably sort this out, but there are many drives that are so similar in actual spinner and firmware, that you could risk picking up just another CD-ROM drive reading the same way. Now you might want to ask «how likely is that?», but bear in mind that we work under the assumption that AR is not accurate enough. You think «drive collision» is less probable than «CRC collision»?
But let us now assume that you then find a bit-different rip where both new and old rip were reported Accurate: How do you tell which one is wrong? Multiple issues will hit you then:
- Lead-in/lead-out. You could probably sort this out from knowledge of the drives.
- Manufactoring errors. It might simply be that the disc has errors which there is no «single true» way of reading. Presumably that is the case I found here: http://forum.dbpoweramp.com/showthread.php?p=88265 .
- Trying a 3rd drive to judge between the two? Bigger chance that this is a rebadge of, or something working identical as, one of the two others.


My suggestion: scrap the re-ripping at least until you want to sell the CDs (edit: or until you find something audibly suspicious)
  • Last Edit: 26 May, 2011, 03:40:08 AM by Porcus

  • Nassoo
  • [*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1366
Hi! Thanks for the great software!
I have may be stupid question, but... There is an option "Write extraction log to 'LOG' tag"... so, after encoding completed, i don't see this LOG tag in the files... where do i go wrong?

  • korth
  • [*][*][*][*][*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1367
when only two conflicting rips have been submitted to the DB and thus await resolution by a third one.


But in that case, it returns just "(0/0) No match" as there is no checksum data available to compare to.

A "(0/0) No match but offset" result indicates there is some data for comparison, but why the 0/0 then? Mind boggles 


OK I rechecked 4 discs that came up "(0/0) No match but offset" in v2.1.1 and they come up "(0/0) No match" in v2.0.9!
korth

  • korth
  • [*][*][*][*][*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1368
Hi! Thanks for the great software!
I have may be stupid question, but... There is an option "Write extraction log to 'LOG' tag"... so, after encoding completed, i don't see this LOG tag in the files... where do i go wrong?


I think you'll need to provide more info about what type of file and what you're using to view the tags.
There are two such check boxes, one in 'verify' and one in 'encode and verify'. Having the wrong one checked for the process you're using will make a difference.
korth

  • liquefied
  • [*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1369
Also, does this support AccurateRip v2?

  • korth
  • [*][*][*][*][*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1370
Also, does this support AccurateRip v2?


In the next release. [a href='index.php?act=findpost&pid=755982']See this.
[/a]
korth

CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1371
I have may be stupid question, but... There is an option "Write extraction log to 'LOG' tag"... so, after encoding completed, i don't see this LOG tag in the files... where do i go wrong?

This option tries to locate EAC log and place it as a tag into resulting file.
For this to work, EAC log should be in the same folder as the source files, and preferably have the same filename as a CUE sheet.
CUETools will try to locate it even if there are several logs and/or the filenames don't match, but it can do this only if those logs are from a version of EAC that puts CD Table Of Contents into the log file.
CUETools 2.1.4

CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1372
OK I rechecked 4 discs that came up "(0/0) No match but offset" in v2.1.1 and they come up "(0/0) No match" in v2.0.9!

This could be the result of AR database cleanup. There are offset-detection CRCs in this entry, but no AR CRCs.
But there might be a chance that there are only ARv2 CRCs for this CD. Wait for CUETools 2.1.2 to find out.
  • Last Edit: 26 May, 2011, 03:44:47 PM by Gregory S. Chudov
CUETools 2.1.4

  • Marc27
  • [*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1373
dBpoweramp, on the other hand, invokes no secure reading mode if a burst rip is reported Accurate.)

So it's strategy is that if there is an error, AR would detect it, in the mean time the rip is faster. Unfortunately I'm not entirely familiar with dBpoweramp yet.

Switching drives is an option, but I was a bit less ambitious  Same drive but different ripping settings. Bottom line is how likely is that some discs may have undetected errors? IMHO I don't like leaving things to chance. Though if the chance of undetected errors is very* very unlikely I wouldn't worry.

Manufactoring errors. It might simply be that the disc has errors which there is no «single true» way of reading. Presumably that is the case I found here: http://forum.dbpoweramp.com/showthread.php?p=88265 ."

So now I understand those results. I initially thought they were slightly different discs/releases.
Also a from the thread that doesn't apply to the thread case, but more to this one.
Quote
It would be interesting to do a series of tests in EAC to see if you could get both AR results on that track from the same disc by tweaking the EAC settings.

So let's say this case scenario

1. A rip with burst mode and other incorrect settings.
2. Same disc rip with secure mode/correct settings.
3. Both rips have different CRC128/64.
4. AR/Cuetools databases report both as accurate.

How likely or unlikely is 4 (as well as 3)?
I wish there was some sort of study on the matter, but considering all the factors (different rip settings, drives, etc) that may be overkill.

  • Porcus
  • [*][*][*][*][*]
CUETools versions 1.9.5 through 2.1.5 (current)
Reply #1374
So let's say this case scenario

1. A rip with burst mode and other incorrect settings.
2. Same disc rip with secure mode/correct settings.
3. Both rips have different CRC128/64.
4. AR/Cuetools databases report both as accurate.

How likely or unlikely is 4 (as well as 3)?



I am not completely sure what you intend to do.  Let me be a total devil's advocate here, and make the worst-possible interpretation -- not because I think it is correct, but it might explain what can go wrong.


What makes AR/Cuetools verification go round, is basically the assumption that an erroneous rip will hardly ever match anything, because it gives a checksum at random -- if we are lucky, not too many orders of magnitude away from a uniform draw. Now if you tweak a setting (like, channel reversal), this is a error that someone else can plausibly have made before you on that disc.

Now you have two events which might trigger an erroneous "Accurate":
- there is a scratch, but the wrong sample does by chance evaluate to an AR match
or
- the CD is otherwise correct, but by human error, someone else has done the same settings mistake as you have
(or both)

I would guess that the latter event is more likely.

Now if you are even tweaking the settings around, it looks like you are deliberately chasing "has someone else done this mistake before me?". That's not what you want. Besides, that's not what anyone else wants -- you are offering a false "Accurate" to the next person who accidentally makes the same settings mistake.



So ... if you might correct my interpretation ...?