Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: an audio file comparision utility (Read 4352 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

an audio file comparision utility

144 GB, comprised of 8,659 FLAC files, distributed in 781 folders, were copied from one USB connected hard drive to another USB connected hard drive. The operation stopped twice, with two different strange error messages about resources being unavailable. About 9 albums were not copied but were easy to locate and copy afterward.

However, an easy check shows the output more than 35MB short, missing 3 files. I say there is no reason to believe that all of the missing bytes are part of those three files, or even that there may be more than three missing files with some duplicates or other trash in the output. Some files may be present but altered, or any other condition one might think up.

In principle, a manual file by file comparison could be made but it would be very tedious and not certain to catch every error. Is there a nice utility that can compare the selected input data with the selected output data (there is also other material on the source drive, so I need to be able to tell the program what to use) and point out every differences -- hopefully in one operation?

an audio file comparision utility

Reply #1 –
I use "RoboCop RoboCopy" to copy a large batch of files.

If there are any hiccups along the way, like you're having, I use "HashCheck Shell Extension" to resolve it.

an audio file comparision utility

Reply #2 –
RoboCopy is Microsoft's free command-line tool for automating file & folder copying. It is geared toward enterprise sysadmins who have complex copying to do, and overall it is not very easy to use. RoboCop RoboCopy is one of several GUI front ends for it which help you pick the options for constructing the right set of arguments to the tool, but when I tried a similar front end, I still felt it was more just for people who already knew how to use the command-line version.

More generally, you need something that can quickly compare two folders and make one match the other. I rely on file synchronization software for exactly this reason... if there's a problem, you can redo or resume the sync with the click of a button. Some file sync software lets you save copies elsewhere temporarily, so you can undo any mistakes.

I paid for an Allway Sync Pro license a long time ago, and have been happy with it, even though it has a rather clumsy UI. The free version has a limit of 40,000 files copied every 30 days, so it would probably work fine for you. For a backup/mirror, you just have to make sure you set it up to propagate changes in one direction, not both. The linked text above may be helpful in choosing something better.

an audio file comparision utility

Reply #3 –
Most stupid question first: Is this "easy check" that "shows the output more than 35MB short, missing 3 files", done drive-to-drive? Then they could be hidden/system files.


Now ... for the audio files, FLAC is a checksummed format. It is possible to detect corruption with VUPlayer's audiotester.exe or fb2k's integrity verifier. Beware though that fb2k has - at least in earlier versions - not recognized too badly damaged audio files at all. (That is, they wouldn't be dumped into the playlist, so they wouldn't be tested.)


There are various copying utilities which can do bit-by-bit checking. I use XXCopy, free for personal use. If S: is your source drive and T: is your target drive, the command would be xxcopy S:\musicfolder T:\musicfolder /BACKUP/CDU/L for merely checking; drop the "/L" for actually overwriting. It will not report the number of different bytes though.

Quote from: AndyH-ha on
The operation stopped twice, with two different strange error messages about resources being unavailable.


You might want to do a chkdisk. (If this were a fresh copy I would have considered scrapping it and copying anew; I don't know whether that saves time on the most rigorous chkdsk operation, which also goes for the emtpy space. And ... is there anything else on the drive? If not, reformatting with "Quick format" UNchecked implies a chkdsk I think.)

And, have you turned off drive caching? Dunno when Microsoft started turning that off by default for USB drives. Probably unrelated, but something one should always be worried about when using USB on (older?) MS-Windows versions.

an audio file comparision utility

Reply #4 –
Compare them with FreeFileSync. It shows exactly what files are missing.

an audio file comparision utility

Reply #5 –
Resources unavailable sounds like some USB power saving kicked in. This happens even while copying something. You can try to disable all power saving for USB while copying.
I use Total Commander fir such copy jobs. It can also compare folders for differences.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

an audio file comparision utility

Reply #6 –
If it was me, it would make me wonder why my computer couldn't perform the most basic of tasks properly, and what else it might be getting wrong that I hadn't noticed yet.

cheers,
David.

an audio file comparision utility

Reply #7 –
The simple test was right clicking on the source drive folder that contained the 780 album folders, to select and look at the properties, which shows # of folders, # of files, and total size in bytes, as well as size on disk.  The same operation on the output drive should give the same numbers, at least for the first three quantities -- and there is no reason to suspect system or hidden files.

I've had many USB disk problems, with a number of different hard drives and different USB interfaces. The "interfaces" are the connection between hard drive and USB port. There is always also a drive power supply. I've used these interfaces on a number of hard drives, both new and used.

The hard drives have been extensively tested with independent utilities before being assigned to backup duties. By this I mean I have a boot from CD Linux disk utility set, a boot from CD Seagate disk utility and a boot from CD Western Digital disk utility that are independent of the computer's OS and that I can use on different computers.

Every once in a while one or another of the hard drives gets declared as unformatted, or is just not recognized. This also happens with a 3.5" USB hard drive (i.e. not a internal hard drive pressed into USB service but one in its own case that can only be used USB) that requires a separate power supply (different power supply with different style connection to the drive than the other USB any-hard-drive interfaces) AND with a notebook sized USB drive that uses the USB port for power.

Most of the time any one of these drives is either completely useable or completely not useable when I first plug it in but a few times this loss of recognition has happened suddenly while the drive is in use. When I can't find one of these drives in Windows Explorer, i.e. the computer does not seem to see the drive at all, I can sometimes find it under computer management/disk management where it is marked as uninitiated.

When one of these annoying conditions arise, it is port independent, i.e. no improvement by changing USB ports or by changing to a different USB interface. It is the same on all three of my (not networked) computers (which have different OS versions) and, for at least one of the drives, the same when taken to another location to try on another computer. It is also often not solved by booting into Linux or some version of DOS from one of the disk test CDs

Yet every one of the drives has always eventually spontaneously recovered from this temporary insanity and worked properly again. As far as I can tell, none have lost or corrupted data, except in a case like this where the problem occurred during an operation.

This problem was different, however, the entire drive was not "lost". The first error message, at the file level, was that the destination file was "unavailable"; it did not effect the entire disk. The second was that the source folder was "unavailable". The drives did not become invisible to the OS.

This is very annoying but so pervasive that I have come to accept it as a basic USB fault, not something that just somehow infects every computer I use, even when its not my computer. So, yes, I have wondered why, but since it is only USB related, I don't think it is the computers per se.

an audio file comparision utility

Reply #8 –
I have about 10TB of USB HDDs and 6TB in my PC. You can imagine how much copying I have done. I have never seen anything like what you describe.  Only problem is the speed of USB2.

Maybe someone else knows what's happening.

Sorry I don't have any good ideas or suggestions for you Andy.

Cheers,
David.

an audio file comparision utility

Reply #10 –
I expect the problem will be that you need to compare files between the source and destination to pickup which files suffered most likely an incomplete file copy. In situations where I need to make sure a duplicate I've undertaken matches I (recursively) parse the source and destination using MD5SUM or similar and redirect the output to a file. I then compare these files with ExamDiff or a diff plugin in Notepad++. The files with mismatched checksum will standout easily. Oh, and it might be necessary to slightly modify your output files if the file path is different. This is easily done with a find/replace so that just the checksum itself will be different between the two files.

an audio file comparision utility

Reply #11 –
It is interesting information that not initialized, not formatted, or simply not existing, USB hard drive problems are not common, but doesn't make my difficulties any clearer.

The two error message conditions I got on this copy operation were brand new, to me. I did copy the two original USB backup drives to the third USB backup drive, without any problems that I can recall. They were not simple copy everything at once operations. I organized things differently on the third drive, but it seems likely that the collection of LP backup I was attempting to copy this time was accomplished in one copy operation at that time too.

Unstable motherboardS, at least four of them, are a possibility but that seems like a low probability. This machine has had a printer, a trackball, and a flashdrive plugged into rear USB ports for quite some time. No difficulties have occurred with any of those. I also use several other flashdrives to transfer information between computers, fairly often, without hassle. It is only USB connected hard drives that act up.

There are several suggestions for controlled copying that might be useful for me to know about but I really hope to avoid repeating the five hours of effort already done. This new drive is just temporary, a working collection. If I can find the data differences, I can recopy just the necessary files, then go on to the next steps.

There are some suggestions offered for checking the files. I haven't yet had time to investigate, so I don't know if any can do what I want: check everything and point out the differences - in batch mode. Finding out will have to be my next move.

an audio file comparision utility

Reply #12 –
Basically it smells like a hardware problem, to force over this I would probably construct and run some sort of rsync script more than once, I guess on windows total commander dir sync should do the trick (I would start with cleanly reformated desc drive).

Another way would be to use a different type of connection or OS, for example;
- a. extract the drive from enclosure (if possible) and mount it into the case or use another usb enclosure.
- b. boot with some live linux and use rsync ( perhaps https://grml.org/ )

p.s.
- And obviously, if the usb drive is new, return it asap (You can tell the seller that it was attacking your dog).
- warning: rsync manual is freaking annoying/unclear.
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

an audio file comparision utility

Reply #13 –
The way I'd approach your problem would be a two-stage solution.

Step 1: ensure all files copied over correctly

To ensure the files were all copied over, let the computer do the donkey work by using a diff tool.

I'll assume you're using Windows and you'd prefer a free graphical tool, so you can use the Notepad++ compare plug-in (http://sourceforge.net/projects/npp-compare/) or you can use Meld (http://meldmerge.org/).

List the contents of the directories in text files to compare them, using the command line console.  Navigate to root folder of your files and list their entire contents in a text file:
Code: [Select]
cd /D E:\MyOriginalFiles
dir /s /b > C:\Users\YourUserName\Desktop\FileList1.txt


Repeat the commands on the files that were copied over, this time outputting the list to FileList2.txt.  Use the diff tool to compare the file lists and it will highlight any differences.  Now manually copy over any missing files.


Step 2: check copied files are identical to originals

Now that we have identical contents on each drive, let the computer do the donkey work again by verifying the content (md5 hash) of each copied file to its original.

I use Linux so an md5 tool is immediately accessible from the terminal but you can download a free Windows tool (http://md5deep.sourceforge.net/).  This program can recursively examine an entire directory tree so you can easily run it for all your original files.

Using the hashes generated from this scan, you can use the program to verify whether all the copied files are bit-perfect copies of the originals.  Manually copy over any files where the md5 doesn't match its original counterpart.

an audio file comparision utility

Reply #14 –
'Tera Copy' is another program , which you can use to copy as well as check. Very versatile program.

an audio file comparision utility

Reply #16 –
To check the source files against the destination copies:

rsync -cnva /source/dir/ /destination/dir/


As for the USB issues, I backup my server's datasets over 1Gbps ethernet to a USB 3 drive connected to another computer.

# zpool list
NAME      SIZE  ALLOC  FREE  EXPANDSZ  FRAG    CAP  DEDUP  HEALTH  ALTROOT
backup1  2.72T  1.63T  1.08T        -      -    60%  1.00x  ONLINE  -

From zpool status:
  scan: scrub repaired 0 in 6h25m with 0 errors on Mon Dec 14 19:55:42 2015

That's 1.63TB of data examined and compared against stored block level checksums without any errors, at north of 70MBps, all over USB. To date I have yet to have a single error crop up during a scrub of my USB connected backup drives that weren't due to a bad drive. It certainly had nothing to do with USB.

You have some other issue going on.

Edit: I think we got necro trolled
 

an audio file comparision utility

Reply #17 –
Quote from: yourlord on
Edit: I think we got necro trolled

No, you got necro-spammed.

Piling on to a dead topic can be prevented by paying attention to the dates of the posts.  Sometimes you might see that someone has added a reply to a discussion but you can't see the reply.  This generally happens when a spammer is banned but the staff member doing the banning didn't follow-through and remove the hidden post from the discussion.