Skip to main content

Topic: Audio (fingerprinting) duplicate finders - do they really all suck big time? (Read 1109 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Porcus
  • [*][*][*][*][*]
Audio (fingerprinting) duplicate finders - do they really all suck big time?
I have tried a few free and trialwares. Some of them I need to re-review because I do not remember why I uninstalled each. Currently I have the following five installed. None of them can identify redundant folders/directories, but Duplicate Cleaner promises that the paid version has such a feature.

* DupeGuru Music Edition from hardcoded.net:
I use this because it quite efficiently scans for audio-only parts, and appears fairly user friendly. It works with my Windows 7, although Windows is not supported anymore - YMMV.
This one does not do "fingerprinting", it only has ambition of checking for identical audio.
Sucks because:
Fails to identify different files with precisely the same audio, because it does not decode. The developer knows e.g. the FLAC format well enough to isolate the audio section from the tags section, but does then evidently just make a hash of the audio part of the file, not the decoded audio.  So two FLAC files with different encoding parameters - as well as FLAC and WAV - of the same audio, are not matched.
(Since it knows tags, then why not match by MD5 sum? I can use a separate utility to verify that the files are not broken ...)

* PerfectTunes from Illustrate (Spoon, the creator of AccurateRip and dBpoweramp, co-developer of fb2k Mobile etc):
Does fingerprinting. But:
Seems to be no way to score similarity, nor highlight the ones that are bit-identical

* Similarity, http://www.similarityapp.com/
Does fingerprinting. Scores in terms of similarity. Has a "quality" meter to help you choose which ones to keep, if you are too lazy for listening; in the least, the "max frequency" figure can help identify upconverted transcodes.
Sucks because:
Does not distinguish out bit-identical files. Gives a full 100.0 % similarity score even to files which are not identical.
Even instrumental vs. vocal versions get nearly full similarity score.
Its quality meter has a clipping indicator which is doubtful to say the least. (You just apply gain to the file, and it is instantly fooled.)
Does not know WavPack.
Does not know ID3v2.4

* Mediamonkey with some script for the purpose (where do I find its name?)
Only picks a selected portion of the file and makes a hash.

* Duplicate Cleaner Free. Too few features in the free version (cannot even identify by audio content), so I can just use Ccleaner instead - but the paid version allegedly scans for duplicate folders and audio content.


I guess others have experience to expand the list?

  • spoon
  • [*][*][*][*][*]
  • Administrator
Re: Audio (fingerprinting) duplicate finders - do they really all suck big time?
Reply #1
RE PectectTUNES, Matches are shown as either:

Certain Matches  where the audio is 100% identical between the two files,

Possible Matches   the audio is similar between the files,

  • Porcus
  • [*][*][*][*][*]
Re: Audio (fingerprinting) duplicate finders - do they really all suck big time?
Reply #2
My apologies for the inaccuracy, and thanks for the correction.

  • Moni
  • [*][*]
Re: Audio (fingerprinting) duplicate finders - do they really all suck big time?
Reply #3
I have acquired enough music over the years, and fragmented my collection over numerous drives, that this is a concern for me. I use Roon and it has certainly grabbed a good number of them but a purpose-built tool is very worthwhile.

  • sanskrit44
  • [*]
Re: Audio (fingerprinting) duplicate finders - do they really all suck big time?
Reply #4
i wish i could contribute, but i just stumbled over this thread while searching for a free linux alternative as well. unfortunately it seems there is no tool avaiable that does fingerprinting at all?

  • Porcus
  • [*][*][*][*][*]
Re: Audio (fingerprinting) duplicate finders - do they really all suck big time?
Reply #5
Why I do not use PerfectTunes for deduplication (although I have used it for AccurateRip verification).  These four are supposed matches:
* https://www.youtube.com/watch?v=LcLc-8Ay_Ys
* Track 1 from https://yithmetal.bandcamp.com/album/demo-3
* Track 5 from the same
* https://www.youtube.com/watch?v=xVANJYZlbv4
And I cannot tweak the selectivity (have not checked the paid version).

Its "Certain" functionality (thanks again, Spoon) remedies some of DupeGuruME's stupidness, but I will still use DupeGuruME for speed, as it does not do fingerprinting.
  • Last Edit: 16 May, 2017, 04:05:32 PM by Porcus

  • Porcus
  • [*][*][*][*][*]
Re: Audio (fingerprinting) duplicate finders - do they really all suck big time?
Reply #6
i wish i could contribute, but i just stumbled over this thread while searching for a free linux alternative as well. unfortunately it seems there is no tool avaiable that does fingerprinting at all?

Sure there are tools that do fingerprinting (like Picard), but are there deduplication utilities?
You could try Similarity with Wine or the OSX version with https://www.darlinghq.org/ ?

  • spoon
  • [*][*][*][*][*]
  • Administrator
Re: Audio (fingerprinting) duplicate finders - do they really all suck big time?
Reply #7
Why I do not use PerfectTunes for deduplication (although I have used it for AccurateRip verification).  These four are supposed matches:
* https://www.youtube.com/watch?v=LcLc-8Ay_Ys
* Track 1 from https://yithmetal.bandcamp.com/album/demo-3
* Track 5 from the same
* https://www.youtube.com/watch?v=xVANJYZlbv4
And I cannot tweak the selectivity (have not checked the paid version).

Its "Certain" functionality (thanks again, Spoon) remedies some of DupeGuruME's stupidness, but I will still use DupeGuruME for speed, as it does not do fingerprinting.

The issue IMHO of allowing a slider of match accuracy is that for anyone with more than a screen full of matches, moving the slider would have unknown effects on other matches. PerfectTUNES allows the quick 'hiding' of matches, which should be used in your instance for those tracks.

  • Porcus
  • [*][*][*][*][*]
Re: Audio (fingerprinting) duplicate finders - do they really all suck big time?
Reply #8
The issue IMHO of allowing a slider of match accuracy is that for anyone with more than a screen full of matches, moving the slider would have unknown effects on other matches. PerfectTUNES allows the quick 'hiding' of matches, which should be used in your instance for those tracks.
Is there a way to quickly 'hide' the most obvious 1200 out of 1500 matches? That 'slider' could easily have done so, couldn't it?

  • Porcus
  • [*][*][*][*][*]
Re: Audio (fingerprinting) duplicate finders - do they really all suck big time?
Reply #9
From https://www.lifewire.com/eliminate-duplicate-songs-with-these-free-tools-2438770 , I have already mentioned Duplicate Cleaner Free (#02 on their list) and Similarity (#03, which I actually use).
#01, AllDup, can seemingly compare audio-only, but not in the free version.  The two others cannot, it seems.


One that fails to work altogether on my computer (gives error message): https://software.amato.com.br/content/mp3-duplicate-finder . Is supposed to use fingerprints, but only .mp3 and .ogg.
  • Last Edit: 17 May, 2017, 02:00:27 PM by Porcus

  • Porcus
  • [*][*][*][*][*]
Re: Audio (fingerprinting) duplicate finders - do they really all suck big time?
Reply #10
AudioDedupe from Mindgems shows some promise. Unlike Similarity, which offers a freemium model, this is trialware-nagware where you have to press OK buttons all the time.

Notes:

- Bad: does not distinguish out bit-identical files. Gives full "100%" score to files with differing audio content (tried before and after a CUETools repair).
- Good: Seems to be able to distinguish better than Similarity on mixes with and without vocals (same instruments, one instrumental version), as well as different-language versions (same singer).
- Good: does not give "100%" similarity to FLAC and an MP3 generated from it, it seems - "only" somewhat above 99%
- Which brings me to an unnecessary nuissance: I can set a minimum similarity threshold in 10ths of percents, e.g. 99.7 - but I cannot display it. If it is > 99%, I must bump up the threshold til it disappears.

And then something I forgot about Similarity: no good at checking for different offsets. Need to test PerfectTunes on that too. Stay untuned.

  • Porcus
  • [*][*][*][*][*]
Re: Audio (fingerprinting) duplicate finders - do they really all suck big time?
Reply #11
Audio Comparer. (http://audiocomparer.com/) certainly should be able to get the dust up from your carpet (and spit it in your face, I guess).
30-days trialware.  Has both a choice beween exact and similar, and can in the preferences set a similarity threshold.

Sucks because of major user-unfriendliness, but let me first consider how it does its thing (when I can get it to do its thing):
-> "Exact" identifies a lossless file with an and MP3 I made of the same song.  That's not "exact".  On the other hand, it does distinguish different masterings of the same recordings, so it isn't completely stupid.
-> Ditches some tracks without notifying me.  (Could be length ... though somewhat more than a minute should be enough?)

And then a long list:
-> The results window can be sorted by a few columns, but not by similarity score - and I cannot customize the columns (and there is none for codec or filetype)
-> Sucks at reporting metadata and the like.  Only reports metadata for a few of the files in the list, leaving the rest blank ("Reload tags" does not help), and routinely reports bitrates at around 1380 for both FLAC and MP3 files. 
-> No drag + drop, asks me to navigate to each folder (PerfectTunes should have a whipping for this too) - but fortunately, I can type the folder names.
-> I have to select a file (possibly type in a file name) for the group and results.  Couldn't it instead offer me an option to save to file the few times it tells me something useful?  (It isn't that stupid when I enter the wizard at startup though.)
-> I can add a single folder to a group, and then it starts scanning.  I must wait for it to finish before I can add another.  (Trying a wildcard hack ... it takes the entire drive?)
-> ... and if I then by mistake add a folder twice, they are reported as duplicate identical files.  Dare not even try asking to delete one of them!
-> I can remove folders from groups, and I can mark several by shift + arrows, but not shift + pagedown.  If I have many to remove ... yawn.
-> Exits fullscreen when I hit the Preferences shortcut.
-> Bugware!  I cannot add a new group after one round of comparisons - then it throws an error message of a missing file (in its own folder - one they forgot to include in the install then?).  Not only that, it requests me to submit the error via email, and even if I uncheck that, it asks me to type in a description on how to reproduce it.  Some times it is kind enough to display a Restart button - and hitting that, it still pops up the description dialogue.
This error sometimes occurs on the first round too (freshly started program).

It appears as being reasonably fast once you have typed all the BS and until it is done comparing (and possibly displays the results, possibly crashes).
  • Last Edit: 25 May, 2017, 01:12:31 PM by Porcus