Deleting Duplicate Files
Discussion
I have a load of MP3s that have been bastardised by various organisation programs over the years. I can't remember what the last one I used on them was but it made a proper mess of them! This was a few years ago, but I finally decided to try to sort them out because I've sorted out a major blocker - I've managed to setup something that will identify albums with missing tracks, so once they're all sorted I can't start filling the gaps back in.
Anyway, I've got a process running now using MusicBrainz Picard to put everything into the right Artist/Artist - Album folders and rename all the tracks to the right format. I'll then import the results into Lidarr and go hunting for missing tracks.
But I've been spot-checking albums and identified an issue with duplication - I don't know why but I naively assumed that MusicBrainz would not create duplicate tracks within an Album, because it's really easy to end up with duplicates and no one wants that. But it doesn't, so I have "01 TRACK" and "01 TRACK (01)" and "01 TRACK (2)" etc, which is more than a little irritating.
Does anyone know an easy way to delete these? Ideally a method that compares the files in the folder rather than relying on the file name, though perhaps I need to do both as I expect there are some different file formats of the same track.
Thanks
Anyway, I've got a process running now using MusicBrainz Picard to put everything into the right Artist/Artist - Album folders and rename all the tracks to the right format. I'll then import the results into Lidarr and go hunting for missing tracks.
But I've been spot-checking albums and identified an issue with duplication - I don't know why but I naively assumed that MusicBrainz would not create duplicate tracks within an Album, because it's really easy to end up with duplicates and no one wants that. But it doesn't, so I have "01 TRACK" and "01 TRACK (01)" and "01 TRACK (2)" etc, which is more than a little irritating.
Does anyone know an easy way to delete these? Ideally a method that compares the files in the folder rather than relying on the file name, though perhaps I need to do both as I expect there are some different file formats of the same track.
Thanks
Doublekiller. Free, and will compare based on one or more of file contents, size, date, you name it, and then either delete things it thinks are duplicates, or optionally move them to another folder for you to check.
https://www.bigbangenterprises.de/en/doublekiller/
https://www.bigbangenterprises.de/en/doublekiller/
bristoltype603 said:
Probably not a lot of help but the way I solved this problem a while ago was using a (free) program called jdupes. Brilliant but as far I can tell only runs on linux. Maybe there's something similar for windows?
Thanks, the files are on a linux-based server and you can run Linux in Windows too now, so I can probably make that work. Will look at that other option too.You could just look for any files that match *(*)* - review the list in case you've got some "The song - the artist (featuring someone else)".
To avoid that you could use a regex to look for anything with (<a number>), sonething like find . -regex .*\([0-9]\).* from memory.
stick the resulting list in to a file and then just alter the file to delete them and execute.
To avoid that you could use a regex to look for anything with (<a number>), sonething like find . -regex .*\([0-9]\).* from memory.
stick the resulting list in to a file and then just alter the file to delete them and execute.
I use https://www.digitalvolcano.co.uk/duplicatecleaner....
It has an audio track mode, where you can compare tags as well as file content.
There is a free version you can see if it does what you want. I was sceptical at first, but it is one of the few free trials that I've actually gone on to buy!
It has an audio track mode, where you can compare tags as well as file content.
There is a free version you can see if it does what you want. I was sceptical at first, but it is one of the few free trials that I've actually gone on to buy!
There are utilities available which find duplicates even when the file names are not the same.
One example is File Fusion from Abelssoft:
https://www.abelssoft.de/en/windows/system-utiliti...
I've used that just to find duplicates, but I don't let it do it's optimisation. As it's trying to be easy to use, it does lack control when it searches etc
Another which I've yet to try, although I should as it's free, is this one:
http://www.find-duplicates.com/Duplicate_File_Find...
This one has various guises too, one is specifically for mp3 files:
http://www.find-duplicates.com/Duplicate_MP3_Finde...
One example is File Fusion from Abelssoft:
https://www.abelssoft.de/en/windows/system-utiliti...
I've used that just to find duplicates, but I don't let it do it's optimisation. As it's trying to be easy to use, it does lack control when it searches etc
Another which I've yet to try, although I should as it's free, is this one:
http://www.find-duplicates.com/Duplicate_File_Find...
This one has various guises too, one is specifically for mp3 files:
http://www.find-duplicates.com/Duplicate_MP3_Finde...
Thanks, I'm currently trying Duplicate Killer first.
It's done a decent job, but it's hit an issue with different bit-rates on the first run. Different file-size = No duplicate. I'm running it through again matching only on checksum, but I'm not sure if this as a good idea, whether the checksums will be different anyway or whether I'd be better using something else for this step.
It's done a decent job, but it's hit an issue with different bit-rates on the first run. Different file-size = No duplicate. I'm running it through again matching only on checksum, but I'm not sure if this as a good idea, whether the checksums will be different anyway or whether I'd be better using something else for this step.
Crafty_ said:
You could just look for any files that match *(*)* - review the list in case you've got some "The song - the artist (featuring someone else)".
To avoid that you could use a regex to look for anything with (<a number>), sonething like find . -regex .*\([0-9]\).* from memory.
stick the resulting list in to a file and then just alter the file to delete them and execute.
It looks like this might be my best shot, I tried a few options but not really getting anywhere. I understand how this is working, but what program should use to perform the search and delete?To avoid that you could use a regex to look for anything with (<a number>), sonething like find . -regex .*\([0-9]\).* from memory.
stick the resulting list in to a file and then just alter the file to delete them and execute.
xeny said:
paulrockliffe said:
Proper slow just comparing checksums, I guess that takes a bit more effort than looking at file sizes. Oh well, I'm in no rush!
If the sizes are different, the checksums will definitely differ.paulrockliffe said:
Crafty_ said:
You could just look for any files that match *(*)* - review the list in case you've got some "The song - the artist (featuring someone else)".
To avoid that you could use a regex to look for anything with (<a number>), sonething like find . -regex .*\([0-9]\).* from memory.
stick the resulting list in to a file and then just alter the file to delete them and execute.
It looks like this might be my best shot, I tried a few options but not really getting anywhere. I understand how this is working, but what program should use to perform the search and delete?To avoid that you could use a regex to look for anything with (<a number>), sonething like find . -regex .*\([0-9]\).* from memory.
stick the resulting list in to a file and then just alter the file to delete them and execute.
then use vi to edit the file, if there are a lot you could probably use another regex to look for "anything followed by a full stop" and replace it with "rm whatever the file name was.", rename the file with a .sh extension and then ./list_file_files.sh
something along those lines anyway, not a total linux geek but you don't need any software.
.. if you are doing it on windows I'd download notepad++, get a file listing use a regex to find the duplicates using a regex like above, then use a macro to insert "del " at the start of each line.
paulrockliffe said:
Yeah, that's what I thought, so I cancelled it out and tried it on a subfolder with duplicates to check. Didn't find any.
That's very odd - I get the impression you're using it on a network file system - does it find duplicates for you on a local file system?It doesn't like files over 2GB, but that shouldn't be a problem for MP3s.
xeny said:
paulrockliffe said:
Yeah, that's what I thought, so I cancelled it out and tried it on a subfolder with duplicates to check. Didn't find any.
That's very odd - I get the impression you're using it on a network file system - does it find duplicates for you on a local file system?It doesn't like files over 2GB, but that shouldn't be a problem for MP3s.
I think what I want to do is more basic - find and delete any file with a name that ends in "(number)". I'm not too bothered if that's a bit crude as the next step will be to run it all through Lidarr and fill in any gaps. I already have gaps so it doesn't matter if there's more to fill.
xeny said:
What happens if you simply use find in Windows Explorer to search for ) do a quick eyeball they're all expendable, select all and delete?
I tried searching in windows and for some reason searching for *(*)* returned all the files! I decided that I didn't understand search well enough and that's what prompted me to ask how I search and delete earlier in the thread! That would be the easiest approach. I think I know how to do it in Power Shell or Linux command prompt thingy, but I'll need some time to google exactly what I need to type in, so I've not tried that yet.
Rampant Golf said:
Treesize search on the md5 checksum. It'll deal with 80% of the duplication and the rest will be a manual effort due to varying sizes and qualities of the files. Remember some duplication is good - compilations etc.
I don't have any compilations as I only listen to proper albums and one of my previous attempts to sort it out would have split any compilations into their component albums anyway, I'm sure. I don't really care about retaining the best sound quality as over time Lidarr will pick up any poor quality versions and replace them, so it doens't matter too much which file ends up being retained.
Looks like this:
~"*(*)" name:
Is the Windows Search I was looking for.
Seems to be working OK, though it would be better if there was a number wildcard available as it's going to kill some song names that have something in brackets in the name such as Don't Ever Think (Too Much) by the Zutons. But I'll probably just murder those and rebuild rather than messing around any more.
~"*(*)" name:
Is the Windows Search I was looking for.
Seems to be working OK, though it would be better if there was a number wildcard available as it's going to kill some song names that have something in brackets in the name such as Don't Ever Think (Too Much) by the Zutons. But I'll probably just murder those and rebuild rather than messing around any more.
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff