Created
August 20, 2018 04:50
-
-
Save sheldonhull/e286bd05ff154b47c8a1f8ecf2bdc22b to your computer and use it in GitHub Desktop.
Fixing Duplicates in Google Drive using Rclone to Dedupe
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<# | |
RCLONE HELP | |
https://rclone.org/commands/rclone_dedupe/ | |
Useful options I choose | |
--max-depth int | |
--dry-run | |
--log-file=PATH | |
--tpslimit 1 # can help prevent rate limiting errors you might see if you run verbose x2, ie `-vv` | |
--checkers 1 | |
#> | |
# Install Rclone and a log viewer to make it nice to review results | |
choco upgrade rclone -y | |
choco upgrade tailblazer -y # nice tail log view for streaming rclone activity | |
# Alias for usage | |
new-alias rclone -value "C:\Program Files\rclone\rclone-v1.42-windows-amd64\rclone.exe" -force | |
# logging to this directory | |
New-Item 'C:\temp' -ItemType Directory -force | |
$LogFile = "C:\temp\rclone-$(Get-Date -format 'yyyy-MM-dd').log" | |
# Tested Against Folder With Dups With Dry Run | |
rclone dedupe newest googleappsdrive:Test --log-file=$LogFile --dry-run | |
# Ran against folder and it removed 3 of the 4, leaving only one file, now deduplicated | |
rclone dedupe newest googleappsdrive:Test --log-file=$LogFile | |
# another folder with dups but larger. This ran into issues when I didn't limit depth | |
rclone dedupe newest --dry-run googleappsdrive:"Amazon Drive\Development" --log-file=$LogFile --max-depth 2 | |
# merge the root folder level only of duplicate folders | |
rclone dedupe newest googleappsdrive:"" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 1 --stats=30s | |
# now dig into my Lightroom Library and deduplicate a bit more. I did this to confirm it was working before I did everything in my folders. | |
rclone dedupe newest googleappsdrive:"Lightroom" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 1 --stats=30s | |
rclone dedupe newest googleappsdrive:"Lightroom" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 2 --stats=30s | |
rclone dedupe newest googleappsdrive:"Lightroom" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 3 --stats=30s | |
# now that I've got everything figured out, I ran against the entire drive. | |
rclone dedupe newest googleappsdrive:"" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --stats=30s |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment