Original idea from Transfer files from an FTP server to S3 by "Hack N Cheese".
I moved roughly a terrabyte in less than an hour. Granted, I couldn't take advantage of lftp
's --parallel=30
switch due to my ftp source limiting me to one connection at a time, but use-pget-n=N
did seem to help out.
- Get a fast Ubuntu 14.4 EC2 box on Amazon for temporary usage (I went with
m1.xlarge
) so data tranfers aren't limited by your local bandwidth at least. I also attached a fat 2TB EBS volume and symlinked it to/bigdisk
, and made sure the EBS volume was deleted after I terminated this EC2 box. I hopelftp
2.6.4 is available as a stable package by the next time I attempt this. - Build
lftp
2.6.4+ (Not easy to compile, so read theINSTALL
file and plow through all your missing dependencies - you'll also need to re-run `sudo ./configure && su