Created
April 19, 2018 14:59
-
-
Save joshmosh/c436b66e9ca037d565c80737bec3fcb8 to your computer and use it in GitHub Desktop.
How to upload a directory of files in parallel to S3 using Ruby and Parallel
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| require "parallel" | |
| require "aws-sdk-s3" | |
| # Read more about the aws sdk config here: https://github.com/aws/aws-sdk-ruby#configuration | |
| Aws.config.update( | |
| region: "replace_with_region_id", | |
| credentials: Aws::Credentials.new( | |
| "reaplce_with_aws_access_key_id", | |
| "reaplce_with_aws_secret_access_key" | |
| ) | |
| ) | |
| s3 = Aws::S3::Resource.new | |
| # Remove the in_process parameter if you want Parallel to detect number of CPUs | |
| Parallel.each(Dir.glob("./test/*.{mp3,wav,m4a}"), in_processes: 4) do |path| | |
| filename = File.basename(path) | |
| object = s3.bucket("reaplce_with_bucket_name").object(filename) | |
| puts "Uploading #{filename}" | |
| object.upload_file(path) | |
| end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # If you're feeling adventurous you can add the ruby-progressbar gem and hook it in with | |
| # parallel. Parallel will automatically detect the amount of items being processed and | |
| # configure ruby-progressbar accordingly. Read more about ruby-progressbar here: | |
| # https://github.com/jfelchner/ruby-progressbar/wiki | |
| require "parallel" | |
| require "ruby-progressbar" | |
| require "aws-sdk-s3" | |
| # Read more about the aws sdk config here: https://github.com/aws/aws-sdk-ruby#configuration | |
| Aws.config.update( | |
| region: "replace_with_region_id", | |
| credentials: Aws::Credentials.new( | |
| "reaplce_with_aws_access_key_id", | |
| "reaplce_with_aws_secret_access_key" | |
| ) | |
| ) | |
| s3 = Aws::S3::Resource.new | |
| audio_files = Dir.glob("./test/*.{mp3,wav,m4a}") | |
| options = { | |
| in_processes: 4, | |
| progress: "Uploading audio files" | |
| } | |
| Parallel.each(audio_files, options) do |path| | |
| filename = File.basename(path) | |
| object = s3.bucket("reaplce_with_bucket_name").object(filename) | |
| object.upload_file(path) | |
| end |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment