Skip to content

Instantly share code, notes, and snippets.

@carbonrobot
Last active January 8, 2019 22:02
Show Gist options
  • Save carbonrobot/bbebeea4ca0a9259b979fcd7c22968b2 to your computer and use it in GitHub Desktop.
Save carbonrobot/bbebeea4ca0a9259b979fcd7c22968b2 to your computer and use it in GitHub Desktop.
Recursively get files from an s3 bucket with nodejs aws-sdk
const AWS = require('aws-sdk');
const s3 = new AWS.S3();
function getObjects({ bucket, marker, prev }, callback) {
var params = {
Bucket: bucket,
Marker: marker,
}
return s3.listObjects(params).promise().then(data => {
let objects = data.Contents;
console.log(`Found ${objects.length} objects...`);
if(prev) {
objects = prev.concat(objects);
}
if(data.IsTruncated){
const length = data.Contents.length;
const marker = data.Contents[length - 1].Key;
return getObjects({ bucket, marker, prev: objects});
}
return objects;
});
}
getObjects({
bucket: 'alliance-pa-platform-dev-audit-records-bucket'
}).then(objects => {
console.log(`${objects.length} objects total`);
});
// Each object is of the format
{
Key: '112435-8509-478d-bf24-d6f4dcb68ae9',
LastModified: 2018-12-20T12:01:12.000Z,
ETag: '"7d73903cfde1wqe3418ef69a2c937073"',
Size: 27970,
StorageClass: 'STANDARD',
Owner: [Object]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment