Created
October 18, 2017 13:14
-
-
Save frosforever/501a1713404a279b479fd4f93743f393 to your computer and use it in GitHub Desktop.
create Redshift manifest from prefix ignoring `SUCCESS` files
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aws s3api list-objects --bucket BUCKET_NAME --prefix WTVR/PREFIX/ --query 'Contents[?!contains(Key,`SUCCESS`)].{Key:Key}' --output json | jq '[.[] | .["url"] = "s3://BUCKET_NAME/" + .Key | .["mandatory"] = true | del(.Key)] | { entries: .}' |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
can then pipe to
s3 cp
to put back on s3:... | aws s3 cp - s3://BUCKET/manifest.json