Scripts to add to your Colab Notebook to sync with an S3 bucket that contains code to run.
- Write AWS credentials to disk in order to use
s3fs
%%writefile ~/.passwd-s3fs
<AWS_ACCESS_KEY_ID>:<AWS_SECRET_ACCESS_KEY>
!chmod 600 ~/.passwd-s3fs
- Set environment variables with AWS credentials
import os
os.environ['AWS_ACCESS_KEY_ID'] = '<AWS_ACCESS_KEY_ID>'
os.environ['AWS_SECRET_ACCESS_KEY'] = '<AWS_SECRET_ACCESS_KEY>'
os.environ['AWS_DEFAULT_REGION'] = 'us-east-1'
- Install and use
s3fs
!apt install s3fs
!mkdir /s3
!chmod 777 /s3
!mkdir /s3/<BUCKET_NAME>
!s3fs <BUCKET_NAME> /s3/<BUCKET_NAME>
- Check that the path exists (this should print all the files in the top level of your S3 bucket)
ls /s3/<BUCKET_NAME>
- Finally, add the code from S3 to your PYTHONPATH so that it can be accessed.
sys.path.append('/s3/<BUCKET_NAME>')
Run the run
function defined in the main.py
module.
import main
main.run()