We setup a dev env for DC/OS in AWS (subnets, multi-az, ags groups, ami images, etc), tagged everything as dcos-dev
, and then used Cloudformer to generate a starter AWS CloudFormation script. Cloudformer allows you to reverse engineer your AWS environment into cloudformation scripts. We then modified what cloudformer produced (to make it work as cloudformer just gets you about 90% of the way there), and then we added mappings, parameters and outputs to our cloudformation script.
Included are the cloudformation and packer scripts. Hope it helps you get setup. Feedback is welcome.
We in this case is my client, DC/OS support, Amazon support, and I. We did this instead of using the canned Amazon support because we needed to run masters, and agents in multiple AZs per region.
Production servers are setup, and integration servers are also setup. It is all immutable infrastructure (for the most part). We have scripts that build a DC/OS cluster from nothing to the full network, security, ELB, auto-scale groups for public agents & private agents, master instances, user-data etc. stack using AWS cloud formation.
The two parts that are not yet automated are the enterprise marathon-lb
setup (you have to fire up a micro into bastion security group, install DCOS cli and then run one script which is checked into git), and the CNAME/DNS bits which are controlled by IT. We might install an NS record to delegate
to AWS Route53 and then control virtual host setup with Route53 aliases, which means that we can automate via cloud formation. Then we might add a Lambda/SNS/event to our cloud formation to handle the marathon-lb
install when the cloud formation is done running.
All the AMI image creation is automated as well. The cloud formations point to AMI images which we create using Packr that have all of the advance prequisites for DC/OS. Packr has excellent support for Amazon AMIs as well as Docker, and just about every other image container that exists. We choose CentOS over Amazon Linux so we are less locked in and could move images to other clouds or containers more readily.
The cloudformation script has a drop down cloudformation parameter which picks which env (prod, int, dev) and it sets up all of the CIDR addresses etc. via cloudformation mappings.
The DC/OS advanced install tar files and config were uploaded to S3. We use IAM roles, and s3 bucket policies to lock down the S3 bucket where the DC/OS advanced install params are. (We run sudo bash dcos_generate_config.ee.sh
and aws s3 sync genconf/serve/ s3://somebucket-dcos-install-bootstrap/prod
to upload the install packages from the DC/OS advanced install.)
The cloudformation script supports multiple AWS regions and multiple AZs. To do this support we use cloudformation mappings
, parameters
and outputs
.
We plan on improving the cloudformation scripts and packer script as follows:
- Adding placements groups,
- Splitting out the networking cloudformation parts from the ags, security groups, etc. cloudformation parts.
- Adding aws cloudtrail agents to the base images
- Firing off lambda to setup VPC peering to base corporate account (something cloudformation does not support)
- Remove public access to pub agents and masters (only accessible via ELB with SSL termination)
- Remove extra ports from secruity groups that we do not need
- Clean-up