Super simple docker cheatsheet.
First build the image:
docker build -t <desired_image_name> .
Example from when I built a container having a TCP/IP client script and TCP/IP server script, so I wanted a generic name:
docker build -t tcpnode .
This may take a while the first time.
Once the image is built it can be run:
docker run -it -p 5000:5000 --name myserver tcpnode
-p exposes docker client port to the host
-it runs in interactive terminal mode
-d runs in detached (headless) mode, manage w/ ps -a, start/stop/kill etc
-v mounts a volume for shared storage
using COPY/ADD in the Dockerfile only copies the folder contents to the container
using -v actually mounts the specified folder as a folder in the container
so writes in the host can be seen immediately in the container and vice-versa
this can be useful for writing code in the host but running tests/devops/etc in a container
--name gives this running instance a name
so e.g. we could run myserver 1, myserver 2, etc all based on the tcpserver base image
In that example, because it is run with -it
we can run arbitrary commands in it from the command prompt.
To run a script/program automatically when the container starts we can use a CMD
command in the Dockerfile.
Alternately, if we want to run multiple containers, each based on the same image but each running a different script, we do not have to create a separate Dockerfile for each.
Instead we can pass the script/app startup command as a parameter:
docker run -it --name myserver tcpnode python tcpServer.py
docker run -it --name myclient tcpnode python tcpClient.py
That starts up two containers, one for the client and one for the server, both from the same tcpnode
image built previously.
Once the image is no longer needed, shut it down (logout, stop/kill, etc) and then remove it:
docker ps -a # if we want to see what is running before we kill it
docker rm tcpnode
More on docker run
here: https://blog.codeship.com/the-basics-of-the-docker-run-command/
Put both apt-get update
and apt-get install
in the SAME RUN
COMMAND.
RUN apt-get update && apt-get install -y \
bzr \
cvs \
git \
mercurial \
subversion
This is done to make sure that the latest packages will be installed. If apt-get install were in a separate RUN instruction, then it would reuse a layer added by apt-get update, which could had been created a long time ago.
RUN
creates and commits new layer each time it executes.
http://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/
- Use
RUN
to alter the image by creating and commiting a new layer - ex: runningapt-get
to configure the container. - Use
CMD
to set a default command which can be overridden by passing a command todocker run
on the command line.- Only the last
CMD
in the Dockerfile is actually executed, and only if there is no command passed todocker run
.
- Only the last
- Use
ENTRYPOINT
to configure the container to run as an executable.ENTRYPOINT
commands are not ignored when a command is passed todocker run
-- unlikeCMD
- Prefer
ENTRYPOINT
toCMD
when building executable Docker image and you need a command always to be executed.
There are two forms for the commands: Shell form and Exec form.
Shell form:
<instruction> <command>
RUN apt-get install python3
CMD echo "Hello world"
ENTRYPOINT echo "Hello world"
Shell form executes /bin/sh -c <command>
behind the scenes, so the RUN
command above is actually:
/bin/sh -c apt-get install python3
To use environment variables in a script run from within a Dockerfile, use shell form.
Exec form:
<instruction> ["executable", "param1", "param2", ...]
RUN ["apt-get", "install", "python3"]
CMD ["/bin/echo", "Hello world"]
ENTRYPOINT ["/bin/echo", "Hello world"]
Shell processing does not occur in exec form. So for example environment variables are not accessible from scripts run from within the Dockerfile.
Shell form is preferred for CMD
and ENTRYPOINT
.
Either form is ok for RUN
.
To run /bin/bash from within a Dockerfile: Use the exec form.
ENV name John Dow
ENTRYPOINT ["/bin/bash", "-c", "echo Hello, $name"]
From the takacsmark tutorial this technique can be helpful for testing a web app where we haven't yet created a gui interface. His example is a Flask app container that stores data in a Redis data store container.
# add a name to the data store
$ curl --header "Content-Type: application/json" \
--request POST \
--data '{"name":"Kumar"}' \
localhost:5000
{
"name": "Kumar"
}
# retrieve list of names from the data store
$ curl localhost:5000
[
"{'name': 'Kumar'}"
]
This is a quick and easy way to validate the two containers are connected without having to build a gui.
To see what images are installed locally:
docker image ls
To remove a local image:
docker image rm imagename:tagname
After a while a bunch of anonymous images may be in the list. To remove all unused images: (will show warning)
docker image prune
To view all current networks:
docker network ls
For more details:
docker network inspect <network_name>
ex: docker network inspect bridge
inspect
returns a JSON object with the configuration details.
To specify which network a container should use when we run it:
docker run <container_name> --net=<network_name>
Good resources:
- https://runnable.com/docker/basic-docker-networking
- https://mesosphere.com/blog/networking-docker-containers/ (excellent diagrams!)
- https://docs.docker.com/docker-for-mac/networking/
(quotes come from a variety of sources including the above)
(src: https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds)
- We cannot see the
docker0
interface on the Docker for Mac host as it is contained entirely in the virtual machine running silently in the background. - Docker for Mac cannot route traffic from the host to the containers, so we can't
ping
a container.
To connect FROM a container TO a service on the Mac host:
The host has a changing IP address (or none if you have no network access). From 18.03 onwards our recommendation is to connect to the special DNS name
host.docker.internal
, which resolves to the internal IP address used by the host. This is for development purpose and will not work in a production environment outside of Docker for Mac.The gateway is also reachable as
gateway.docker.internal
.
Remember: gateway == network border router.
To connect TO a container FROM the Mac host:
Port forwarding works for localhost; --publish, -p, or -P all work. Ports exposed from Linux are forwarded to the host.
Our current recommendation is to publish a port, or to connect from another container. This is what you need to do even on Linux if the container is on an overlay network, not a bridge network, as these are not routed.
Important: Per the takacsmark tutorial and video below, containers in the same network can refer to each other using the service name (specified in docker-compose.yml
) as host names. It can be useful when experimenting with new services to find out what it expects other services to be named by default and use that default as the corresponding service name. In his example, he checked inside the Wordpress code and instructions to discover that the default name for the backend MySQL database is mysql
so he gave the service the name mysql
in the docker-compose.yml
file. That way it just works out of the box which is what we want for experimentation.
Docker can manage multiple networks which is useful when spinning up containers we want to network together. Each container can be assigned to a network, so we can have multiple containers running on different networks on the same host.
Docker creates 3 networks automatically: bridge, host, and none.
Note: It is best to create a custom network instead of using one of the builtins.
This is the default network and the simplest to use. All containers connect here unless explicitly connected to another network.
Docker automatically creates a subnet and gateway for the bridge network, and docker run automatically adds containers to it. Any containers on the same network may communicate with one another via IP addresses (ONLY). Docker does not support automatic service discovery on bridge. You must connect containers together with the
--link
option in yourdocker run
command.The Docker bridge supports port mappings and docker run
--link
allowing communications between containers on thedocker0
(en0
etc) network. However, these error-prone techniques require unnecessary complexity. Just because you can use them, does not mean you should. It’s better to define your own networks instead.
Note: I read elsewhere that creating our own network is the best practice for a litany of reasons, including the fact that adopting the basic bridge network requires exposing all ports to the world or none. A custom network allows us to expose only what we want. (so creating a custom network is similar to creating a firewall?)
This offers a container-specific network stack that lacks a network interface. This container only has a local loopback interface (i.e., no external network interface).
So in other words a container using the none
network has no network connectivity with any other container nor with the host.
This enables a container to attach to your host’s network (meaning the configuration inside the container matches the configuration outside the container).
This is essentially allowing the container to run on our host's network. So running a container here should make it reachable from other computers on our network.
You can create multiple networks with Docker and add containers to one or more networks.
So a single container can be hooked to multiple networks, this is important to understand because:
Containers can communicate within networks but not across networks.
So to have two containers communicate they must be attached to the same network.
A container with attachments to multiple networks can connect with all of the containers on all of those networks.
This part is interesting:
This lets you build a “hub” of sorts to connect to multiple networks and separate concerns.
To forward a host port to a container port:
docker run -p host_port_num:container_port_num ...
So to run a container having a web server with connections from the host on port 8000 forwarded to port 80 on the container:
docker run -p 8000:80 ...
Building a new network based on the default bridge
network is the easiest way to set up an isolated network for a set of containers.
To create a custom network based on the default bridge
network type:
docker network create --driver bridge my_network
To view the network details:
docker network inspect my_network
Inspecting will also show the containers that are currently wired into the network.
There is a command that can attach/detach containers to a network after the containers are started but I can't recall offhand what that is.
All containers in the same docker network can reference each other using service names as if they were machine names.
For example from the takacsmark tutorial there is a redis
service in the docker-compose.yml
file, and the app.py
app points to the redis "server" by using this line:
redis = Redis(host="**redis**", db=0, socket_timeout=5, charset="utf-8", decode_responses=True)
If we rename the service in the docker-compose.yml
file we must rename it in all references in the app as well.
Docs: https://docs.docker.com/compose/compose-file/
Orchestrates a cluster of containers.
- Best Tutorial: https://takacsmark.com/docker-compose-tutorial-beginners-by-example-basics/
- His video walkthrough: https://www.youtube.com/watch?v=4EqysCR3mjo
- Docker Compose file reference: https://docs.docker.com/compose/compose-file/
- Important: This is for both Docker Compose and Docker Swarm! Some features here (e.g.
secret
,deploy
) only apply to Docker Swarm while others (e.g.build
) only apply to Docker Compose. (Swarm only accepts pre-built images) The documentation includes both but not everything is applicable to both.
- Important: This is for both Docker Compose and Docker Swarm! Some features here (e.g.
Limitation: Docker Compose only composes clusters on a single machine. It cannot manage containers across multiple machines. To manage containers across machines use Docker Swarm or Kubernetes. Docker Swarm uses the same docker-compose.yml
file, so we can develop locally with Docker Compose and deploy to a cloud environment with Docker Swarm.
docker-compose.yml
describes the desired state, but during execution the system's actual state may deviate.
Built into the system is an ability to bring the entire stack back into the desired state:
docker-compose up
or for detached mode of course:
docker-compose up -d
- if an image is already running and if it is in the desired state then it is not touched
- if an image is already running and has deviated from the desired state then it is destroyed and recreated
- if an image from the
docker-compose.yml
file does not exist in the stack it is created
Many commands are very similar to their counterparts in docker
except they work across the entire stack unless a specific service is named in the command.
Use docker-compose up
and docker-compose down
to spin up the stack and to teardown and destroy it, respectively. It can setup and teardown containers, networks, images, and volumes.
Use docker-compose ps
to view a list of containers (started & stopped) in our stack, the commands they are running, and their port mappings.
Use docker-compose top
to list the top process for each service in the stack.
Use docker-compose logs -f
to follow the logs from both services.
Use docker-compose logs -f <service_name>
to follow the logs from only one service.
Use docker-compose stop
to stop all containers, docker-compose start
to start them again, or docker-compose restart
to do both together.
Use docker-compose exec <service_name> <command>
similar to docker exec
, to execute <command>
against the running service.
There are other useful commands such as kill
, rm
, etc.
docker-compose down
will tear down the entire stack and dispose of them.
docker-compose up --build
will do a rebuild first if the Dockerfile or image have changed before spinning the stack up.
docker-compose build --no-cache
will force a rebuild regardless of change status, and we can follow it up with a normal docker-compose up -d
command.
We can define networks and volumes in the docker-compose.yml
file using the networks
and volumes
keywords.
Using named volumes gives us the flexibility of starting and destroying containers that use persistent data volumes, so we can share state between containers (e.g. in a DevOps pipeline) or between instances of the same container image. (e.g. spawning multiple Redis data stores to store state, with state shared via persistent named volume)
exec
runs the command in a running containerrun
starts up a completely new container to run the command
docker-compose exec <service_name> /bin/bash
To enter a shell in a new container based on a service (from docker-compose.yml
) use docker-compose run <service_name> /bin/bash
.
Use the build
section under the service definition in docker-compose.yml
. It defines the build context
, the build args
, the dockerfile
location and even the target
build stage in a multi-stage build.
Docker Compose can automate docker run
because we can specify runtime paraneters in a separate Config file.
Use the args
element under build
. Arguments defined in args
are passed to the Dockerfile. Inside the Dockerfile the argument must be defined in the ARG
instruction, after which it can be used by prepending a dollar sign.
This makes the Dockerfile more generic and allows us to drive the configuration from the docker-compose.yml
file.
Ex from takacsmark:
**In docker-compose.yml**
...
services:
app:
build:
context: .
dockerfile: Dockerfile
args:
- PYTHON_VERSION=3.7.0-alpine3.8
...
**In Dockerfile**
ARG PYTHON_VERSION
FROM python:$PYTHON_VERSION
It is of course a bad idea to put sensitive info in the Dockerfile or docker-compose.yml
file.
The better way is to point docker-compose.yml
to a file containing the environment variable values, that way we can store them separately and securely.
Use the env_file
directive inside of a service definition in docker-compose.yml
to specify the name of the file to use:
env_file:
- .env.txt
This further abstracts out the variables from the docker-compose.yml
just like putting them in docker-compose.yml
abstracts them out from the Dockerfile.
Likewise if we create a file named .env
in the same folder as docker-compose.yml
it should automatically be picked up by Docker Compose and can be used to define variables within docker-compose.yml
itself.
Example:
(in docker-compose.yml)
services:
app:
build:
context: .
dockerfile: Dockerfile
args:
- PYTHON_VERSION=${PYTHON_VERSION} # notice the braces!
...
redis:
image: redis:${REDIS_VERSION}
And the corresponding .env
file:
PYTHON_VERSION=3.7.0-alpine3.8
REDIS_VERSION=4.0.11-alpine
Now when docker-compose up
is run, the environment variables will be read from the .env
file, preprocessed into docker-compose.yml
, and then docker-compose.yml
will be used to orchestrate the spinup of the stack. This means the variables defined in .env
will be passed through to the Dockerfile, and the resulting containers will be configured as specified in the .env
file.
To run N copies of a single container:
docker-compose up --scale <service_name>=N
Ex: docker-compose up --scale app=3 # for service named 'app' in docker-compose.yml
Note: If we specified a custom container_name
then we cannot scale up past one container!
Note: If we map ports in a Dockerfile we will get errors on the above because we can't spawn multiple copies that map the same port.
This can be useful to have one service that listens on a port and then have multiple worker containers in the background. The frontend container listening on the port then manages distributing the workload among the worker containers as needed.
For example, we have a web app container that allows a user to upload an image to be resized, and it passes off the image to a worker container behind the scenes.
To increase from 3 worker containers (based on a single service named worker
in docker-compose.yml
) to 6 in response to increased load we can do this:
docker-compose up --scale worker=6
Then Docker Compose will spin up 6 workers.
Presumably we need a means to register the workers with the web service so the service can hand off work to the new workers, and a way to monitor worker load.
Specify service dependencies with the depends_on
directive in docker-compose.yml
and those service containers will be started in the specified order before the enclosing service.
Note: Docker does not guarantee the dependency services are functional and ready to be used only that the containers have started.
Put a file named docker-compose.override.yml
in the same folder as docker-compose.yml
.
When we run docker-compose up
, any directives in docker-compose.override.yml
will override their corresponding directives in docker-compose.yml
.
Caveat: items in YAML lists are appended, not necessarily overridden.
We can stack files to set up various environments, e.g. docker-compose.dev.yml
, docker-compose.test.yml
etc.
If we have multiple docker-compose.yml
files we can opt to use only one of them during a stack deployment:
docker-compose -f docker-compose.yml up -d
We can apply multiple files in sequence to configure our stack precisely to our environment:
docker-compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.dev.yml up -d
This will spin up the stack by running each of the files in sequence to give us our desired dev environment.
We can't orchestrate a stack across multiple computers, but we can copy the stack to other computers.
Use docker-compose push
to push the stack images to the selected central repository (e.g. Docker Hub) and docker-compose pull
to pull it down to another computer. Then use docker-compose up
etc to run it on the other computer.