We build a large number of images in CI and rely on images built with buildkit
that contain inline cache. Recently, after switching from using buildctl build
directly to using docker build
with DOCKER_BUILDKIT=1
exported and
the BUILDKIT_INLINE_CACHE=1
build arg, we've noticed that cache works as
expected only up until a COPY
instruction, as shown with steps to reproduce
below.
Build an image with inline cache, push it to Docker Hub, clear out everything locally, and show that rebuilding from cache works correctly.
$ docker version
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:21:11 2020
OS/Arch: darwin/amd64
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:29:16 2020
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
$ docker info
Client:
Debug Mode: false
Plugins:
app: Docker Application (Docker Inc., v0.8.0)
buildx: Build with BuildKit (Docker Inc., 0.0.0+unknown)
Server:
Containers: 25
Running: 24
Paused: 0
Stopped: 1
Images: 14
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.76-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.945GiB
Name: docker-desktop
ID: MSI5:RISV:MJPQ:UCVR:I5VT:HXAA:3KSL:GMYJ:PTPE:XCOW:LX4I:J2IC
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 171
Goroutines: 147
System Time: 2020-05-27T14:58:47.3048602Z
EventsListeners: 4
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
localhost:5000
0.0.0.0:5000
192.168.1.72:5000
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
$ cat <<EOF > Dockerfile
FROM alpine
RUN echo foo
COPY bar bar
RUN echo baz
EOF
$ echo bar > bar
$ DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --tag vanstee/test .
[+] Building 3.4s (10/10) FINISHED
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 91B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.5s
=> [1/4] FROM docker.io/library/alpine@sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb 1.0s
=> => resolve docker.io/library/alpine@sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb 0.0s
=> => sha256:39eda93d15866957feaee28f8fc5adb545276a64147445c64992ef69804dbf01 528B / 528B 0.0s
=> => sha256:f70734b6a266dcb5f44c383274821207885b549b75c8e119404917a61335981a 1.51kB / 1.51kB 0.0s
=> => sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 2.81MB / 2.81MB 0.4s
=> => sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb9a54 1.64kB / 1.64kB 0.0s
=> => extracting sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 0.3s
=> [internal] load build context 0.0s
=> => transferring context: 34B 0.0s
=> [2/4] RUN echo foo 0.7s
=> [3/4] COPY bar bar 0.1s
=> [4/4] RUN echo baz 0.6s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:7334553d104b0b485f1dc613d4f5005917ec93864fb64f2f2fc7371851f764d0 0.0s
=> => naming to docker.io/vanstee/test 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export 0.0s
$ docker push vanstee/test
The push refers to repository [docker.io/vanstee/test]
5f70bf18a086: Layer already exists
9a178b6109c6: Pushed
c338759da1a5: Pushed
3e207b409db3: Layer already exists
latest: digest: sha256:151842699becdaaaa461c7c64d4b6ac38e88e8799263de5d7638e38c52165327 size: 1148
$ docker system prune -a -f > /dev/null
$ DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from vanstee/test --tag vanstee/test .
[+] Building 2.7s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 91B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.5s
=> importing cache manifest from vanstee/test 0.4s
=> [1/4] FROM docker.io/library/alpine@sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 34B 0.0s
=> CACHED [2/4] RUN echo foo 0.0s
=> CACHED [3/4] COPY bar bar 0.0s
=> CACHED [4/4] RUN echo baz 1.6s
=> => pulling sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 0.3s
=> => pulling sha256:a6a654dae9dc15ecd273df6c7c8149e60f8a2f2f2374a5e1ea0b7316359c949d 0.3s
=> => pulling sha256:868e8a0ced0ebe5a970c6767909c0e06189a2f0ab908bf657d04fc58da06495d 0.3s
=> => pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.1s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:7334553d104b0b485f1dc613d4f5005917ec93864fb64f2f2fc7371851f764d0 0.0s
=> => naming to docker.io/vanstee/test 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export 0.0s
Build the same image using the cache from the previoulsy built image and show that everything works fine.
# docker version
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:25:46 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:24:19 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
# docker info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-58-generic
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 985.5MiB
Name: vagrant
ID: QDUF:7BKT:LZVN:NQDV:R2ZJ:NN7D:NN7H:2OLM:6EN2:2VHV:BJ7C:WA5N
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
# cat <<EOF > Dockerfile
FROM alpine
RUN echo foo
COPY bar bar
RUN echo baz
EOF
# echo bar > bar
# DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from vanstee/test --tag vanstee/test .
[+] Building 5.2s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 88B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 3.1s
=> importing cache manifest from vanstee/test 0.6s
=> [1/4] FROM docker.io/library/alpine@sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 34B 0.0s
=> CACHED [2/4] RUN echo foo 0.0s
=> CACHED [3/4] COPY bar bar 0.0s
=> CACHED [4/4] RUN echo baz 1.3s
=> => pulling sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 0.4s
=> => pulling sha256:868e8a0ced0ebe5a970c6767909c0e06189a2f0ab908bf657d04fc58da06495d 0.1s
=> => pulling sha256:a6a654dae9dc15ecd273df6c7c8149e60f8a2f2f2374a5e1ea0b7316359c949d 0.2s
=> => pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.1s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:7334553d104b0b485f1dc613d4f5005917ec93864fb64f2f2fc7371851f764d0 0.0s
=> => naming to docker.io/vanstee/test 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export 0.0s
Build the same image using the cache from the previoulsy built image and show that only the first instruction is cached.
# docker version
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:27:04 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:25:42 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
# docker info
Client:
Debug Mode: false
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 2
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-1127.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 487MiB
Name: localhost.localdomain
ID: ZITA:KMKP:5JI7:OQPA:IHJ3:RKRT:IKMR:GOVV:77AO:UIH5:KYSF:3RXQ
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
# cat <<EOF > Dockerfile
FROM alpine
RUN echo foo
COPY bar bar
RUN echo baz
EOF
# echo bar > bar
# DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from vanstee/test --tag vanstee/test .
[+] Building 2.8s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 146B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 1.0s
=> importing cache manifest from vanstee/test 0.4s
=> [1/4] FROM docker.io/library/alpine@sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 92B 0.0s
=> CACHED [2/4] RUN echo foo 0.7s
=> => pulling sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 0.2s
=> => pulling sha256:868e8a0ced0ebe5a970c6767909c0e06189a2f0ab908bf657d04fc58da06495d 0.1s
=> [3/4] COPY bar bar 0.0s
=> [4/4] RUN echo baz 0.5s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:61a1aa8f6d5c8291e80f47aceacee8bd9322c815b0160e3bc22ba18f735e3085 0.0s
=> => naming to docker.io/vanstee/test 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export 0.0s
Notice that only the first instruction is cached, and once the COPY
instruction is run the cache is no longer used. We were able to replicate this
every time a COPY
instruction was introduced when building on CentOS.