Microk8s is a Canonical project to provide a kubernetes environment for local development, similar to minikube but without requiring a separate VM to manage. These instructions describe setting it up for common development use cases with Cilium and may be helpful in particular for testing BPF kernel extensions with Cilium.
Microk8s will run its own containerd runtime, which may be initially confusing when building containers locally with docker. This guide assumes that you will use docker locally for building containers, and push these into a microk8s registry for use by containerd in the microk8s environment.
This guide works with MicroK8s 1.14 or above, with containerd. If you are running an earlier version, see the previous instructions.
- Linux with kernel 4.9 or newer (Full Cilium Requirements)
- Snap (default installed in recent Ubuntu distros)
- docker-ce for new docker client binary, used in local image build.
In this howto setup was run on packet.net c1.small.x86 node with Ubuntu 17.10.
Quick howto:
# apt-get install snapd apt-transport-https ca-certificates curl software-properties-common build-essential flex bison clang llvm libelf-dev libssl-dev libcap-dev gcc-multilib libncurses5-dev pkg-config
# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
# add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# apt-get update
# apt-get install docker-ce
# wget https://dl.google.com/go/go1.13.8.linux-amd64.tar.gz
# tar xvf go1.13.8.linux-amd64.tar.gz -C /usr/local/
# mkdir -p ~/go/src/github.com/cilium/
And add to bashrc:
export GOPATH=/<home>/go/
export GOROOT=/usr/local/go/
export PATH=/snap/bin/:/usr/local/go/bin/:/root/go/bin/:$PATH
Then follow with building Cilium itself:
# cd ~/go/src/github.com/cilium/
# git clone https://github.com/cilium/cilium.git && cd cilium/
# go get -u github.com/gordonklaus/ineffassign
# go get -u github.com/jteeuwen/go-bindata/...
# SKIP_DOCS=true make
The official documentation has a page for Getting Started Using MicroK8s. Follow those instructions to configure your local environment with microk8s for use with Cilium.
This allows to run microk8s
commands without sudo.
sudo usermod -a -G microk8s $USER
MicroK8s itself includes a kubectl
binary, which can be accessed by running the microk8s.kubectl
command.
While using the prefix microk8s.kubectl
allows for a parallel install of another system-wide kubectl
without impact, you can easily get rid of it by using the snap alias command:
sudo snap alias microk8s.kubectl kubectl
See https://kubernetes.io/blog/2019/11/26/running-kubernetes-locally-on-linux-with-microk8s/ for details.
# microk8s.enable dns registry
- The registry is available on localhost:32000 (via NodePort).
-
Make your local changes to your Cilium repository.
-
Build and deploy the container image
# make microk8s
This uses your local docker-ce (and docker daemon hosted at
/var/run/docker.sock
) to push into the registry that was configured above. It wil pre-pull the custom container image atcilium/cilium:local
(orcilium/cilium:LOCAL_IMAGE_TAG
ifLOCAL_IMAGE_TAG
is set to something other thanlocal
).
If you have trouble with the above steps, check the Troubleshooting section.
Per the instructions from the output of make microk8s
, you just need to set the container image in your Cilium DS YAML (or delete the pods to pull a fresh version), then observe that the rollout is successful:
# kubectl -n kube-system set image ds/cilium cilium-agent=localhost:32000/cilium/cilium:local
# kubectl -n kube-system rollout status ds cilium
If the tag is already pointing to your custom image, you should just need to delete the pods:
# kubectl -n kube-system delete po -l k8s-app=cilium
# kubectl -n kube-system rollout status ds cilium
If the rollout gets stuck it can be debugged through ...
# kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
[...]
kube-system cilium-hbcs8 0/1 ErrImagePull 0 75s 147.75.80.23 test <none>
[...]
# kubectl describe pod -n kube-system cilium-hbcs8
[...]
Warning Failed 30s kubelet, test Failed to pull image "localhost:32000/cilium/cilium:my-image": rpc error: code = Unknown desc = Error while pulling image: Get http://localhost:32000/v1/repositories/cilium/cilium/images: read tcp localhost:53302->127.0.0.1:32000: read: connection reset by peer
Normal BackOff 4s (x4 over 103s) kubelet, test Back-off pulling image "localhost:32000/cilium/cilium:my-image"
Warning Failed 4s (x4 over 103s) kubelet, test Error: ImagePullBackOff
[...]
... e.g. in this case the imagePullPolicy
was probably set to Always
.
The daemon set updates are undone via:
# kubectl -n kube-system rollout undo ds cilium
Check if Cilium is up and running:
# kubectl get pods --all-namespaces -o wide
# kubectl -n kube-system logs --timestamps cilium-1234
Deploying a sample application for testing Cilium w/o policy first:
# kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/minikube/http-sw-app.yaml
# kubectl exec -it -n kube-system cilium-1234 -- cilium endpoint list
# kubectl exec -it tiefighter -- netperf -t TCP_STREAM -H 10.23.177.124
[...]
Force endpoint regeneration:
# kubectl delete po tiefighter
First, add the cilium.io/ci-node=k8s1
labels to your node:
$ kubectl edit node $(hostname)
Under .metadata.labels
, add the label from the above.
Run specific ginkgo tests (in this case MonitorAggregation
) against the microk8s environment from the test/
directory:
$ CNI_INTEGRATION=microk8s CILIUM_IMAGE="localhost:32000/cilium/cilium:local" CILIUM_OPERATOR_IMAGE="docker.io/cilium/operator:latest" K8S_VERSION=1.16 ginkgo -v -focus="MonitorAggregation*" -- -cilium.provision=false -cilium.kubeconfig=/home/joe/.kube/config -cilium.passCLIEnvironment=true -cilium.testScope=k8s -cilium.holdEnvironment=true -cilium.skipLogs=true
The options:
CNI_INTEGRATION
is required to properly configure the Cilium YAMLs for microk8sCILIUM_IMAGE
: Depends on an image built and pushed viamake microk8s
CILIUM_OPERATOR_IMAGE
: This is using upstream master operator; if doing operator development, you can manually bulid / push / set this operator image differently.-focus
: Specify which tests to run-cilium.provision=false
: Disable the k8s cluster provisioning-cilium.kubeconfig
: You will need to update this to point to your own kubeconfig.-cilium.passCLIEnvironment=true
: Necessary to run CI tests in custom cluster (like microk8s)-cilium.testScope=k8s
: Only run the tests in the kubernetes scope-cilium.holdEnvironment=true
: If the test fails, stop everything so the developer can debug the environment in the failed state-cilium.skipLogs=true
: Disable log-gathering at the end of a failed test.
When the test is finished, this will remove cilium from your microk8s environment (as of October 2019). If this bothers you, please send a patch upstream to avoid this.
Some CI tests are written to run against multiple nodes. As of October 2019, these will not successfully pass on microk8s.
General microk8s troubleshooting steps may reveal the problem. A few common issues are also documented below.
Ensure that the docker socket has been updated in the Cilium-DS YAML:
https://gist.github.com/joestringer/60a5f53d59e57274ed4c2a1736a7b101#set-up-cilium
In case there is a change in host IP, you can restart kubernetes API server the following way in order to propagte the new IP to all kubernetes cluster members:
# microk8s.stop
# microk8s.start
The following error may occur when IP connectivity to the registry is not available:
# docker push localhost:32000/cilium/cilium:my-image
The push refers to repository [localhost:32000/cilium/cilium]
Get http://localhost:32000/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
This may occur for multiple reasons:
- Your exernal IP has changed (for instance, your laptop was suspended and restored on a new network)
- For this case, you can follow the instructions above to Restart μK8s.
- Your localhost attempts to connect to the registry via IPv6
- The docker daemon is not listening on IPv4, so if the docker destination of
localhost:32000
resolves to IPv6 this may cause timeouts. To overcome this, use127.0.0.1
instead oflocalhost
, or remove the IPv6 (::1
) host alias from/etc/hosts
.
- The docker daemon is not listening on IPv4, so if the docker destination of
When deploying cilium daemon set for the first time, the following error may occur. Make sure to have --allow-privileged
option set in kube-apiserver and apiserver reloaded:
# kubectl create -n kube-system -f https://raw.githubusercontent.com/cilium/cilium/1.3.0/examples/kubernetes/addons/etcd/standalone-etcd.yaml
[...]
The DaemonSet "cilium" is invalid:
* spec.template.spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
* spec.template.spec.initContainers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
Some quick-links for general troubleshooting: