I use these ansible scripts (see also) to bring up kubernetes clusters, on centos and fedora. The scripts are awesome, but they put selinux into permissive mode, because, "tokens and selinux don't work together."
That's unfortunate, because one of the main things I'm doing when I turn up a kubernetes cluster is bug-hunting, and selinux can be a source of bugs.
I commented out the bit that put selinux into permissive mode:
#- name: Set selinux permissive because tokens and selinux don't work together
# selinux: state=permissive policy={{ ansible_selinux.type }}
# when: ansible_selinux is defined and ansible_selinux.status == "enabled"
I then brought up a cluster, and found that most/all of the kubernetes-addons
refused to work.
Someone asked a question about this on Stack Overflow, and the answer provided was simple enough:
You probably need to run the following command to set the SELinux context correctly in the volumes directory. I have an open issue to make this happen automatically in the future:
sudo chcon -Rt svirt_sandbox_file_t /var/lib/kubelet
To implement this suggested fix, I added this bit to roles/node/tasks/main.yml
:
- name: chcon of kubelet rather than set selinux permissive
command: chcon -Rt svirt_sandbox_file_t /var/lib/kubelet
when: ansible_selinux is defined and ansible_selinux.status == "enabled"
Following the change, the kube-addons pods all seemed to work, with the exception of the fluentd-elasticsearch
pod, which still had selinux denials.
Dan Walsh wrote a blog post about selinux and the fluentd container in which he suggested a policy fix that partly worked (the fluentd container would run, but it couldn't actually read the logs it was supposed to read). A commenter on that post suggested an additional line to add to the policy that made the container work. Here's the policy:
policy_module(container_logger, 1.0)
virt_sandbox_domain_template(container_logger)
##############################
# virt_sandbox_net_domain(container_logger_t)
gen_require(`
attribute sandbox_net_domain;
')
typeattribute container_logger_t sandbox_net_domain;
##############################
logging_manage_all_logs(container_logger_t)
docker_read_lib_files(container_logger_t)
To use the policy, paste the above into a file called container_logger.te
, and then, on your centos or fedora (swap dnf
in for yum
) host:
$ sudo yum install selinux-policy-devel
$ make -f /usr/share/selinux/devel/Makefile container_logger.pp
$ sudo semodule -i container_logger.pp
This ought to do it for the container, but for the container+kubernetes case, we need to tell kubernetes to run the fluentd container with the appropriate selinux type. Here's what the yaml for that should look like:
securityContext:
seLinuxOptions:
type: container_logger_t
Here it is again, sandwiched into the otherwise-stock yaml from upstream:
apiVersion: v1
kind: Pod
metadata:
name: fluentd-elasticsearch
namespace: kube-system
labels:
k8s-app: fluentd-logging
spec:
containers:
- name: fluentd-elasticsearch
image: gcr.io/google_containers/fluentd-elasticsearch:1.15
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
securityContext:
seLinuxOptions:
type: container_logger_t
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
In order for this to work, make sure that there's no SecurityContextDeny
in the KUBE_ADMISSION_CONTROL=
line of the file /etc/kubernetes/apiserver
in your kube master.
With the ansible changes and the policy applied to my nodes, and with the modded fluentd yaml, the kube-addons work.
Is this the best/right way to accomplish this?
At this point... I think so? There's some worthwhile discussion in this github issue. In any case, this is better than no selinux at all, which is the status quo for a kube cluster w/ kube-addons running, AFAICT.
That extra line added to the policy that Dan Walsh blogged about, is that legit?
It makes the container work, otherwise... I don't know. The line is the last one in the policy above, docker_read_lib_files(container_logger_t)
How do you compile selinux policy on atomic, where you don't have selinux-policy-devel
installed?
With a container, I figure. I'll work out / write down those steps. I hope soon to see this policy ship w/ centos and fedora, so there'd be no policy building required.
Can't this Just Work?
I'm going to get this into a PR and into the kubernetes/contrib repo. We'll need to modify the yaml for the fluentd container on the fly or get upstream to accept some selinux-foo into the upstream yaml, and we'll need to figure out how to deal with the policy building bit. I'm not sure if what a non-selinux-supporting kube cluster makes of seLinuxOptions
stuff -- it'd make sense just to ignore it, but I don't know if that's how it works.