Azure AKS BYOCNI Cluster with Cilium

Cluster

Prerequisites

az extension add --name aks-preview
az extension update --name aks-preview

az feature register --namespace "Microsoft.ContainerService" --name "KubeProxyConfigurationPreview"
az provider register --namespace "Microsoft.ContainerService"

Cluster Creation

set -gx CLUSTERNAME "test-philip"
set -gx RESOURCEGROUP "test-philip"
set -gx LOCATION "eastus2"

az group create --name $RESOURCEGROUP --location $LOCATION

echo >kube-proxy.json '{
  "enabled": false,
  "mode": "IPVS",
  "ipvsConfig": {
      "scheduler": "LeastConnection",
      "TCPTimeoutSeconds": 900,
      "TCPFINTimeoutSeconds": 120,
      "UDPTimeoutSeconds": 300
  }
}'

az aks create --name $CLUSTERNAME --resource-group $RESOURCEGROUP \
    --network-plugin none \
    --kube-proxy-config kube-proxy.json \
    --kubernetes-version 1.29

Cluster Access

az aks get-credentials -g $RESOURCEGROUP --name $CLUSTERNAME

kubectl get nodes -o wide

Cilium Installation

Prerequisites

Add Helm repo:

helm repo add cilium https://helm.cilium.io/
helm repo update cilium

Install Gateway API CRDs (optional):

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml

Prepare Cilium Helm values

echo >cilium-values.yaml "\
aksbyocni:
  enabled: true

kubeProxyReplacement: \"true\"
k8sServiceHost: $(kubectl config view --minify --output jsonpath="{.clusters[*].cluster.server}" | awk -F[/:] '{print $4}')
k8sServicePort: $(kubectl config view --minify --output jsonpath="{.clusters[*].cluster.server}" | awk -F[/:] '{print $5}')

hubble:
  relay:
    enabled: true

ingressController:
  enabled: true
  loadbalancerMode: shared
gatewayAPI:
  enabled: true"

Cilium installation

helm upgrade -i cilium cilium/cilium \
  --version 1.16.0 \
  -n kube-system \
  -f cilium-values.yaml

Verification

Download the Cilium CLI from here.

$ cilium status                                                                                                                                          641ms  10:44:35
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       OK
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3
Deployment             hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1
Deployment             cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet              cilium-envoy       Desired: 3, Ready: 3/3, Available: 3/3
Containers:            cilium             Running: 3
                       cilium-envoy       Running: 3
                       hubble-relay       Running: 1
                       cilium-operator    Running: 2
Cluster Pods:          6/6 managed by Cilium
Helm chart version:
Image versions         cilium             quay.io/cilium/cilium:v1.16.0@sha256:46ffa4ef3cf6d8885dcc4af5963b0683f7d59daa90d49ed9fb68d3b1627fe058: 3
                       cilium-envoy       quay.io/cilium/cilium-envoy:v1.29.7-39a2a56bbd5b3a591f69dbca51d3e30ef97e0e51@sha256:bd5ff8c66716080028f414ec1cb4f7dc66f40d2fb5a009fff187f4a9b90b566b: 3
                       hubble-relay       quay.io/cilium/hubble-relay:v1.16.0@sha256:33fca7776fc3d7b2abe08873319353806dc1c5e07e12011d7da4da05f836ce8d: 1
                       cilium-operator    quay.io/cilium/operator-generic:v1.16.0@sha256:d6621c11c4e4943bf2998af7febe05be5ed6fdcf812b27ad4388f47022190316: 2

Demo Workload

Checkout https://github.com/PhilipSchmid/echo-app.

Cleanup

az aks delete --name $CLUSTERNAME --resource-group $RESOURCEGROUP

az group delete --name $RESOURCEGROUP

Sources

• https://medium.com/@amitmavgupta/installing-cilium-in-azure-kubernetes-service-byocni-with-no-kube-proxy-825b9007b24b
• https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/#gs-gateway-api
• https://docs.cilium.io/en/stable/installation/k8s-install-helm/#k8s-install-helm
• https://github.com/cilium/cilium/blob/v1.16.0/install/kubernetes/cilium/values.yaml
• https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-cilium-cli