Let's say you have a set of disparate nodepools in a cluster:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-26541295-vmss000004 Ready agent 28h v1.26.3
aks-nodepool1-26541295-vmss000005 Ready agent 28h v1.26.3
aks-nodepool1-26541295-vmss000006 Ready agent 28h v1.26.3
aks-nodepool1-26541295-vmss000007 Ready agent 28h v1.26.3
aks-poisonpool-25406481-vmss000000 Ready agent 25h v1.26.3
aks-poisonpool-25406481-vmss000001 Ready agent 25h v1.26.3
aks-poisonpool-25406481-vmss000002 Ready agent 25h v1.26.3
aks-poisonpool-25406481-vmss000003 Ready agent 25h v1.26.3
You'll notice we've got two distinct nodepools here, nodepool1 and poisonpool.
Let's say we want to ensure that TAP is only installed on the nodepool1 nodes. We can do this by adding a taint to the nodepool1
nodes:
kubectl taint nodes aks-nodepool1-26541295-vmss000004 reservedfor=tap:NoSchedule
kubectl taint nodes aks-nodepool1-26541295-vmss000005 reservedfor=tap:NoSchedule
kubectl taint nodes aks-nodepool1-26541295-vmss000006 reservedfor=tap:NoSchedule
kubectl taint nodes aks-nodepool1-26541295-vmss000007 reservedfor=tap:NoSchedule
We'll have to add the inverse taint to the poisonpool
nodes:
kubectl taint nodes aks-poisonpool-25406481-vmss000000 reservedfor=notap:NoSchedule
kubectl taint nodes aks-poisonpool-25406481-vmss000001 reservedfor=notap:NoSchedule
kubectl taint nodes aks-poisonpool-25406481-vmss000002 reservedfor=notap:NoSchedule
kubectl taint nodes aks-poisonpool-25406481-vmss000003 reservedfor=notap:NoSchedule
The following ytt will add the annotation kapp.k14s.io/exists to all namespaces. Why are we doing this? In summary, this ytt overlay is designed to add the kapp.k14s.io/exists annotation to all Namespace resources within the tap-install namespace. The annotation is added if it doesn't already exist. This overlay modification can be useful in various scenarios, such as tracking the existence of a Namespace resource within your Kubernetes environment.
apiVersion: v1
kind: Secret
metadata:
name: ns-kapp-exists-overlay
namespace: tap-install
stringData:
annotate-namespace-with-exists.yaml: |
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.subset({"kind": "Namespace"}), expects="0+"
---
metadata:
#@overlay/match missing_ok=True
annotations:
#@overlay/match missing_ok=True
kapp.k14s.io/exists: ""
Add that overlay Secret to the cluster:
kubectl apply -f ns-kapp-exists-overlay.yaml
Let's ensure that we're creating all of these namespaces ahead of time so we're targeting the correct toleration:
namespaces=(accelerator-system api-auto-registration api-portal appliveview-tokens-system app-live-view-connector app-live-view-conventions app-live-view appsso build-service cert-injection-webhook stacks-operator-system kpack cartographer-system cert-manager knative-serving crossplane-system developer-conventions triggermesh knative-eventing knative-sources vmware-sources flux-system learning-center-guided-ui learningcenter metadata-store tap-namespace-provisioning cosign-system scan-link-system service-bindings services-toolkit source-system spring-boot-convention tap-gui tap-telemetry vmware-system-telemetry tekton-pipelines-resolvers tekton-pipelines tap-install kapp-controller secretgen-controller tanzu-cluster-essentials tanzu-package-repo-global tap-workload dev)
for ns in ${namespaces[@]}; do
kubectl create ns "$ns"
kubectl annotate ns "$ns" 'scheduler.alpha.kubernetes.io/defaultTolerations'='[{"operator": "Equal", "effect": "NoSchedule", "key": "reservedfor", "value": "tap"}]'
done
We'll have to now account for that annotation in our TAP installation. We'll do that by adding the following to our TAP values:
shared:
ingress_domain: tap.az.anew.io
image_registry:
project_path: "registry.az.anew.io/tap-build"
username: "registry-username-example"
password: "hunter2"
profile: full
ceip_policy_disclosed: true
supply_chain: basic
scanning:
metadataStore:
url:
contour:
envoy:
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "aks-subnet"
service.beta.kubernetes.io/azure-load-balancer-internal-vnet: "aks-vnet"
service.beta.kubernetes.io/azure-load-balancer-internal-dns-label-name: "tap-contour"
service.beta.kubernetes.io/azure-load-balancer-internal-dns-zone-name: "tap.az.anew.io"
accelerator:
ingress:
include: true
enable_tls: false
appliveview:
ingressEnabled: true
appliveview_connector:
backend:
ingressEnabled: true
sslDeactivated: false
package_overlays:
- name: accelerator
secrets:
- name: ns-kapp-exists-overlay
- name: api-auto-registration
secrets:
- name: ns-kapp-exists-overlay
- name: api-portal
secrets:
- name: ns-kapp-exists-overlay
- name: appliveview
secrets:
- name: ns-kapp-exists-overlay
- name: appliveview-apiserver
secrets:
- name: ns-kapp-exists-overlay
- name: appliveview-connector
secrets:
- name: ns-kapp-exists-overlay
- name: appliveview-conventions
secrets:
- name: ns-kapp-exists-overlay
- name: appsso
secrets:
- name: ns-kapp-exists-overlay
- name: bitnami-services
secrets:
- name: ns-kapp-exists-overlay
- name: buildservice
secrets:
- name: ns-kapp-exists-overlay
- name: cartographer
secrets:
- name: ns-kapp-exists-overlay
- name: cert-manager
secrets:
- name: ns-kapp-exists-overlay
- name: cnrs
secrets:
- name: ns-kapp-exists-overlay
- name: crossplane
secrets:
- name: ns-kapp-exists-overlay
- name: developer-conventions
secrets:
- name: ns-kapp-exists-overlay
- name: eventing
secrets:
- name: ns-kapp-exists-overlay
- name: fluxcd-source-controller
secrets:
- name: ns-kapp-exists-overlay
- name: grype
secrets:
- name: ns-kapp-exists-overlay
- name: learningcenter
secrets:
- name: ns-kapp-exists-overlay
- name: learningcenter-workshops
secrets:
- name: ns-kapp-exists-overlay
- name: metadata-store
secrets:
- name: ns-kapp-exists-overlay
- name: namespace-provisioner
secrets:
- name: ns-kapp-exists-overlay
- name: ootb-delivery-basic
secrets:
- name: ns-kapp-exists-overlay
- name: ootb-supply-chain-basic
secrets:
- name: ns-kapp-exists-overlay
- name: ootb-templates
secrets:
- name: ns-kapp-exists-overlay
- name: policy-controller
secrets:
- name: ns-kapp-exists-overlay
- name: scanning
secrets:
- name: ns-kapp-exists-overlay
- name: contour
secrets:
- name: ns-kapp-exists-overlay
- name: service-bindings
secrets:
- name: ns-kapp-exists-overlay
- name: services-toolkit
secrets:
- name: ns-kapp-exists-overlay
- name: source-controller
secrets:
- name: ns-kapp-exists-overlay
- name: spring-boot-conventions
secrets:
- name: ns-kapp-exists-overlay
- name: tap-auth
secrets:
- name: ns-kapp-exists-overlay
- name: tap-gui
secrets:
- name: ns-kapp-exists-overlay
- name: tekton-pipelines
secrets:
- name: ns-kapp-exists-overlay
Note that we've added the ns-kapp-exists-overlay
Secret to all the packages that we want to install. This will ensure that the overlay is applied to all the packages that we install.
Now, we can install TAP on the cluster, and it will only be installed on the nodes that don't have the reservedfor=nontap:NoSchedule
taint:
tanzu package install tap -p tap.tanzu.vmware.com -v 1.5.0 --values-file tap-values.yaml -n tap-install
We might want an example of using the overlay by adding in the defaultTolerations to ytt overlay as I think that might be a more common pattern vs what we are showing how to handle if the namespace already exists?