Skip to content

Instantly share code, notes, and snippets.

@ivanthelad

ivanthelad/keepingpublicip.md

Last active November 12, 2020 17:41
Show Gist options
  • Save ivanthelad/e3ed930657399d89cf01836a1b0cb975 to your computer and use it in GitHub Desktop.
Save ivanthelad/e3ed930657399d89cf01836a1b0cb975 to your computer and use it in GitHub Desktop.
Download ZIP
Ensuring business critical IPs are not deleted by an kubernetes deployments/deletes
Raw
keepingpublicip.md

Ensuring business critical IPs are not deleted by an kubernetes deployments/deletes

Background:

Kubernetes azure cloud provider will dynamically create public IPs when a service of type=Loadbalancer is created. From the Kubernetes cloud provider perspective it owns this dynamically created IP, meaning it is responsible for updates and deletes to this IP. If I delete the service in Kubernetes the cloud provider will cascade the delete by deleting also the Public IP in Azure. (The cloudprovider calls this process reconciliation). Once an IP is deleted in azure it is returned to azures global pool of IP. You have no guarantees that you will get this IP again

This behaviour unfortunately can cause challenges if your application now relies on this public IP and it is accidentally deleted. To ensure a system critical Azure IP is not deleted when modifying deployments in Kubernetes the following steps should be performed.

Note: These steps are to ensure a dynamically created IP is not deleted by AKS. if you supply your own IP like outlined in these instructions then the cloudprovider does not own it and therefore will not delete it

Recommendation to ensure a IP is not deleted

1)Service YAML files should statically reference IPs using the following annotations

  • service.beta.kubernetes.io/azure-load-balancer-resource-group
  • service.beta.kubernetes.io/azure-pip-name

https://docs.microsoft.com/en-us/azure/aks/load-balancer-standard#additional-customizations-via-kubernetes-annotations

2)Apply Lock to IP resource in Azure

This ensure any accidental delete by a system or human is prevented. A user/system must explicitly remove a lock to proceed with delete

3)Remove/modify tags on dynamically created public IP

  • Azure cloud provider in Kubernetes uses tags to identity Public IPs which it dynamically created and therefore “owns”. If Kubernetes cloudprovider “owns” an IP then it can reconcile the IP. This can result in IP address deletion when its corresponding service in Kubernetes is deleted. If you want to ensure an ip is not deleted then remove “service” tag on the public ip
  • Interestingly this is the reason why a customer create IP is not removed https://docs.microsoft.com/en-us/azure/aks/static-ip. It does not have these tags applied therefore the Cloudprovider does not own it.
  • The logic behind the deletion can be found here(1.15). The providers tag deletion logic is more or less the same up to version 1.19

Optional Recommendations

I would recommend, if the IP is a dynamically created by the azure cloud provider and is under the MC_ resource group then it should be moved to a custom Resource group. This would enable clearer ownership and governance of the _P and it would also prevent IP deletion in the event you want to rebuild the Kubernetes cluster

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment