The year is specified, because things can change and better ways to do it will possibly appear. I'm documenting what worked for me right now, since trying to bootstrap an Akka Cluster instance on K8s with Istio took a few days of investigation, and I couldn't easily find a working solution.
This is not a detailed description of Akka Cluster and related configuration, but rather a short description of the minimal amount of things that are necessary to allow Akka Cluster nodes to see each other on K8s, if istio is in the way.
The first attempt to find something about "Akka Cluster Bootstrap on K8s" will likely show this page:
This API allows Akka Management to use kubernetes-api Akka Discovery method which detects IPs of your application pods by using pod-label-selector. It works perfectly if istio is not used in your cluster. Because istio prevents direct pod-by-pod communication, and cluster nodes can't see each other because of this.
If you're allowed to exclude some ports from your Istio protection, probably this page can be helpful:
In my case it wasn't possible, so I had to find another way:
This page:
Mentions akka-dns discovery-method and headless service, but it doesn't really give a good explanation of what exactly we need to make it work when we use Istio.
In this example we use the following K8s concepts:
- StatefulSets to get a pod URL for every pod.
- Headless Service as a governing service for your stateful set. Basically, to allow K8s to create unique static pod URLs for every pod.
- NetworkPolicy to tell Istio that we indeed allow these pods to interact with these pods via our services.
If you use default configuration, like port 8558 for Akka management, port 2552 for Akka remoting and some app named my-app in my-namespace namespace, you'd need a Service like this to be created:
apiVersion: v1
kind: Service
metadata:
name: my-app-cluster
namespace: my-namespace
labels:
app: my-app
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
clusterIP: None
publishNotReadyAddresses: true
selector:
app: my-app
ports:
- name: management
port: 8558
targetPort: 8558
- name: remoting
port: 2552
targetPort: 2552These tolerate-unready-endpoints annotation and publishNotReadAddresses parameter are taken from Akka configuration and are expected to allow your pods to see each other via this service even while they're still not Ready.
If you're wondering, why you'd expect them to be seen in this state - this service is used only for internal communication between cluster nodes, and normally a node that hasn't joined a cluster, is not Ready yet (if you follow Akka Cluster recommendations, for sure). If this Service was not showing these nodes, they would not be successfully reached by other cluster members and it wouldn't be possible to bootstrap our cluster.
Now, we need to tell Istio that we want traffic for ports 8558 and 2552 to go through this service between pods labeled with app: myapp. We would need something like this:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: my-app-cluster
namespace: my-namespace
spec:
podSelector:
matchLabels:
app: my-app
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: my-app
ports:
- protocol: TCP
port: 2552
- protocol: TCP
port: 8558
egress:
- to:
- podSelector:
matchLabels:
app: my-app
ports:
- protocol: TCP
port: 2552
- protocol: TCP
port: 8558The main benefit of a StatefulSet is described here.
In simplicity, it means, that if we have a StatefulSet named my-app that contains 3 replicas and its Headless Service my-app-cluster, and they are in my-namespace namespace, and our cluster domain is cluster.local, then we would have pod hostnames like this:
my-app-0.my-app-cluster.my-namespace.svc.cluster.local
my-app-1.my-app-cluster.my-namespace.svc.cluster.local
my-app-2.my-app-cluster.my-namespace.svc.cluster.local
The benefit of using these hostname is - istio allows us to reach pods via them. So for all inter-pod communcation we need to use these hostnames instead of IPs.
Keep in mind that this is just a template, and you would likely have more stuff on your StatefulSet, like istio annotations, extra environment variables, etc:
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: my-namespace
name: my-app
labels:
app: my-app
spec:
serviceName: my-app-cluster
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app
image: <your app image>
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: SERVICE_NAME
value: my-app-cluster
- name: AKKA_REMOTE_ARTERY_CANONICAL_HOSTNAME
value: "$(POD_NAME).$(SERVICE_NAME).$(NAMESPACE).svc.cluster.local"
ports:
- name: management
containerPort: 8558
- name: remoting
containerPort: 2552NB: Notice that serviceName value is the same as the name of the Service that we created: my-app-cluster.
Environment variables POD_NAME, NAMESPACE, SERVICE_NAME and AKKA_REMOTE_ARTERY_CANONICAL_HOSTNAME are used for proper Akka Bootstrap and Akka Remote configuration.
We need to ensure, that:
- Every node can correctly detect out other nodes' URLs.
- Nodes can talk to each other successfully.
This is done by using Akka Management library, and specifically Akka Cluster Bootstrap.
To configure Akka Management to listen on port 8558 and detect other cluster members via the service that we just created, you'd need to add something like this to your application.conf:
management {
http {
bind-hostname = 0.0.0.0
bind-port = 8558
}
cluster.bootstrap {
contact-point-discovery {
service-name = ${?SERVICE_NAME}
service-namespace = ${?NAMESPACE}
port-name = management
discovery-method = akka-dns
}
}
}This is the minimal configuration that tells cluster bootstrap to use SERVICE_NAME service in NAMESPACE namespace for nodes discovery with akka-dns discovery method and use its management port to reach other nodes.
There are some other configuration fields that you can find in Cluster Bootstrap documentation if you want to fine-tune it. But this is good enough for a start.
This is done via Akka Remoting, and in our example we use Artery over TCP, which is the default choice.
You would need something like this to make it work:
remote.artery {
bind.hostname = 0.0.0.0
bind.port = 2552
canonical.hostname = ${?AKKA_REMOTE_ARTERY_CANONICAL_HOSTNAME}
canonical.port = 2552
}This Canonical hostname thing is this pod's URL that we mentioned in the K8s part. It is used to advertise to other pods, how they should reach this pod, i.e. that's this pod's external URL. See Canonical address for more information.
Binding to 0.0.0.0 ensures that your pod will be listening on canonical.hostname and will be able to process incoming requests. Maybe duplicated port information is redundant, but just to be on the safe side I left it as is.
It is a bit misleading that Akka Management doesn't mention here that it supports akka-dns directly, through bootstrap configuration. But it does.
If you add this when you initialise your application:
AkkaManagement(as).start()
ClusterBootstrap(as).start()and you have all necessary libraries installed (see Akka Management and Akka Remoting documentation above), then when your pods are starting up, they should be able to see each other and establish a cluster.
This doesn't fully cover the whole topic of running an Akka Cluster on K8s and possible issues that you could encounter.
There is a chance that this would not work because my cluster was different to yours or configuration changed since the days when this was written down. Though I believe, it should be helpful overall to figure out what to do if you suddenly realise that istio doesn't allow you to use kubernetes-api, and you don't know what to try next.