We are trying to expose some pods as nodeport to allow inter k8s communication. But the nodeport does not get routed properly to the target pod when the origin is one of the pods themselves.
We run KOPS v1.11.9
Using this test scenario:
---
apiVersion: v1
kind: Service
metadata:
name: echo-server
namespace: hector-test
spec:
externalTrafficPolicy: Cluster
ports:
- name: http
nodePort: 30180
port: 30180
protocol: TCP
targetPort: 80
selector:
app: echo-server
sessionAffinity: None
type: NodePort
---
apiVersion: v1
kind: Service
metadata:
name: echo-server-local
namespace: hector-test
spec:
externalTrafficPolicy: Local
ports:
- name: http
nodePort: 30181
port: 30181
protocol: TCP
targetPort: 80
selector:
app: echo-server
sessionAffinity: None
type: NodePort
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: echo-server
namespace: hector-test
spec:
serviceName: echo-server
podManagementPolicy: OrderedReady
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: echo-server
template:
metadata:
labels:
app: echo-server
spec:
containers:
- image: ealen/echo-server
imagePullPolicy: Always
name: echo-server
ports:
- containerPort: 80
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
There are 2 nodeports:
- 30180 uses
externalTrafficPolicy: Cluster
- 30181 uses
externalTrafficPolicy: Local
We run this script to see the responses:
TARGET_NODE=1.2.3.4
# Port 30180 with Cluster
for i in $(seq 100); do curl -qs ${TARGET_NODE}:30180 | jq -r .environment.HOSTNAME; done | sort |uniq -c
# Port 30181 with Local
for i in $(seq 100); do curl -qs ${TARGET_NODE}:30181 | jq -r .environment.HOSTNAME; done | sort |uniq -c
-
From any pod running on node outside that is NOT these echo-server pods, inside of KOPS (or any other node):
- 30180 Cluster => Roundrobins both pods
- 30181 Local => Hits only one Pod
-
From one of the pods (on KOPS v1.11.9)
- 30180 Cluster => Roundrobins both pods
- 30180 Local => Roundrobins both pods !!!??! This is unexpected
-
On a more modern k8s, eg. EKS v1.16.15, from one of the pods
- 30180 Cluster => Roundrobins both pods
- 30181 Local => Hits only one Pod
- A nodeport with
externalTrafficPolicy: Cluster
will do always round robin of the service - KOPS v1.11.9 has some bug, so ``externalTrafficPolicy: Local` does roundrobin when hit from the pods associated to the service.