Skip to content

Instantly share code, notes, and snippets.

@yankcrime
Created June 21, 2021 20:51
Show Gist options
  • Save yankcrime/d08f771257b1031e6ca71ee694a5db60 to your computer and use it in GitHub Desktop.
Save yankcrime/d08f771257b1031e6ca71ee694a5db60 to your computer and use it in GitHub Desktop.

Kubernetes and External Authentication with Rancher

Users added in Rancher get an object of kind User.

For them to be able to do anything, they need to login to Rancher and authenticate via the external AuthN provider. An instance of kind Token is then created which contains the User Principal account details:

$ kubectl describe user u-smckoeh6vq
Name:          u-smckoeh6vq
Namespace:
Labels:        CTKN8Q3LC9FNASR5E8T2UBPM6SP3ECPN6GOG=hashed-principal-name
               cattle.io/creator=norman
Annotations:   authz.management.cattle.io/creator-role-bindings: {"created":["user"],"required":["user"]}
               lifecycle.cattle.io/create.mgmt-auth-users-controller: true
API Version:   management.cattle.io/v3
Description:
Display Name:  crimeyank
Kind:          User
Metadata:
  Creation Timestamp:  2020-06-22T14:38:04Z
  Finalizers:
    controller.cattle.io/mgmt-auth-users-controller
  Generation:        3
  Resource Version:  12127212
  Self Link:         /apis/management.cattle.io/v3/users/u-smckoeh6vq
  UID:               76eb0284-0fe6-47be-a21c-695e9a6299b3
Principal Ids:
  github_user://67273741
  local://u-smckoeh6vq
Spec:
Status:
  Conditions:
    Last Update Time:  2020-06-22T14:38:04Z
    Status:            True
    Type:              InitialRolesPopulated
Events:                <none>
$ kubectl get tokens -l authn.management.cattle.io/token-userId=u-smckoeh6vq
NAME          AGE
token-2p56w   113m

$ kubectl describe tokens -l authn.management.cattle.io/token-userId=u-smckoeh6vq
Name:              token-2p56w
Namespace:
Labels:            authn.management.cattle.io/kind=session
                   authn.management.cattle.io/token-userId=u-smckoeh6vq
                   cattle.io/creator=norman
Annotations:       <none>
API Version:       management.cattle.io/v3
Auth Provider:     github
Current:           false
Description:       UI Session
Expired:           false
Expires At:        2020-06-23T06:40:09Z
Group Principals:  <nil>
Is Derived:        false
Kind:              Token
Last Update Time:
Metadata:
  Creation Timestamp:  2020-06-22T14:40:09Z
  Generate Name:       token-
  Generation:          2
  Resource Version:    12127751
  Self Link:           /apis/management.cattle.io/v3/tokens/token-2p56w
  UID:                 4150dd5e-390a-4535-856a-816648c0d494
Token:                 ftqxtltnlwgsgdjr4tkwlrx7xh4s4wm7zqd6kxtrtgf7kgs5tgt8xg
Ttl:                   57600000
User Id:               u-smckoeh6vq
User Principal:
  Display Name:  crimeyank
  Login Name:    crimeyank
  Me:            true
  Metadata:
    Creation Timestamp:  <nil>
    Name:                github_user://67273741
  Principal Type:        user
  Profile Picture:       https://avatars0.githubusercontent.com/u/67273741?v=4
  Provider:              github
Events:                  <none>

API access to downstream clusters

There’s an API call made when the user logs in, navigates to a target cluster to which they've been granted access, and requests the cluster's kubeconfig. This API call results in another Token object being created:

$ kubectl describe tokens -l authn.management.cattle.io/token-userId=u-smckoeh6vq,authn.management.cattle.io/kind=kubeconfig
Name:              kubeconfig-u-smckoeh6vq
Namespace:
Labels:            authn.management.cattle.io/kind=kubeconfig
                   authn.management.cattle.io/token-userId=u-smckoeh6vq
                   cattle.io/creator=norman
Annotations:       <none>
API Version:       management.cattle.io/v3
Auth Provider:     local
Current:           false
Description:       Kubeconfig token
Expired:           false
Expires At:
Group Principals:  <nil>
Is Derived:        true
Kind:              Token
Last Update Time:
Metadata:
  Creation Timestamp:  2020-06-22T16:35:59Z
  Generation:          1
  Resource Version:    12157196
  Self Link:           /apis/management.cattle.io/v3/tokens/kubeconfig-u-smckoeh6vq
  UID:                 0f65b528-573f-4322-ae95-264810a42142
Token:                 8hfhkt6hvd8vpj2j5whskmbdrpqmxr754fkw8g294v5g5jfqdft8sb
Ttl:                   0
User Id:               u-smckoeh6vq
User Principal:
  Metadata:
    Creation Timestamp:  <nil>
Events:                  <none>

This is the token that the user can use to authenticate with a downstream cluster's API.

For downstream clusters that are imported, the only way that users can authenticate with an external source is by using Rancher's authentication proxy, i.e all API requests are via Rancher.

For clusters that have been deployed by Rancher and which have the Authorised Cluster Endpoint enabled, it's possible for users to connect directly to the API of these downstream clusters.

Understanding authentication flow with ACEs

If the user is authenticating directly with a downstream cluster that has ACEs enabled, the kube-api-auth service (which runs on each node with the 'controlnode' role) handles the webhook authentication based on various custom resources that are defined when the cluster has been installed.

One of the custom resources is ClusterAuthTokens - objects of this type contain a hashed copy of the user's token. This is verified by the kube-api-auth as part of each client API request.

The ClusterUserAttributes CRD contains additional context such as any groups that the user may belong to.

$ kubectl ctx
Switched to context "azurenetes"

$ kubectl get clusterauthtokens -n cattle-system
NAME                              AGE
kubeconfig-u-smckoeh6vq.c-s8d2f   78m
kubeconfig-user-2kml8.c-s8d2f     110m

$ kubectl describe clusterauthtoken kubeconfig-u-smckoeh6vq.c-s8d2f -n cattle-system
Name:         kubeconfig-u-smckoeh6vq.c-s8d2f
Namespace:    cattle-system
Labels:       cattle.io/creator=norman
Annotations:  <none>
API Version:  cluster.cattle.io/v3
Enabled:      true
Hash:         $1:20b85c044ca5fc53:15:8:1:dFYFsaVy4mjLmUF+hhZ0PEVgEAnT1sRclgJNOvA6OOw3E/PrHUTxvJx6nVsTo6c5grezAIr/CYg+xemIMChTfg
Kind:         ClusterAuthToken
Metadata:
  Creation Timestamp:  2020-06-23T09:51:48Z
  Generation:          1
  Resource Version:    13394
  Self Link:           /apis/cluster.cattle.io/v3/namespaces/cattle-system/clusterauthtokens/kubeconfig-u-smckoeh6vq.c-s8d2f
  UID:                 44df162d-a3a9-4378-ac10-6e72bdf70a03
User Name:             u-smckoeh6vq
Events:                <none>

$ kubectl get clusteruserattributes u-smckoeh6vq -n cattle-system
NAME           AGE
u-smckoeh6vq   80m

$ kubectl describe clusteruserattributes u-smckoeh6vq -n cattle-system
Name:          u-smckoeh6vq
Namespace:     cattle-system
Labels:        cattle.io/creator=norman
Annotations:   <none>
API Version:   cluster.cattle.io/v3
Enabled:       true
Groups:
  github_org://1234567
  github_org://8765432
Kind:          ClusterUserAttribute
Last Refresh:  2020-06-23T09:51:32Z
Metadata:
  Creation Timestamp:  2020-06-23T09:51:47Z
  Generation:          1
  Resource Version:    13393
  Self Link:           /apis/cluster.cattle.io/v3/namespaces/cattle-system/clusteruserattributes/u-smckoeh6vq
  UID:                 8f2137f8-a86e-41d1-9716-62dc56a46de4
Needs Refresh:         false
Events:                <none>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment