Skip to content

Instantly share code, notes, and snippets.

@darkowlzz
Last active March 4, 2022 16:41
Show Gist options
  • Save darkowlzz/30c31f2e81c48b20398edc082d4fcc96 to your computer and use it in GitHub Desktop.
Save darkowlzz/30c31f2e81c48b20398edc082d4fcc96 to your computer and use it in GitHub Desktop.

kstatus - conditions and generations check

This document describes how the checker tool (CLI and library) can perform Kubernetes object status conditions and generations check as per kstatus.

Before running the tests, the checker is provided with some context about the controller:

  • The status conditions it supports with their polarity and priority of the conditions.
  • The scheme of the controller managed object.

Configuration

  • List of status conditions in the order of their priority.
# List of supported conditions.
conditions:
  negativePolarity:
    - FetchFailed
    - ArtifactOutdated
    - Stalled
    - Reconciling
  positivePolarity:
    - RemoteAvailable
    - SecretsAvailable

In the conditions above, among the negative polarity conditions, the first condition (FetchFailed) has the highest priority and the last condition (Reconciling) has the lowest priority. In case of multiple negative polarity conditions being present, the values of the condition with the highest polarity is also reflected in the Ready condition value.

Usage

The checker is executed whenever there's a need to verify if the current object's status is valid.

  • For tests that execute terminal commands, for example in smoke tests, the checker could be run after every execution of a change, like create an object, update an object, etc.
$ controller-check --config cc.yaml --name targetobj --namespace targetns --kind Foo
  • Similarly, for go tests, the checker could be run after every change by calling the checker function.
conditions := cc.Conditions{NegativePolarity: []string{C1, C2, C3}}
checker := cc.NewChecker(client, scheme, conditions)
checker.CheckErr(ctx, obj)

Status condition check

Given an object, ensure that:

  • There are no negative polarity conditions set when Ready condition is True. Suggest that the negative polarity condition can be removed. (WARN0001)
  • There are no True negative polarity conditions when Ready condition is True. (FAIL0001)
  • There is Ready condition set always with value True or False based on the situation. (FAIL0002)
  • When there are multiple negative polarity conditions with True value present, the Ready condition value is equal to the value of the negative polarity condition with the highest priority. (WARN0003)

If the controller supports Reconciling condition, ensure that:

  • Ready condition is False when Reconciling condition is True. (FAIL0003)
  • Reconciling condition is not present when its value is False. (WARN0003)

If the controller supports Stalled condition, ensure that:

  • Ready condition is False when Stalled condition is True. (FAIL0004)
  • Stalled condition is not present when its value is False. (WARN0004)

If the controller supports both Reconciling condition and Stalled condition, ensure that:

  • Reconciling condition and Stalled condition are mutually exclusive to one another. (FAIL0005)
  • In presence of Reconciling condition or Stalled condition, Ready condition is always set to False. (FAIL0004, FAIL0005)

NOTE: Reconciling condition and Stalled condition conditions need not be present whenever Ready condition is False. There could be situation where an object is not ready and not reconciling, like a transient error due to unavailable of some resource.

Status observed generation and conditions check

Given an object, ensure that:

  • The status root ObservedGeneration and the status conditions ObservedGeneration are always less than or equal to the object Generation. (FAIL0006)
  • When the status root ObservedGeneration or the status conditions ObservedGeneration is less than the object Generation, the object's Ready condition status is False. It indicates that the desired state and the actual state aren't equal. (FAIL0007, FAIL0008)
  • When the Ready condition is True, all the status conditions' ObservedGeneration equal to the object status root ObservedGeneration. (FAIL0009)
  • All the status conditions have ObservedGeneration defined. (WARN0005)
  • When the Reconciling condition is True, the Reconciling condition ObservedGeneration is greater than the status root ObservedGeneration, if present. (FAIL0010)

NOTE: All the status conditions' ObservedGeneration, root ObservedGeneration and object Generation can be equal when Ready condition is True or Stalled condition is True.

Warning and Failure codes

Warnings:

Code Message
WARN0001 Negative polarity condition present when Ready condition is True
WARN0002 Ready condition should have the value of the negative polarity conditon that's present with the highest priority
WARN0003 Reconciling condition can be removed when its value is False
WARN0004 Stalled condition can be removed when its value is False
WARN0005 Missing ObservedGeneration from status condition

Failures:

Code Message
FAIL0001 Negative polarity condition cannot be True when Ready condition is True
FAIL0002 Ready condition must always be present
FAIL0003 Ready condition must be False when Reconciling condition is True
FAIL0004 Ready condition must be False when Stalled condition is True
FAIL0005 Only one of Reconciling condition or Stalled condition must be present at a time
FAIL0006 The ObservedGeneration must be less than or equal to the object Generation
FAIL0007 Ready condition must be False when the ObservedGeneration is less than the object Generation
FAIL0008 Ready condition must be False when any of the status condition's ObservedGeneration is less than the object Generation
FAIL0009 The status conditions' ObservedGenerations must be equal to the root ObservedGeneration when Ready condition is True
FAIL0010 The root ObservedGeneration must be less than the Reconciling condition ObservedGeneration when Reconciling condition is True

Go package implementation: https://pkg.go.dev/github.com/darkowlzz/controller-check@main/status


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment