-
Let's get started
$ tmp=`mktemp -d` && cd $tmp && pwd /var/folders/7w/tx9gxzkd4p79y2nsj7b5dmdw0000gn/T/tmp.YvRWeerz
-
Set up simple monorepo directories, preview, and add all in one commit
$ for env in common stage prod; do for app in app1 app2 cert-manager ingress-nginx; do mkdir -p $env/$app touch $env/$app/.keep done done $ tree . ├── common │ ├── app1 │ ├── app2 │ ├── cert-manager │ └── ingress-nginx ├── prod │ ├── app1 │ ├── app2 │ ├── cert-manager │ └── ingress-nginx └── stage ├── app1 ├── app2 ├── cert-manager └── ingress-nginx $ git init && git add . && git commit -m "Adding app directories" [main (root-commit) b6bad6e] Adding app directories 12 files changed, 0 insertions(+), 0 deletions(-)
-
Add some example manifests for app1, preview and commit just app1 directory
$ echo 'apiVersion: v1 kind: Namespace metadata: name: app1' > common/app1/ns.yaml $ echo 'apiVersion: apps/v1 kind: Deployment metadata: name: app1 namespace: app1 spec: selector: matchLabels: app: app1 minReadySeconds: 5 template: metadata: labels: app: app1 spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80' > stage/app1/deploy.yaml $ cp stage/app1/deploy.yaml prod/app1/deploy.yaml $ tree . ├── common │ ├── app1 │ │ └── ns.yaml │ ├── app2 │ ├── cert-manager │ └── ingress-nginx ├── prod │ ├── app1 │ │ └── deploy.yaml │ ├── app2 │ ├── cert-manager │ └── ingress-nginx └── stage ├── app1 │ └── deploy.yaml ├── app2 ├── cert-manager └── ingress-nginx $ git add . && git commit -m "Adding app1 manifests" [main 2f2202d] Adding app1 manifests 3 files changed, 44 insertions(+) create mode 100644 common/app1/ns.yaml create mode 100644 prod/app1/deploy.yaml create mode 100644 stage/app1/deploy.yaml
-
Push the repo
Could do this either before or after
flux bootstrap
, which creates the repo for you if your specified repo doesn't already exist.$ gh repo create gitops-repos-monorepo --private ✓ Created repository scottrigby/gitops-repos-monorepo on GitHub $ open https://github.com/scottrigby/gitops-repos-monorepo $ git remote add origin git@github.com:scottrigby/gitops-repos-monorepo.git $ git branch -M main $ git push -u origin main
-
Botstrap flux components
$ flux bootstrap github \ --interval 10s \ --owner scottrigby --personal \ --repository gitops-repos-monorepo \ --branch main \ --path clusters/stage Please enter your GitHub personal access token (PAT): ► connecting to github.com ► cloning branch "main" from Git repository "https://github.com/scottrigby/gitops-repos-monorepo.git" ✔ cloned repository ► generating component manifests ✔ generated component manifests ✔ committed sync manifests to "main" ("ec15492995f046c41820c5d58daceb99d1f9e74d") ► pushing component manifests to "https://github.com/scottrigby/gitops-repos-monorepo.git" ✔ installed components ✔ reconciled components ► determining if source secret "flux-system/flux-system" exists ► generating source secret ✔ public key: ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBFbZIryvvxhmP7HCuPcSjYBQ4mkw0D5AHIi37ihiL/ssYvzhZhBXGUrdsgTAslyaEvhroHgVsBFmvZIUubggNX+lJFCdENMyOivmSZytvNrsoDCg0iMAApgxn2bw1hAbGQ== ✔ configured deploy key "flux-system-main-flux-system-./clusters/stage" for "https://github.com/scottrigby/gitops-repos-monorepo" ► applying source secret "flux-system/flux-system" ✔ reconciled source secret ► generating sync manifests ✔ generated sync manifests ✔ committed sync manifests to "main" ("5bf7dc6c62a4b17d770347604aa74618ad245abc") ► pushing sync manifests to "https://github.com/scottrigby/gitops-repos-monorepo.git" ► applying sync manifests ✔ reconciled sync configuration ◎ waiting for Kustomization "flux-system/flux-system" to be reconciled ✔ Kustomization reconciled successfully ► confirming components are healthy ✔ helm-controller: deployment ready ✔ kustomize-controller: deployment ready ✔ notification-controller: deployment ready ✔ source-controller: deployment ready ✔ all components are healthy
You can see this added the bootstrapped manifests used to run Flux itself
$ git pull $ tree . ├── clusters │ └── stage │ └── flux-system │ ├── gotk-components.yaml │ ├── gotk-sync.yaml │ └── kustomization.yaml ├── common │ ├── app1 │ │ └── ns.yaml │ ├── app2 │ ├── cert-manager │ └── ingress-nginx ├── prod │ ├── app1 │ │ └── deploy.yaml │ ├── app2 │ ├── cert-manager │ └── ingress-nginx └── stage ├── app1 │ └── deploy.yaml ├── app2 ├── cert-manager └── ingress-nginx
-
Tell Flux in our stage environment to reconcile the monorepo directory containing manifests common to all environments
We're going to export this information into a custom resource that Flux understands, and put it inside our monorepo under
cluster/stage
because this is a configuration for Flux running in our staging environment.$ mkdir -p clusters/stage/app1 $ flux create kustomization common \ --namespace flux-system \ --source GitRepository/flux-system \ --path "./common" \ --interval 10s \ --prune=true \ --export > clusters/stage/apps-common-kustomization.yaml
-
Now going to do the same for the monorepo directory containing manifests specific to staging environment only
$ flux create kustomization stage \ --namespace flux-system \ --source GitRepository/flux-system \ --path "./stage" \ --interval 10s \ --prune=true \ --export > clusters/stage/apps-stage-kustomization.yaml
Push that up to Git
$ git add git add clusters/stage $ git commit -m "Adding common and staging kustomizations" $ git push
-
Check it out
$ flux get kustomizations -A NAMESPACE NAME REVISION SUSPENDED READY MESSAGE flux-system common main/5bf7dc6 False True Applied revision: main/5bf7dc6 flux-system flux-system main/5bf7dc6 False True Applied revision: main/5bf7dc6 flux-system stage main/5bf7dc6 False True Applied revision: main/5bf7dc6 kubectl get deploy -n app1 NAME READY UP-TO-DATE AVAILABLE AGE app1 1/1 1 1 12m
-
Now do app2
$ cp common/app1/ns.yaml common/app2 $ git add common/app2/ns.yaml $ sed -i 's/app1/app2/g' common/app2/ns.yaml $ git add -p diff --git a/common/app2/ns.yaml b/common/app2/ns.yaml index f39399b..b990c93 100644 --- a/common/app2/ns.yaml +++ b/common/app2/ns.yaml @@ -1,4 +1,4 @@ apiVersion: v1 kind: Namespace metadata: - name: app1 + name: app2 (1/1) Stage this hunk [y,n,q,a,d,e,?]? y $ cp stage/app1/deploy.yaml stage/app2 $ git add stage/app2/deploy.yaml $ sed -i 's/app1/app2/g' stage/app2/deploy.yaml $ git diff diff --git a/stage/app2/deploy.yaml b/stage/app2/deploy.yaml index fe85006..c258099 100644 --- a/stage/app2/deploy.yaml +++ b/stage/app2/deploy.yaml @@ -1,17 +1,17 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: app1 - namespace: app1 + name: app2 + namespace: app2 spec: selector: matchLabels: - app: app1 + app: app2 minReadySeconds: 5 template: metadata: labels: - app: app1 + app: app2 spec: containers: - name: nginx $ cp stage/app2/deploy.yaml prod/app2/deploy.yaml $ git add . $ git commit -m "Adding app2 manifests" $ watch kubectl get deployments.apps -n app2 NAME READY UP-TO-DATE AVAILABLE AGE app2 1/1 1 1 59s
Let's take a look at our monorepo's Git commit history
$ git log --oneline 8b03a81 (HEAD -> main, origin/main) Adding app2 manifests 5bf7dc6 Add Flux sync manifests ec15492 Add Flux v0.28.5 component manifests a7f51a3 Adding app1 manifests 47422e8 Adding app directories
-
Time to make an example single app repo
$ tmp=`mktemp -d` && cd $tmp && pwd /var/folders/7w/tx9gxzkd4p79y2nsj7b5dmdw0000gn/T/tmp.OAXjBFux $ git clone git@github.com:scottrigby/gitops-repos-monorepo.git gitops-repos-app1 $ cd gitops-repos-app1/ $ mkdir -p src deploy/common deploy/stage deploy/prod $ touch src/hello-world.py $ git add src $ git commit -m "Adding app1 source files" $ git filter-repo \ --path src \ --path-glob '*/app1' \ --path-rename common/app1:deploy/common \ --path-rename stage/app1:deploy/stage \ --path-rename prod/app1:deploy/prod \ --force $ tree tree . ├── deploy │ ├── common │ │ └── ns.yaml │ ├── prod │ │ └── deploy.yaml │ └── stage │ └── deploy.yaml └── src └── hello-world.py $ git log --oneline 5ef26b7 (HEAD -> main) App 1 source files 3799600 Adding app1 manifests 6a5e966 Adding app directories $ gh repo create gitops-repos-app1 --private --source=. --remote=origin $ open https://github.com/scottrigby/gitops-repos-app1
-
Temporarily suspend the monorepo kustomizations we're splitting out into app1
If we don't do this first, we'll have multiple kustomizations competing over the same resources.
$ flux suspend kustomization -n flux-system common ► suspending kustomization common in flux-system namespace ✔ kustomization suspended $ flux suspend kustomization -n flux-system stage ► suspending kustomization stage in flux-system namespace ✔ kustomization suspended $ flux get kustomizations -n flux-system NAME REVISION SUSPENDED READY MESSAGE common main/8b03a81 True True Applied revision: main/8b03a81 flux-system main/8b03a81 False True Applied revision: main/8b03a81 stage main/8b03a81 True True Applied revision: main/8b03a81
-
Now create a source and kustomizations for our new app1 repo
For this pattern, the source and kustomizations are usually stored in a separate config repo.
First, create the secret in the cluster so Flux can access the new private app1 repo:
$ flux create secret git app1-auth \ ∙ --url=ssh://git@github.com/scottrigby/gitops-repos-app1 \ ∙ --ssh-key-algorithm=ecdsa \ --ssh-ecdsa-curve=p521 ✚ deploy key: ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBABWl3QM+pGmHni5Ua9o8B5vS0vL59Bwww/Z79TD6HJFxkWEZaKyPDyKCj7YVRgwPBoZBjYVWt71Cg+zjaikIWhGsQA6Gx53buwFaEFXq0jlelrBNF+69vJXkVeWJ5gdImEIDXl6Q984Oae3Dweu09NpFdfNFDFJK2zzsocoqUNBu8k0Dw==
Copy the deploy key output above and create a read-only GitHub deploy key in your repo.
open https://github.com/scottrigby/gitops-repos-app1/settings/keys
Now create the new repo source and kustomizations inside the initial monorepo, which is now our config cluster for the new per-app repos:
$ flux create source git app1 \ --url ssh://git@github.com/scottrigby/gitops-repos-app1 \ --branch main \ --secret-ref app1-auth \ --export > clusters/stage/app1/git-source.yaml $ flux create kustomization app1-common \ --namespace flux-system \ --source GitRepository/app1 \ --path "./deploy/common" \ --interval 10s \ --prune=true \ --export > clusters/stage/app1/common-kustomization.yaml $ flux create kustomization app1-stage \ --namespace flux-system \ --source GitRepository/app1 \ --path "./deploy/stage" \ --interval 10s \ --prune=true \ --export > clusters/stage/app1/stage-kustomization.yaml $ git add clusters/stage/app1 $ git commit -m "Creating a source and kustomizations for new app1 repo" $ git push $ flux reconcile kustomization -n flux-system app1-stage $ flux get kustomizations -A NAMESPACE NAME REVISION SUSPENDED READY MESSAGE flux-system app1-common main/a136159 False True Applied revision: main/a136159 flux-system app1-stage main/a136159 False True Applied revision: main/a136159 flux-system common main/6e13fc0 True True Applied revision: main/6e13fc0 flux-system flux-system main/c1a588c False True Applied revision: main/c1a588c flux-system stage main/8b03a81 True True Applied revision: main/8b03a81
-
Keep paused while repeating for all other apps. Once complete, remove the old kustomizations from the config repo
$ git rm clusters/stage/apps-common-kustomization.yaml \ clusters/stage/apps-stage-kustomization.yaml $ git add . $ git commit -m "Remove old apps-common and apps-stage kustomizations" $ git push
The old kustomizations will automatically be removed from the system
$ flux get kustomizations -A NAMESPACE NAME REVISION SUSPENDED READY MESSAGE flux-system app1-common main/795ba05 False True Applied revision: main/795ba05 flux-system app1-stage main/795ba05 False True Applied revision: main/795ba05 flux-system flux-system main/45b2e03 False True Applied revision: main/45b2e03
-
To merge multiple repos from one pattern to another, just reverse the above steps!
-
Quick recap
- This demo shows that with flexible tooling and techniques, you can iterate on repo structure – in this example, splitting apart a monorepo into separate per-app repos.
- You can also do the same with repos per environment and repos per team, as well as reverse this by merging repos together into any of the above patterns. You can use a mix of these on running systems all at the same time.
- You can do any of the above without downtime.
- You can do this while retaining your Git commit history, for auditing purposes or just better developer experience 🎉
-
Clean up by deleting your kind cluster and 2 private demo repos on GitHub (your local temp directories will auto-clean) 🙂
Last active
January 24, 2023 20:52
-
-
Save scottrigby/0be557cdf4b60e1f6ccf7d9a7332dfdb to your computer and use it in GitHub Desktop.
How to Structure GitOps Repos
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment