3. The fleet

3.3 Keeping clusters in sync: Flux for stateless, pipelines for stateful

Requirements: R25, R26, R34.

The platform repository holds what every cluster runs, one repository per application team that team’s applications; the cluster IaC hands both to Flux:

<platform repository>/
├── clusters/<env>-<exposure>-stateless-az<N>/   # entry point of one cell, e.g. prd-int-stateless-az1
│   ├── cluster-settings.yaml                    # ConfigMap whose ${...} values the cluster IaC fills in (below)
│   ├── infrastructure.yaml                      # Kustomizations: infra-controllers → infra-configs → tenants, external-dns, observability, ...
│   ├── zones.yaml                               # zone-<zone> per enabled isolation zone
│   └── tenants.yaml
├── infrastructure/
│   ├── controllers/, configs/                   # External Secrets Operator, Cilium baseline, the platform namespaces
│   ├── zones/                                   # applied once per zone: Gateway, ESO, certificate, Cilium policies; gateway-implementations/{traefik,approuting-istio}, firewall-egress
│   ├── autoscaler-priority/                     # the cluster autoscaler's priority expander: spot pools first
│   ├── platform-gateway/, flux-webhook/         # only in a cell with a public entry point that takes GitHub's webhooks
│   └── external-dns/, flux-system-key-vault/, observability/, image-verification/, storage/, namespace-access/{view,edit,admin}
├── tenants/<namespace>/                         # a namespace's SecretStore and key-vault ServiceAccount (and its webhook Receiver)
└── policy/egress-allow-lists.yaml               # the zones' egress allow-lists, read by the CI check

<team repository>/                               # one repository per team
├── apps/<app>/                                  # Deployment, Service, HTTPRoute, VPA – cloud-neutral
├── apps/<app>-home/, <app>-global/              # the home-cell route (internal zones) and the Traffic Manager route (external zones)
└── <cell directory>/                            # what one cell syncs: image tag, replicas, patches (stateless/, firewall-egress/, ...)

The stateful cluster’s infrastructure/stateful (storage classes, operators, no Gateway API) and the applications’ stateful/<env> directories are part of the same layout, added with the first approved stateful workload.

Why two mechanisms. The stateless clusters are many, identical and rebuilt often: a pull-based reconcile loop brings a fresh cluster to the desired state without anyone pushing to it, and corrects drift continuously. The stateful cluster is a single, long-lived cluster whose changes must be ordered, gated and observed (StatefulSet rollouts, operator upgrades, schema steps) – a pipeline with explicit stages, approvals and a visible run history fits better, and the Internet-less cluster runs one component less.

Common: Git is the source, a pull request the only way in

• Policies are not in the fleet repository: they are Azure Policy definitions and assignments in the IaC repository (section 2.7); CI still tests the rendered fleet manifests against the same constraint templates.
• The merged commit is the release. CI validates every pull request (kustomize build, kubeconform against every Kubernetes minor that runs in the fleet – both stateless speeds and the stateful LTS version, see update policy – the egress names against the zone’s allow-list and policy tests); the branches take no other push than the merge GitHub makes, which GitHub signs with its own key, and Flux and the stateful pipeline verify that signature before they apply a commit (verify on the GitRepository).
• A change enters an environment by pull request. What differs per environment is in the environment’s cell directories: the image digest (section 2.7), replicas and patches. Promotion dev → acc → prd is a pull request that copies the tested values from the dev-* cell directories to the acc-* ones, then to prd-* (generated by the release agent, merged by the risk tiers of section 6.1); the images themselves are imported between the environments’ registries by the promotion pipeline (section 3.2). What is shared – apps/<app>/, infrastructure/ – needs no copy: it reaches an environment when the environment’s cells take the commit (next heading).

Stateless clusters: FluxCD

• Flux installation: the AKS GitOps extension (microsoft.flux, release train Stable, minor versions auto-upgraded, tolerating the system pool’s taint), installed by the cluster IaC, so Microsoft manages its lifecycle like the Azure Policy add-on. The stateless clusters are not network isolated; the cluster extension FQDNs (<region>.dp.kubernetesconfiguration.azure.com) are on their Firewall allow-list. (The extension is not supported on network isolated clusters, which no longer matters because the stateful cluster does not run Flux.)
• Sources are Flux configurations the cluster IaC defines (GitRepository + Kustomizations, both managed by the extension). The cluster-scope configuration platform follows the platform repository (the cell’s branch, path ./clusters/<cluster>, prune: true, source read every minute in dev and every 5 minutes by default). Each managed namespace gets a namespace-scoped configuration of its own (the cluster IaC names the team’s repository and path), applied as the namespace’s flux-applier with the post-build variables cluster_name, zone, zone_domain and home_cell; the platform repository holds for a tenant only its SecretStore and key-vault ServiceAccount (tenants/<ns>/). Every source follows the cell’s branch (below), verifies the commit signature and reads GitHub with a GitHub App’s short-lived installation token (provider: github), never a stored key.
• Order inside a cluster: dependsOn chains, prune: true to remove anything deleted from Git. infra-controllers → infra-configs (Cilium baseline, platform namespaces) → tenants; per zone node-pools-<zone> and infra-controllers → zone-<zone> → firewall-egress-<zone>; infra-controllers → flux-system-key-vault → observability; external-dns → external-dns-internal; in a cell that takes GitHub’s webhooks platform-gateway → flux-webhook. The applications’ configurations are outside this chain: a route simply waits for its gateway. Nothing waits for a zone or a team, so one broken zone or tenant never blocks the others.
• One branch per cell, in the platform repository and in every team repository: sl-az1 follows branch <env>-sl-az1, sl-az2 follows <env>-sl-az2, and so on. The release pipeline’s move is a fast-forward, and a ruleset lets only the pipeline’s identity push these branches (releases cell by cell), so a bad change never reaches every cell at once.
• Placement of internal applications is not part of the cell branches: the team’s placement.yaml is read from main by a Kustomization of its own in every cell (path placement/, interval 1 min), so a migration takes effect within minutes and independently of the release train; the application’s route carries the home cell as platform/home-cell, and the internal ExternalDNS of a cell publishes only the routes that name its own cluster (internal zones). External applications are in every cell.
• Previews and sandboxes (dev only, R34) are read the same way, outside the cell branches: a pull request marked for preview is deployed by the dev cells from its branch into <app>-pr-<n>, with the application’s own configuration, and pruned when it closes; a team’s sandbox follows the branch the team names. Neither moves with the release train, and neither reaches acc or prd (section 1.5).
• Drift is corrected on every reconcile: the Kustomizations every 10 minutes (retry after 1 minute), the sources every minute in dev. A cell with a public entry point may also take GitHub’s push webhooks: a Receiver behind its platform gateway, the platform’s one HTTP/1.1 path (section 4.1), reconciles at once. The internal cells have no public entry point, so a merge reaches them within the source interval. Flux’s notification-controller alerts the platform channel when a reconcile fails.

Stateful cluster: CI/CD pipeline (built with the first approved stateful workload)

• Where it runs: self-hosted agents in the management VNet (picture 2.2); the agents reach GitHub, the cluster never does, and it needs no Git, no reconcile controller and no extra egress.
• Identity: workload identity federation from the pipeline to a managed identity per environment (no client secret). Azure RBAC for Kubernetes gives it write access only to the stateful cluster, scoped to the platform and application namespaces; humans stay read-only.
• Stages of one run, for a given commit:
  1. check out the commit and verify its signature – exactly the content that Flux deploys;
  2. check that all assigned Azure Policy constraints are present in the cluster (policy readiness gate);
  3. kubectl diff of infrastructure/stateful, then apps/*/stateful/<env>, published as the run’s change summary;
  4. approval (prd always, acc for operator or CRD changes);
  5. server-side apply with pruning (kubectl apply --server-side --prune --applyset=…), infrastructure first;
  6. wait for StatefulSet / operator rollouts (kubectl rollout status, operator health), optionally partitioned canary first; run smoke tests against the applications’ internal load balancers;
  7. on failure stop and alert; rollback = re-run the pipeline with the previous commit (data migrations are forward-only, see the compatibility rule in section 3.4).
• Order inside an environment: the stateful stage runs first – providers before consumers. Stateful changes must be backwards compatible with the running stateless version anyway (expand → migrate → contract).
• Platform changes that touch both cluster types stay in lock-step because there is one commit: a new Cilium policy template, a shared CRD or an operator version in infrastructure/ is in the commit that the stateful stage applies first and the cells then follow branch by branch. The stateful pipeline refuses a commit whose shared infrastructure/ differs from the one the cells of the environment run, and a cell’s branch cannot move to a commit whose stateful stage has not completed, so the two never diverge by more than one release train.
• Drift: there is no reconcile loop, so a scheduled run (nightly) does steps 1–3 only and alerts on any difference. With read-only human access, drift can only come from break-glass actions.