3. The fleet
3.4 Zero-downtime application upgrades
The same rules apply in every cluster and are enforced by admission policy (Azure Policy add-on), so no application can be deployed in a way that breaks a zero-downtime rollout or a node drain:
| Guardrail | Stateless | Stateful |
|---|---|---|
| Replicas | replicas / HPA minReplicas ≥ 2, or 0 (switched off; nothing in between) |
≥ 3, one per AZ |
| Spread | topologySpreadConstraints over nodes |
topologySpreadConstraints over topology.kubernetes.io/zone |
| PodDisruptionBudget | Mandatory, maxUnavailable ≥ 1 |
Mandatory, maxUnavailable: 1 |
| Rollout strategy | RollingUpdate, maxUnavailable: 0, any maxSurge ≥ 1 |
RollingUpdate (optionally partition for canary) or operator-managed |
| Probes | readiness + liveness (+ startup) required | readiness gated on replication / quorum |
| Graceful shutdown | preStop delay + terminationGracePeriodSeconds longer than the gateway / traffic layer drain time |
same, plus clean leader hand-over |
| Resources | requests required (HPA and the cluster autoscaler depend on them); resized in place by the VPA within Git bounds (section 2.3) | requests = limits for memory; VPA recommendations only |
| Images | From the environment’s ACR only, by digest (section 2.7) | same |
| Non-serving workloads | See below: the replica, spread, budget and rollout rows apply to serving workloads only | same |
- Scaling on load (stateless): HPA on CPU/memory (KEDA on queue length / request rate when a team needs it);
the cluster autoscaler adds nodes for
pods that do not fit and removes them again.
maxReplicasand the zone pools’ maximum are sized so that one cluster can take the full load. Vertically, the VPA resizes running pods in place – no new pod, usually no restart – within the bounds in Git (pod right-sizing), so following the load needs no release; only changing the bounds does. - No scaling on load in the stateful cluster: replica counts and node pool sizes are fixed and changed as planned changes through the pipeline and IaC.
- Compatibility rule: during a release two versions run at the same time (in different cells, and for a few minutes inside a cell during the rolling update), so API and message changes must be backwards compatible and database changes follow expand → migrate → contract over separate releases. DNS affinity (external zones) and the single home cell (internal zones) reduce what clients see of this, but clients that change networks, rollbacks and the data shared by all cells still meet both versions.
Non-serving workloads
Jobs, scheduled jobs and queue consumers serve no request, so the rules above that keep a request flowing do not apply to them (R23, R24); what applies is that their work completes exactly as often as intended while cells are drained and released:
| Workload | Runs as | The rules |
|---|---|---|
| A finite job | Job with backoffLimit, activeDeadlineSeconds and ttlSecondsAfterFinished, judged by its exit status |
Idempotent, so a restart after a drain repeats nothing twice; resource requests; the same image, security and placement rules as a pod of the application |
| A scheduled job | CronJob with concurrencyPolicy: Forbid and a startingDeadlineSeconds |
Runs in the application’s home cell - in both exposures, an external application’s non-serving workloads are placed in one cell like an internal application and migrated the same way (internal zones) - so a schedule fires once per environment, never once per cell |
| A queue consumer | Deployment with at least 1 replica, scaled on queue length (KEDA), readiness and liveness probes, a graceful shutdown longer than its longest message, RollingUpdate |
May run in every cell, because the queue serialises the work; below 2 replicas it needs no disruption budget and a drain costs it a pause, not data |
During a cell drain a running job is given the drain timeout to finish; a job still running then is stopped and
restarted in the target cell by its CronJob or owner, which is why it must be idempotent. A release moves a
non-serving workload with the cell’s branch like any other manifest; the SLO gates do not see it, so a team sets an
alert on its own measure of success (last run, queue lag) in its observability directory.
Releases cell by cell
A release – a signed commit of main with any number of application changes (section 3.3) – moves through the
cells of an environment one cell at a time, with the same drain → change → test → return procedure as a
cluster upgrade (section 3.5). External applications run in
every cell behind Traffic Manager’s DNS affinity; internal applications run only in their home cell
(incoming traffic).
Either way, every interactive user switches from the old to the new version exactly once and never back:
- Stateful stage first (only if the environment has a stateful cluster): providers before consumers (section 3.3).
- Drain cell 1: disable its Traffic Manager endpoints and wait for the DNS answers to expire and the cell’s connections to drain. Its external users move to the other cells, which still run the old version, so nobody sees a change yet. Internal applications whose home is cell 1 keep serving.
- Deploy: the pipeline moves the cell’s branch (
<env>-sl-az1) to the commit; Flux rolls it out and the pipeline waits until all Kustomizations areReady. No external traffic reaches the cell during the rolling update; the internal applications of the cell are updated in place by their zero-downtime rolling update – this is the moment their users switch to the new version. - Test through the per-cell test host names (external zones) and against the cell’s gateway IPs with the applications’ real host names (internal zones): smoke and synthetic tests over the real TLS path.
- Return the cell to Traffic Manager: its external clients come back in steps, a quarter of its address blocks at a time, and this is the moment they switch to the new version. The pipeline watches the error-rate and latency SLOs per cell in Grafana Cloud (SLO gates); on a breach it drains the cell again – its external clients fall back to the old version in the other cells – and moves the branch back, which also rolls back the cell’s internal applications.
- Soak, then repeat for the next cell. After the last cell all clients are on the new version.
An external client only moves when its own cell is drained or returned. While its cell is drained it uses another cell, which runs either the old version or – if that cell was already released – the new one; when its cell returns, it gets the new version. An internal client always uses its application’s home cell. So every client switches from the old to the new version once and never back:
| Client | What it sees during a release |
|---|---|
| Internal client (corporate network, other Azure workloads) | One version at a time, except during the few minutes of the rolling update in the home cell; switches once, old → new |
| External client with a stable IP address (browser, mobile app, API client, other system) | One version at a time; switches once, old → new |
| External client whose IP address changes (e.g. a phone moving between networks) | May change cells with its address and meet the other version – covered by the compatibility rule |
| Long-lived connection (WebSocket, SSE) | Reconnects when its cell is drained or returned, or its pod is replaced, then follows the rules above |
Consequences:
- Releases are trains. Each cell costs one drain time plus rollout, tests and the return, so many changes ride in one release instead of one release per application change. An urgent fix takes the same path with a shorter soak.
- N+1 capacity is needed for releases too: while one cell is drained, the others carry its load (the same sizing as for cluster upgrades).
- The first cell is the canary for external users; for internal applications the canary is the first cell
that is home to some of them, so placing a few low-risk internal applications in every cell makes sense.
In-cell progressive delivery (Flagger with
HTTPRouteweights on the zone’s gateway) is not used for interactive applications: it splits requests inside the cell, so their clients would jump between versions again. It stays an option for back-end APIs that are strictly backwards compatible.