1. Start here
1.2 How the design meets the requirements
Requirements: R1, R2, R3, R4, R5, R6, R7, R8, R9, R10, R11, R12, R13, R14, R15, R16, R17, R18, R19, R20, R21, R22, R23, R24, R25, R26, R27, R28, R29, R30, R31, R32, R33, R34.
The requirements are owned by humans; this page, like the rest of the design, shows how the design meets each of them (how we work). The acceptance criteria prove it: the requirement and isolation tests are tagged with the requirement number they cover and run after every change of a cell.
| # | How the design meets it |
|---|---|
| R1 | A subscription per environment and exposure (sub-aks-<env>-int, sub-aks-<env>-ext), each with its own clusters, VNets, Key Vaults, ACR and policy assignments; zones int-* only in internal clusters, ext-* only in external ones (section 3.1) |
| R2 | Separate isolation zones per country (*-fi, *-se, …) inside the clusters of each exposure |
| R3 | Every zone has its own Key Vault, subnets (NSG + route table) and AKS node pools (section 2.3); certificates are the one shared exception, kept in the per-environment platform Key Vault with a role assignment per certificate |
| R4 | Deny-by-default on three layers: NSG, Azure Firewall (each zone’s route table takes zone-to-zone traffic to it, pods included because they carry their zone’s addresses), Cilium network policy (picture 2.4); an application reaches another zone’s application only as a client of its published entry point, with a Firewall rule like any external API (section 2.1) |
| R5 | A zone is a slot of every cell’s address space with the same index in every cell; a new country declares two zones (one per exposure) and gets its subnets, NSGs, node pools, policies and Firewall rules without re-addressing or routing changes (adding a country) |
| R6 | Azure CNI Pod Subnet powered by Cilium in every cluster: Cilium’s layer 3 and 4 policies for the zone baseline and namespace isolation, kube-proxy replaced; Advanced Container Networking Services (FQDN filtering, Hubble, eBPF host routing) is not used (section 5.1) |
| R7 | A private cluster over Private Link, the endpoint in the cell’s system subnet and no public endpoint; reached only from the management network through the hub Firewall (section 2.2) |
| R8 | Pod labels platform/arch and platform/capacity, turned into node selection by Azure Policy mutation; per zone an on-demand and a tainted spot pool for both architectures with fallback to on-demand; defaults amd64 on-demand; stateful cluster on-demand amd64 only (section 2.3) |
| R9 | Per-application workload identity with Entra ID auth to all Azure services, local auth disabled by Azure Policy; Key Vault secrets only for targets without Entra ID (section 2.5) |
| R10 | Per-application secret reader identity, usable only by External Secrets Operator, with secret-name-prefix conditions (picture 2.5) |
| R11 | One ESO controller per zone reads Key Vault on behalf of the applications; application identities have no Key Vault role, application pods cannot use the reader identity or reach a vault, only ESO may write Secrets in application namespaces, and the Secrets Store CSI driver add-on is disabled; the zone’s gateway uses ESO like an application, other platform components may use Key Vault directly with their own narrow roles (section 2.5) |
| R12 | One shared platform Key Vault per environment and exposure (kv-<env>-<exposure>-platform) holds all certificates of that exposure’s zones; a renewal job per environment issues one certificate per zone into the vault of the zone’s exposure, and External Secrets Operator in every cluster of the exposure reads it with plain Azure RBAC scoped to the individual certificate (section 4.1) |
| R13 | Namespace per application, default-deny policies (picture 2.6); egress out of the cluster only through the hub Firewall’s rules for the zone (R29) |
| R14 | Every namespace carries platform/team, platform/cost-center and the annotation platform/contact, set at onboarding and required by Azure Policy; OpenCost splits each cell’s cost by namespace and so by team and cost centre, in Grafana Cloud; a monthly budget per environment subscription alerts the owners at 80 % and 100 % (section 2.6) |
| R15 | Azure Policy add-on for AKS in every cluster, assigned per environment subscription; the same Azure Policy also governs the Azure resources (section 2.7) |
| R16 | Applications, their manifests and the platform’s in-cluster configuration use only open standards, and data services open protocols; cloud-specific choices live only in the landing zone, the cluster IaC and cluster-settings (section 1.3) |
| R17 | Per exposure, stateless clusters in the frontend network and one optional stateful cluster in the backend network, built only when a workload requires it (picture 3.1) |
| R18 | Placement order stateless cluster → Azure PaaS → stateful cluster; the stateful cluster needs a documented exception (where does a workload run) |
| R19 | Backend spoke with PaaS private endpoints; network isolated AKS (outbound type none), no public IPs, Firewall deny-all for backend prefixes except the two allow-listed destinations (flows) |
| R20 | Fixed-size node pools per AZ 1/2/3 (no autoscaling), ZRS storage, Premium tier with Long Term Support |
| R21 | Azure CNI (VNet, dynamic pod IP allocation) powered by Cilium; applications are published with internal LoadBalancer Services and terminate TLS themselves, TLS 1.3 only like the gateways (section 4.1) |
| R22 | Azure CNI Pod Subnet powered by Cilium, every pod with its zone’s address, Azure Virtual Network encryption between nodes, a Gateway API implementation per cluster with Let’s Encrypt certificates distributed through Key Vault, HTTPS-only listeners offering HTTP/2 alone and TLS 1.3 alone, and h2c from every gateway to every pod; an application is reached from outside its namespace only through the gateway, also by the zone’s other applications, and uses any protocol inside it (section 2.6, section 4.1) |
| R23 | sl-az1, sl-az2 (+ optional sl-az3); external applications in every cell behind a zone-redundant traffic layer, internal applications in one home cell each, migrated to another cell on failure; policy-enforced replicas ≥ 2 for serving applications, non-serving workloads (jobs, scheduled jobs, queue consumers) with the copies their work needs in one home cell per environment (section 3.4), HPA/KEDA, VPA in-place right-sizing (section 2.3), the cluster autoscaler on every zone pool |
| R24 | Readiness and liveness (+ startup) probes enforced by admission policy on every long-running workload, a finite job judged by its exit status (section 2.7); applications behind a gateway answer HTTPS GET /healthz with 200 only while they can serve, probed per cell by Traffic Manager (external zones) and Grafana Cloud synthetic monitoring, which leave out a cell whose probe fails (section 3.1, section 6.2) |
| R25 | 2 exposures × (2 + 2 + 3) = 14 clusters (20 with the stateful clusters) from the same IaC modules, all in-cluster state from one Git repository: FluxCD in the stateless clusters, CI/CD pipeline in the stateful cluster (picture 3.2, picture 3.3) |
| R26 | Policy-enforced rollout guardrails, releases one cell at a time with DNS affinity (external zones), zero-downtime migration of internal applications between cells, pipeline-driven stateful rollout, drain-and-upgrade per stateless cluster, PDB-guarded per-AZ upgrade of the stateful cluster (section 3.4, section 3.5) |
| R27 | Every cluster starts on 1.36, the platform floor. Stateful: created on an LTS minor and kept on it until 6 months before its LTS ends, nodes patched in place where possible. Stateless: sl-az1 on the second latest minor with patches after 1 week per environment, sl-az2 one minor behind with 3 weeks per environment; node images pinned per cell (update policy) |
| R28 | Keys in the platform vault of the cluster’s exposure: per cluster a KMS etcd key and an OS-disk key (AKS supports one disk encryption set per cluster, not per node pool); in the stateful cluster one key per isolation zone for its persistent volumes; encryption at host on every node (section 4.2) |
| R29 | The hub Firewall allows a zone only the destinations of its per-application rule files, proposed by the egress agent from the denied flows and merged by a human (section 5.1); Azure Policy rejects an egress policy in an application namespace that would open anything out of the cluster, so the Firewall is the one decision (section 2.7) |
| R30 | Agents with read-only identities per agent and environment open pull requests; a merge gate owned by humans computes a risk tier from the plan and requires checks and 0–2 human approvals; only the existing deployment identities write; runtime guardrails unchanged (section 6.1) |
| R31 | One Grafana Cloud stack per environment; Grafana Alloy on every node collects node-locally and sends through Azure Private Link, also from the Internet-less stateful cluster; every user of a stack sees all of its data, so nothing the users may not see is sent; Azure-produced logs stay in Log Analytics and are read in place (section 6.2) |
| R33 | The non-negotiable rules are the first page of the onboarding request and are applied by admission, CI and the agents, never by a person: the update policy’s timetable, platform-owned base images with the base image agent rebuilding every application from main and the release agent promoting the digest at tier low on the automated gates, an image past its deadline refused at admission, exceptions with an expiry (section 1.5, patching) |
| R34 | A preview namespace per pull request and a sandbox namespace per team inside the dev cells, deployed from the pull request’s branch outside the cell branches, sharing the application’s identity, vault prefix, rule file and groups, under every policy of dev, removed with the pull request (section 1.5, section 3.2) |
| R32 | Every PaaS data store, the stateful cluster’s persistent volumes and the secrets that cannot be issued again are backed up into the paired region, in locked immutable vaults and storage accounts of a backup subscription per environment; clusters, Kubernetes objects and images are rebuilt from IaC and Git; region recovery declared by humans and drilled in acc (section 6.3) |
Exposure = internal (int) or external (ext): its own subscription and clusters per environment (R1).
Isolation zone = one type = one combination of exposure × country, inside the clusters of its exposure:
| Zone | Exposure (subscription, clusters) | Country | Node pools (stateless) | Node pools (stateful) | Key Vault (one per environment) | Subnets |
|---|---|---|---|---|---|---|
int-fi |
internal (sub-aks-<env>-int, aks-<env>-int-*) |
FI | intfi, intfispot, intfiarm, intfiarmspot |
intfiz1–z3 |
kv-<env>-int-fi |
snet-int-fi-* |
int-se |
internal (sub-aks-<env>-int, aks-<env>-int-*) |
SE | intse, intsespot, intsearm, intsearmspot |
intsez1–z3 |
kv-<env>-int-se |
snet-int-se-* |
ext-fi |
external (sub-aks-<env>-ext, aks-<env>-ext-*) |
FI | extfi, extfispot, extfiarm, extfiarmspot |
extfiz1–z3 |
kv-<env>-ext-fi |
snet-ext-fi-* |
ext-se |
external (sub-aks-<env>-ext, aks-<env>-ext-*) |
SE | extse, extsespot, extsearm, extsearmspot |
extsez1–z3 |
kv-<env>-ext-se |
snet-ext-se-* |
Adding a country adds two zones (int-xx to the internal clusters, ext-xx to the external ones) following the same pattern, online (adding a country).
Next to the zone vaults, every environment has exactly one shared platform Key Vault per exposure:
| Key Vault | Exists | Holds | Access |
|---|---|---|---|
kv-<env>-<zone> |
Once per zone and environment, in the zone’s exposure subscription; its private endpoint in the environment’s snet-shared-pe |
Application secrets of that zone, named <namespace>--<secret> |
Read by the zone’s External Secrets Operator with one reader identity per namespace, written by the namespace’s edit / admin groups and pipelines - all under the namespace’s prefix by RBAC + ABAC (section 2.5); nobody lists |
kv-<env>-<exposure>-platform |
Once per environment and exposure (kv-dev-int-platform, kv-dev-ext-platform, …) |
The gateway certificates of the exposure’s zones (<zone>-gateway--wildcard, platform-gateway--wildcard), the platform namespaces’ own secrets under their prefixes (flux-system--…), the renewal job’s ACME account key, the image-signing certificate and the customer-managed keys of the exposure’s clusters (key-<cluster>-kms, key-<cluster>-disk, key-sf-pv-<zone>) |
Plain Azure RBAC: the gateway and platform namespaces read their own prefix like any namespace share; the keys and the signing certificate are assigned at single-object scope (section 4.1, section 4.2) |
Where another section says kv-<env>-platform, it means the platform vault of the exposure in question.
Colour legend in all pictures: grey = control plane / platform / PaaS, blue = internal, orange = external, purple = hub / management, green = identity, teal = stateless cluster (cell), pink = stateful cluster, yellow = notes / policy, red dashed = blocked.
Cells and bulkheads
The design uses two isolation patterns at two levels, and this document uses their names:
| Term | In this design | Pattern | Limits the impact of |
|---|---|---|---|
| Exposure | Internal or external: a subscription per environment with its own cells, stateful cluster, VNets, vaults, ACR and policies | Hard boundary (subscription and cluster) | A compromise of an Internet-facing cluster or subscription: it cannot reach the internal workloads’ control plane, nodes, identities or secrets |
| Cell | One stateless cluster of an exposure (aks-<env>-<exposure>-sl-az<N>) with its own VNet, all of it in one availability zone: an external cell runs every external application, an internal cell the internal applications whose home it is |
Cell-based architecture | An availability zone outage, a cluster failure, a bad release or a bad cluster upgrade: changes reach one cell at a time |
| Cell router | External zones: Azure Traffic Manager answers each client’s DNS query with one cell, by the client’s network (active-active). Internal zones: the hub Private DNS, kept by ExternalDNS, points each application’s name to its home cell (incoming traffic) | Cell-based architecture | – (zone-redundant, outside the cells) |
| Isolation zone | One exposure × country (int-fi, ext-se, …) with its own address space, subnets, nodes, Key Vault and policies, present in every cell of its exposure |
Bulkhead | A noisy, failing or compromised workload of one zone: it cannot use another zone’s nodes, network or secrets |
| Shared tier | The hubs (one per exposure with its own Firewall; gateways, DNS), Azure PaaS, ACR, Key Vaults, the optional stateful cluster | – | Not split into cells; zone-redundant instead, and changed with extra care |
Two differences from the classic cell-based architecture:
- Every cell serves every external user. For external zones the cells are full replicas, aligned with availability zones (as in the AWS guidance for EKS, one cluster per availability zone); users are not partitioned between cells by tenant or customer. The cell router only pins an external user’s session to a cell (DNS affinity); internal applications run in one home cell each and are moved between cells by a migration workflow (internal zones).
- The data is shared. All cells use the same PaaS services and stateful cluster, so a cell isolates compute and releases, not data. Data changes therefore follow the compatibility rule.
In this document “zone” means an isolation zone, unless it says availability zone (AZ) or zone-redundant.
The canonical names are: a cell is sl-az<N> in prose, and its resources follow the cell’s name
<env>-<exposure>-stateless-az<N>: the cluster aks-<cell> (aks-prd-int-stateless-az1), its VNet vnet-<cell>,
its NSGs nsg-<cell>-<subnet>, its route table rt-<cell>; the stateful cluster is sf in prose and
aks-<env>-<exposure>-stateful; exposures are int and ext; zones are int-fi, ext-se, and so on. Key Vaults
have globally unique names, kv-<org>-<region>-<zone> and kv-<org>-<region>-platform; where this document writes
kv-<env>-<zone> it means the zone’s vault of that environment. These are example names: an estate names its
resources by its landing zone’s own convention, which is not published - a
naming policy of a template per resource type over the location, the exposure and the environment, whose defaults are
the names above, kept outside Git.