4. Encryption

4.1 Encryption in transit and TLS

Requirements: R12, R21, R22.

Every hop is encrypted, but each cluster type does it in the way that costs the applications least. Encryption at rest with customer-managed keys is in section 4.2.

Stateless clusters: VNet encryption + TLS-only zone gateway

• No service mesh, no mTLS, no TLS configuration in the applications. Applications listen on plain HTTP inside the cluster. Traffic between pods on the same node never leaves the host; traffic between nodes is encrypted by Azure Virtual Network encryption, enabled on every stateless spoke VNet (and the peering to its hub; Azure Firewall Premium takes part in it, Standard does not – one reason the Firewalls are Premium).
  • Requires node VM sizes that support VNet encryption (accelerated networking); because the only enforcement mode is AllowUnencrypted, an unsupported VM size would silently send clear text. The allowed VM sizes are therefore enforced with Azure Policy on the node pools’ VM sizes, and VNet encryption on the VNets.
  • VNet encryption covers VM-to-VM traffic in the VNet and peered VNets. Everything that leaves the stateless cluster (to Azure Firewall, the stateful cluster, PaaS, Internet) is TLS anyway – see below.
• Gateway API only, no Ingress. HTTP traffic into a cell is configured exclusively with the Kubernetes Gateway API (GatewayClass, Gateway, HTTPRoute; GRPCRoute where needed), standard channel only. The Kubernetes Ingress API is not used anywhere: no Ingress controller is installed (the managed NGINX of App Routing is not enabled), and Azure Policy rejects Ingress objects.
• HTTP/2 only and TLS 1.3 only, from day 1: at the edge and between a gateway and a pod. HTTP/1.1 cannot be made safe against request smuggling, and the dangerous hop is the one a reverse proxy shares between users - its pool of connections to the back ends (PortSwigger, “HTTP/1.1 must die”, 2025); accepting HTTP/2 at the edge and downgrading upstream is worse than HTTP/1.1 end to end. So:
  • At the edge the gateway offers HTTP/2 only (Traefik’s default TLS options list h2 as the one ALPN protocol) and TLS 1.3 only as the minimum version: no TLS 1.2 client to keep, no cipher-suite choices, and Go’s defaults add the hybrid post-quantum key exchange X25519MLKEM768. The policy gateway-tls reads both settings back in every gateway namespace so neither can drift (section 2.7). The residual: a client that offers only http/1.1 still gets HTTP/1.1 from Go’s server (golang/go#46310). App Routing’s Istio exposes neither setting on a listener, so Traefik is the implementation in every environment until it can.
  • Between a gateway and a pod the platform speaks HTTP/2 cleartext with prior knowledge (h2c), chosen by one field on the application’s Service port, appProtocol: kubernetes.io/h2c, which Azure Policy requires on every routed Service port (section 2.7). The application contract gains one line: an application serves h2c on its route port (most frameworks do; the onboarding agent checks the image with one curl --http2-prior-knowledge). The kubelet’s probes stay HTTP/1.1 until Kubernetes 1.38 lets a probe speak HTTP/2 (KEP-5999) - harmless, because the vulnerability is the proxy’s reused upstream connection.
  • Nothing that downgrades may stand in front of a gateway: Application Gateway v2 and Front Door speak HTTP/1.1 to their back ends.
  • One exception: Flux’s webhook receiver and GitHub’s deliveries are HTTP/1.1, behind the platform gateway alone, in cells with a public entry point.
• Two Gateway API implementations, chosen per cluster: Traefik Proxy in every environment, because App Routing cannot yet offer HTTP/2 alone and TLS 1.3 alone at the edge, and AKS App Routing’s Gateway API implementation
  • the Azure standard offering - when it can. App Routing runs a sidecar-less Istio control plane per cell that serves the GatewayClass approuting-istio and an Envoy Deployment per Gateway; it is not a service mesh, AKS upgrades it with the cluster, and it cannot run together with the Istio service mesh add-on.
  • One Gateway per isolation zone in the platform namespace <zone>-gateway (a reserved application name), its Envoy pods on the system pool (so that a zone without applications has no nodes, section 2.3), behind an internal load balancer in snet-<zone>-ilb (an external zone’s: a public load balancer, see incoming traffic). The platform owns the GatewayClass and one Gateway per zone; application teams own only HTTPRoutes in their namespaces – the Gateway API role split between cluster operator and application developer. allowedRoutes selects namespaces with platform/zone=<zone>, so a route can never attach to another zone’s gateway.
  • With App Routing the data plane is per zone but the control plane is the cell’s one istiod: if it fails, the gateways keep serving their last configuration until it is back - the cell is the bulkhead, not the zone. Neither implementation offers rate limits, request size limits or an in-gateway WAF (section 3.1). App Routing’s own Key Vault and DNS integration is not used: the certificate comes through ESO (below) and the DNS records from ExternalDNS.
• Traefik Proxy (a per-cluster flag of the cluster IaC) also costs less than istiod’s requests (1 CPU and 4 GiB per cell, not tunable – Azure/AKS#5939), a second system node in a small cell. With Traefik each Gateway gets its own Traefik Deployment and GatewayClass traefik-<gateway> on the system pool – no shared control plane, so a zone’s gateway depends on nothing of another zone’s – reading its configuration and certificate from the API server (two replicas, a PDB, externalTrafficPolicy: Local, a Cilium policy to the API server). The Gateway, its listener on 443, its certificate, the load balancer and the Cilium policies are the same for both, and so are the teams’ HTTPRoutes; the implementation-specific part is a kustomize component of the platform repository, and a change of class re-creates the Gateways (a few minutes without gateways, so the cell is drained first).
• The platform gateway, only with the Flux webhook. A cell with a public entry point may run a Gateway platform in the namespace platform-gateway with one HTTPS listener flux-webhook-<cell>.<dns zone> for the Flux webhook receivers (a per-cell flag of the cluster IaC brings it). GitHub delivers webhooks over HTTP/1.1 and the receiver speaks nothing else - the platform’s one HTTP/1.1 path, which takes only deliveries signed with the token in the platform vault. An internal cell has none - a public load balancer cannot answer behind the hub Firewall - and polls Git every minute. The public addresses are the external zones’ gateways and, where the webhook is on, the platform gateway.
• DNS names with Let’s Encrypt certificates. Each zone has its own DNS name space, e.g. *.int-fi.prd.example.com; an internal zone’s names are resolved only by its Private DNS zone in the hub (split horizon: the name has no public A record), an external zone’s are public. The zone’s gateway serves the zone’s wildcard certificate from the platform Key Vault (below). Because the certificate is publicly trusted, Internet clients and clients inside the corporate network need no private CA.
• No non-TLS traffic. The zone’s Gateway has only an HTTPS listener on 443; port 80 is not exposed at all (no HTTP→HTTPS redirect listener either), the internal load balancer has only port 443, and the NSG on snet-<zone>-ilb allows only TCP 443. Azure Policy rejects Gateway listeners with protocol HTTP, HTTPRoutes that do not attach to the HTTPS listener, hostnames outside the zone’s domain, Ingress objects, and LoadBalancer / NodePort Services outside <zone>-gateway. The gateway does not add HSTS by itself: applications add the Strict-Transport-Security header with a ResponseHeaderModifier filter on their HTTPRoute (the platform may later require it with Azure Policy).

Stateful cluster: the application terminates TLS

• No Gateway API implementation and no Ingress – the CRDs are not installed and Azure Policy rejects Ingress, Gateway and HTTPRoute objects. Fewer moving parts in the cluster that is hardest to upgrade.
• Azure CNI (VNet-integrated): pod IPs are routable in the backend spoke, so NSGs, the Firewall and Cilium policies see real addresses.
• Each application is published with a LoadBalancer Service that must be internal (service.beta.kubernetes.io/azure-load-balancer-internal: "true"), placed in its zone’s snet-<zone>-ilb (…-internal-subnet annotation) and use externalTrafficPolicy: Local; NodePort and public load balancers are rejected by Azure Policy.
• The application handles TLS itself (TLS 1.3 only, the same rule as the gateways’), which is an onboarding requirement for the stateful cluster: it terminates TLS on its listener and reloads the certificate on rotation. Most products that end up here (databases, brokers, search engines) support this natively.
• The application uses the zone’s gateway certificate (below) and is reached under a name in the zone’s domain, e.g. app-x.int-fi.prd.example.com.
• Clients in the stateless clusters connect with TLS and verify the name; plain-text ports are not allowed by the Firewall rules between frontend and backend.

Certificates issued centrally, distributed through the platform Key Vault

No cluster issues certificates – there is no cert-manager or other ACME client in any cluster, so no cluster needs Internet access to Let’s Encrypt or write access to DNS.

• One platform Key Vault per environment and exposure. kv-<env>-<exposure>-platform – kv-<env>-platform for short, section 1.2; the real names are kv-<org>-<region>-platform – (RBAC permission model, public access disabled, purge protection) exists once per environment and exposure, is created by the environment IaC module and is shared by all clusters of that exposure. It holds all certificates of the exposure’s zones – the zone vaults hold none – plus the renewal job’s ACME account key, and the customer-managed keys of the exposure’s clusters (section 4.2). It has one private endpoint per environment in the shared subnet snet-shared-pe, reached from every cell through the hub (picture 2.1).

• Certificates per environment. Each environment gets its own certificates in its own vault; nothing is copied between environments, and a dev or acc identity can never read a prd key:

  Environment	Platform Key Vaults	Certificates (one wildcard per isolation zone, plus the platform gateway’s)
  dev	kv-dev-int-platform, kv-dev-ext-platform	int-fi-gateway--wildcard (*.int-fi.dev.example.com) and int-se-gateway--wildcard in the internal vault; ext-fi-gateway--wildcard, ext-se-gateway--wildcard and platform-gateway--wildcard in the external one
  acc	kv-acc-int-platform, kv-acc-ext-platform	the same names for *.<zone>.acc.example.com
  prd	kv-prd-int-platform, kv-prd-ext-platform	the same names for *.<zone>.prd.example.com

  The certificate names are identical in every environment, so the fleet manifests (SecretStore, ExternalSecret) need only the vault URL of the cluster’s environment and exposure, substituted from cluster-settings. The names follow the vault’s <namespace>--<secret> convention (section 2.5): the zone’s gateway namespace <zone>-gateway owns <zone>-gateway--wildcard. An application that must not share the zone key gets its own certificate <namespace>--<name> in the same vault.

• One renewal job per environment (a pipeline or a Container Apps job): its identity holds Key Vault Certificates Officer and Key Vault Secrets Officer on that environment’s platform vault and DNS Zone Contributor on the validation zone, to which the zone names are delegated by CNAME, so it cannot change any other record. The ACME account key is the secret letsencrypt-account-key, outside every namespace’s prefix. It renews when less than 30 days are left, and Key Vault CertificateNearExpiry events alert the platform team if renewal fails. The jobs are rolled out dev → acc → prd like any other platform change.
• All clusters of an environment use the same certificate: the zone gateways in sl-az1, sl-az2, sl-az3 and the TLS-terminating applications in sf. A rebuilt cluster needs no new certificate, and Let’s Encrypt rate limits are never an issue (a handful of certificates per environment).
• Access: the same prefix share as every other secret. The gateway namespace <zone>-gateway is an ordinary Key Vault share of the platform vault (section 2.5): its reader identity id-<region>-<env>-shared-kv-<zone>-gateway, federated with the namespace’s key-vault ServiceAccount, holds Key Vault Secrets User with the ABAC condition on the <zone>-gateway-- prefix, so it reads the zone’s wildcard and nothing else; only External Secrets Operator uses it. Stateful applications and any application with a client or server certificate get the same kind of share on the platform vault. Key Vault exposes a certificate together with its private key as a secret of the same name, which is what ESO reads. Neither the gateway nor any application calls Key Vault itself. No identity has a data-plane role on the vault scope except the renewal job and the platform team (PIM); the clusters’ control plane identities have a scoped role on their own key, and Key Vault Contributor (management plane only) where KMS needs it (section 4.2). The shares are created by the environment IaC module, so a certificate can be added before it exists; until the job’s first run the gateway reports InvalidCertificateRef and answers nothing.
• Delivery and rotation with the zone’s External Secrets Operator (the Secrets Store CSI driver is disabled): an ExternalSecret reads the certificate’s secret object (PKCS#12) through the namespace’s platform-vault SecretStore, and its template unpacks it (pkcs12cert, pkcs12key) into a kubernetes.io/tls Secret gateway-tls (tls.crt with the chain, tls.key). The refresh interval is 1 hour – a renewed certificate is in every cluster well within the 30-day renewal margin.
  • The zone’s gateway: the ExternalSecret in <zone>-gateway writes the Secret that the zone’s Gateway listener references; istiod hands the changed Secret to the gateway’s Envoy pods, and Traefik watches it itself – both reload without a restart.
  • Applications: the Secret is mounted as PEM files (no subPath) and the application reloads them on change.
• Things to be aware of:
  • The parent domain must be a registered public domain – suffixes like .internal, .local or .corp cannot get Let’s Encrypt certificates.
  • Certificates appear in public Certificate Transparency logs; the per-zone wildcard keeps application names out of them.
  • The wildcard private key is shared by everything in the zone that needs a certificate. Its blast radius is one zone of one environment, and it is renewed every 60 days.
  • The platform vault is shared by all zones of an environment, so zone separation for certificates rests on the prefix conditions (and Cilium egress policy) rather than on a separate vault – the same model that separates the applications’ secrets inside a zone vault.
  • Egress acme-v02.api.letsencrypt.org is allowed only for the renewal job, not for any cluster.

Service mesh or not

The stateless clusters run no service mesh (R22). The question comes back with every new Istio or AKS feature, so this is the reasoning.

What a mesh offers, and what the platform does instead:

What a mesh costs: a data plane in every pod’s path (sidecars, or ambient’s per-node ztunnel and waypoints) that the platform runs, upgrades and debugs; a control plane that cannot run next to App Routing’s istiod; a second policy language next to Cilium policies and HTTPRoutes; and the largest cloud-specific component in the cluster.