2. Inside a cluster

2.7 Policy enforcement with the Azure Policy add-on

Requirements: R3, R8, R11, R14, R15, R22, R24, R29.

The AKS-native Azure Policy add-on is the single policy engine, enabled by the cluster IaC in every cluster. It runs Gatekeeper, but AKS installs and upgrades it, and the policies are ordinary Azure Policy definitions and assignments – the same tool that governs the Azure resources (cluster settings, allowed VM sizes, VNet encryption, local auth disabled, private endpoints only). No Gatekeeper is installed by Flux.

• Assignments per environment and exposure subscription (sub-aks-dev-int, sub-aks-dev-ext, … sub-aks-prd-ext), so a new or rebuilt cluster gets every policy automatically; the zone mutations and validations of a subscription are made for its exposure’s zones only. Compliance of all clusters is visible in one place (Azure Policy, and the company’s security tool - not Microsoft Defender).
• Built-in definitions first: the Kubernetes pod security restricted initiative, allowed images, required probes and resource requests, and deployment safeguards.
• Custom definitions (Microsoft.Kubernetes.Data mode) for the platform’s own rules, with the constraint or mutation template embedded as Base64 in the definition. Templates are kept in the IaC repository and tested in CI with gator verify. The templates are Rego; CEL-based definitions (generated as Kubernetes ValidatingAdmissionPolicy, evaluated in-process) are the choice for new validations once Azure Policy supports them for the rule in question.
• Rollout dev → acc → prd: definitions are versioned; a new or changed definition is assigned with effect audit in dev, then deny, then promoted to acc and prd with the rest of the platform change.
• Exemptions expire. An Azure Policy exemption always has an owner, a reason, a ticket and an expiresOn date (at most 6 months). The security agent reports exemptions that expire within 30 days, and renewing one is a new pull request in the high tier (section 6.1). Exceptions should stay visible and a little uncomfortable, not pile up silently.
• Continuous evidence, not a one-time hardening. The compliance state of the assignments is the drift signal. Next to it, the CIS Kubernetes benchmark of the nodes (kube-bench, run as one read-only Job per node, weekly in acc and prd) and the security tool’s posture score per environment go to Grafana Cloud. A drop in either raises an alert to the security agent, which matters more than the absolute score. The CIS AKS benchmark and the NSA/CISA Kubernetes hardening guidance are the reference controls, and the checklist is repeated with every Kubernetes minor upgrade, because new minors can change defaults. The full benchmark – nodes (kube-bench), Kubernetes policies (Kubescape) and the cluster’s Azure settings – also runs as a pipeline stage after every cell change, with every accepted failure listed with its reason (section 0.3).
• Platform namespaces: kube-system, gatekeeper-system and azure-arc are excluded by the definitions’ defaults; the assignments exclude the platform’s own namespaces (flux-system, external-secrets, external-dns, platform-gateway, infra-observability, aks-command, the AKS add-on namespaces, <zone>-gateway and <zone>-secrets) from the application rules through the namespace exclusion parameters; the secrets rules keep <zone>-gateway in. The zones’ own platform namespaces, <zone>-gateway (the zone’s gateway) and <zone>-secrets (External Secrets Operator), run on the system pool so that a zone without applications has no nodes (section 2.3); they are excluded from zone pinning and the other application rules, and their policies are the platform’s own. The <zone>-gateway namespaces enforce the Pod Security Standard baseline and only audit and warn on restricted: App Routing’s gateway pods set no seccomp profile, and their securityContext cannot be customised.
• Mutations fill in, validations decide. A mutation only adds what a pod leaves out, so a pod that states its placement differently is not changed but reported (audit) or refused (deny) by the matching validation. A recorded exception is visible the same way: a pod labelled platform/zone-placement: "off" is skipped by the zone mutation and stays non-compliant in the zone validation.
• Which clusters a rule reaches is set by the assignment, not the definition: a Microsoft.Kubernetes.Data rule can test only a cluster’s type, location and tags. The stateless rules leave out the stateful cluster with notScopes, and the stateful rules leave out the stateless cells. Custom mutations cannot be parameterized, so the zone mutations are one definition per zone, generated by the IaC.

Limits of the add-on and how the design handles them:

Day-1 set. The first cluster of any environment is accepted only with these rules in deny (they are the acceptance checklist of a new cell): zone placement from the namespace (mutation and validation), platform/arch and platform/capacity mutations, images by digest from the environment’s ACR, Secrets only through ESO and no Secrets Store CSI, no application egress policy out of the cluster, PersistentVolumeClaim rejected in stateless cells, Gateway and HTTPS-only listener rules, h2c on every routed Service port, LoadBalancer and NodePort only in <zone>-gateway, no Service.spec.externalIPs (the built-in “allowed external IPs” rule with an empty list: deprecated in Kubernetes 1.36, removal planned for 1.43, and the oldest service hijack in the book - CVE-2020-8554), requests and probes set, the restricted pod security standard, and the cluster-level rules (Cilium, Azure Linux, local accounts and SSH off, CMK). Everything else in the table starts in audit and is promoted rule by rule.