2. Inside a cluster

2.5 Workload identity first, Key Vault secrets only when needed

Requirements: R3, R9, R10, R11, R12.

Rule: an application authenticates with its workload identity wherever the target supports Microsoft Entra ID; a secret is used only when there is no other way. No passwords, connection strings with keys, SAS tokens or storage account keys for Azure services.

Workload identity per application

• Each application has, per isolation zone and environment, its own user-assigned managed identity (id-<zone>-<app>), created by IaC together with the namespace. Its Kubernetes ServiceAccount carries the azure.workload.identity/client-id annotation; pods get a short-lived projected token that is exchanged for an Entra ID token – nothing long-lived is stored anywhere.
• Federated credentials per cluster: every cluster has its own OIDC issuer, so the identity has one federated credential per cluster of the environment (sl-az1, sl-az2, (sl-az3), sf) for subject system:serviceaccount:<namespace>:<app>. A rebuilt stateless cluster gets a new issuer; the IaC that creates the cluster also updates the federated credentials (a managed identity allows at most 20). When an identity would need more - a platform identity federated to every cluster of a large fleet - the path is AKS’s identity bindings (GA since September 2026): one federated credential per identity, the clusters bound to it by AKS; the applications’ identities, one per namespace and environment, stay far below the limit and keep the plain credentials.
• The identity gets data-plane roles directly on the Azure resources of its zone, never on other zones:

“Local auth disabled” is enforced with Azure Policy on the resources, so a key-based fallback cannot be switched on later. Application configuration contains only endpoints and client IDs, which are not secrets. The table lists the Azure services; the ones without an equivalent in every cloud (Azure SQL, Service Bus, Cosmos DB’s NoSQL API) are available only as a documented cloud-pinning exception (section 1.3).

• Platform components use workload identity too: Flux (provider: github on its Git sources: a GitHub App’s short-lived installation token, no stored key), the stateful deployment pipeline (workload identity federation, no client secret), External Secrets Operator (below), external-dns. Unlike applications, platform components may use Key Vault directly with their own identity (below). Images are pulled with the kubelet identity, which has only AcrPull on the environment’s ACR – no imagePullSecrets.

Secrets only when needed

A secret is allowed only when the target cannot use Entra ID: third-party APIs and SaaS keys, partner systems and legacy protocols. TLS certificates are not in the zone vaults; they live in the environment’s platform Key Vault (section 4.1). Such a secret always lives in the zone’s Key Vault – never in Git, Helm values or a Kubernetes Secret created by hand.

Each zone has its own Key Vault per environment (kv-<env>-<zone>, RBAC permission model, public access disabled), shared by all clusters of the environment, with one private endpoint in the environment’s shared snet-shared-pe, reached from the cells through the hub Firewall (section 3.1). The tenant of a vault is the namespace: every secret a namespace owns is named <namespace>--<secret> (int-fi-example--api-key), and a namespace name may not contain --. Inside a zone’s vault, applications share the vault but are separated with Azure ABAC conditions:

• The namespace’s secret reader identity id-<region>-<env>-shared-kv-<namespace> (see below – not the application’s own identity) gets Key Vault Secrets User on the zone vault with condition @Resource[Microsoft.KeyVault/vaults/secrets:name] StringStartsWith '<namespace>--'.
• Application pipelines (writers) get Key Vault Secrets Officer under the namespace’s write condition - the resource name for existing secrets (get, update, delete, recover, purge), the request name for new ones (setSecret, restore); rotation is owned by the application team (expiry dates set, Event Grid SecretNearExpiry alerts).
• The namespace’s Entra ID groups get what their Kubernetes role gets on Secrets, under the same prefix conditions: admin Key Vault Secrets Officer, edit the custom Key Vault Secrets Editor role (writes and reads attributes, never a value), view nothing (section 2.6).
• The platform viewers (section 2.6) list every secret’s name and attributes, never a value.

Key Vault ABAC is preview and conditions only on the vault and secret name, so the naming convention <namespace>--<secret> is mandatory and enforced by the platform. ABAC is not the only boundary between two applications of a zone: each has its own reader identity and its own SecretStore, and admission rejects an ExternalSecret that names another application’s prefix (section 2.7), so a change in the preview cannot open a path on its own; the fallback, one vault per application, is a per-application flag.

Soft multi-tenancy

A zone’s applications share one vault, and the separation between them is in the data plane, not in the resource: this is soft multi-tenancy. What each tenant – a namespace, its identity and its groups – can do:

The conditions cover readMetadata too, and Key Vault evaluates a condition on a list call against the whole collection, which has no name, so no tenant can list the vault: each works with its own secrets by name, which is all External Secrets Operator does, and nobody can create a secret outside the naming convention (the condition on a new secret is on the name in the request). Tools that browse the vault – the Azure portal among them – are for the platform viewers, whose role lists every name; so names must never carry secrets. The isolation tests sign in as a member of each group and as the namespace’s identity and check every row of the table (section 0.3). The hard alternative – one vault per namespace – stays a per-namespace option.

The same pattern applies to other shared PaaS services: one instance per zone (or environment), a data-plane role per namespace scoped below the instance - a database per namespace on a PostgreSQL or SQL server, a Cosmos DB database, a storage container, a Service Bus queue or topic - the namespace’s workload identity and groups as the only principals, and the names of other tenants’ objects visible at most to the platform viewers. Where a service cannot scope its data plane below the instance, or a tenant needs its own keys, network or performance isolation, the namespace gets its own instance – hard multi-tenancy, a decision per onboarding.

Delivered only by External Secrets Operator

Rule: no application reads Key Vault itself. Secrets and certificates reach application pods only as Kubernetes Secrets written by External Secrets Operator (ESO). The rule applies to application namespaces; platform components are the exception (below). The Secrets Store CSI driver is not used: the AKS Key Vault secrets provider add-on (azure-keyvault-secrets-provider) is disabled by the cluster IaC on every cluster, so its CRDs are not installed, and the stateless storage rule refuses inline csi volumes (section 2.7); an Azure Policy denies enabling the add-on and an admission rule refuses SecretProviderClass objects.

• One ESO controller per isolation zone, in the platform namespace <zone>-secrets (a reserved application name, like <zone>-gateway), on the system pool – so that a zone without applications has no nodes (section 2.3) – and deployed by Flux (stateless) or the pipeline (stateful). Each instance has its own controllerClass (<zone>) and reconciles only the stores of its zone; the CRDs and the conversion webhook are installed once per cluster. Its path to Key Vault is the zone’s own flow ⑤ (⑤b to the platform vault, section 2.4), so a controller can never reach another zone’s vault and a compromised controller exposes one zone at most. It writes Secrets and requests ServiceAccount tokens only through RoleBindings in its own zone’s namespaces, not through the chart’s cluster-wide role (CIS benchmark). The CRDs, the conversion webhook and the certificate controller are installed once per cluster by the platform release in the namespace external-secrets (controllerClass: platform), which also serves the platform namespaces’ own stores.
• A secret reader identity per namespace (id-<region>-<env>-shared-kv-<namespace>, created by IaC with every managed namespace), federated with one credential aks-<cluster> per cluster, as in the picture. Application pods may not run as key-vault (Azure Policy). The teams’ edit and admin roles do include serviceaccounts/token, so their members could obtain its token – which gives them nothing they do not already hold, since those groups manage the namespace’s secrets in the vault themselves; no workload can.
• One SecretStore named key-vault per application namespace, generated from the onboarding by a platform preset: provider azurekv, vaultUrl of the own zone vault (the gateway and platform namespaces point theirs at the platform vault, section 4.1), authType: WorkloadIdentity with serviceAccountRef: key-vault, controller: <zone>. Azure Policy rejects any other store configuration. Applications write only ExternalSecrets that reference their own namespace’s store; ClusterSecretStore, ClusterExternalSecret and PushSecret are platform only. An application can therefore only ever pull with its own reader identity, and Key Vault ABAC still decides which secret names that is – application A’s ExternalSecret for b-… fails with 403.
• No other way in. Application identities have no Key Vault role at all; no Cilium policy of an application namespace reaches a vault (only <zone>-secrets and the platform namespaces below do), the Firewall’s rule for the vaults’ private endpoints takes the cells’ system subnets as its source, and Azure Policy rejects an application policy that opens egress out of the cluster (section 2.7). In application namespaces, admission accepts Secrets only from the zone’s ESO controller (plus Helm release Secrets), so a secret in Git, Helm values or created with kubectl is rejected as before. Only the environment IaC module assigns Key Vault data-plane roles; an acceptance test and the security agent (section 6.1) check that no application identity holds one.
• Consumption and rotation. Applications mount the resulting Secret as a volume (preferred: the kubelet updates the files after a rotation; no subPath) or as environment variables (picked up only on the next rollout). refreshInterval is 1 hour by default, shorter where a rotation must take effect faster; the SecretSynced condition and ESO’s metrics alert the platform and the application team when a sync fails.

Platform components may use Key Vault directly

Platform components in platform namespaces (<zone>-secrets, external-secrets, flux-system, infra-observability) are owned, reviewed and upgraded by the platform team, so they are not forced through ESO. Such a component may read Key Vault directly with its own workload identity, with a data-plane role scoped as narrowly as Key Vault allows (a single certificate or secret where possible), declared in the IaC module and reviewed with the platform change, and with the vault in its namespace’s platform-owned egress policy. The Secrets Store CSI driver stays disabled for them too.

• The zone’s gateway uses ESO exactly like an application. <zone>-gateway has the standard key-vault ServiceAccount, federated to its reader identity id-<region>-<env>-shared-kv-<zone>-gateway, which has Key Vault Secrets User on the platform vault under the prefix <zone>-gateway-- - the zone’s certificate is the secret <zone>-gateway--wildcard, the gateway namespace’s share like any other (section 4.1). The gateway pods have no Azure identity at all (with App Routing istiod hands them the Secret, Traefik reads it itself; App Routing’s own Key Vault integration through the Secrets Store CSI driver is not used), their egress policy names no vault, and the ESO rules of section 2.7 apply to <zone>-gateway as to any application namespace.
• External Secrets Operator itself, Flux and the monitoring components follow the rules above: own identity, own narrow roles, egress policy owned by the platform.
• Grafana Alloy cannot read Key Vault, so its Grafana Cloud credential is delivered by the platform ESO controller in external-secrets through flux-system’s share of the platform vault (the secret flux-system--grafana-cloud, read by flux-system’s reader identity under its prefix, section 6.2).

Trade-offs of delivering secrets as Kubernetes Secrets instead of CSI file mounts:

• The values are stored in etcd, encrypted with the cluster’s customer-managed KMS key (section 4.2), and readable by anyone with get secrets in the namespace: only the application team in its own namespaces, the platform team through PIM; the AI agents’ Azure Kubernetes Service RBAC Reader role does not include Secrets.
• ESO becomes a critical platform component (one controller per zone and cluster, upgraded by the platform). In return a Key Vault or Entra ID outage no longer stops pods from starting – the Secret is already in the cluster, whereas a CSI mount fails at pod start.