6. Operations

6.3 Backup and region recovery

Requirements: R32.

The platform survives the loss of an availability zone with its cells and zone-redundant services (section 3.1). To survive the loss of the whole Azure region, every piece of persistent data is backed up immutably into the region’s Azure paired region. Nobody can change or delete those copies before their retention ends: not the platform team, not an Owner, not a compromised pipeline or agent.

Only persistent data is backed up, never clusters (section 3.2: clusters are cattle).

Backup is built with the stateful cluster and stays open until then on purpose: a stateless cell holds no data and is rebuilt from IaC and Git in a disaster, and the PaaS pattern below is applied with the first data store an application onboards.

What is backed up and what is rebuilt

Every data store is named at onboarding. The onboarding request lists each data store of the application with its type, the backup mechanism from the table below, the RPO and RTO the business expects of it and the team that owns its restore; the onboarding agent turns the list into the Azure Policy-checked backup posture and a row in the environment’s dependency inventory in Grafana. An application team that keeps data anywhere else must name it there too. That data is either moved into one of the data stores below or backed up by an export job into the zone’s immutable backup account.

Where the backups go

Per environment there is a backup subscription sub-bkp-<env>. It sits in its own management group, and only the backup administrators group (not the platform team) has roles on it. Inside it, everything is per isolation zone, so the copies of different countries stay apart like the data itself (R2):

• Immutable and locked means a recovery point or blob version can be neither changed nor deleted before its retention ends, and the policy itself can no longer be weakened. Lowering a retention, disabling soft delete or removing protection on a vault also needs a second person’s approval through the Resource Guard.
• No platform or agent identity can delete. The vaults’ managed identities get the roles Azure Backup needs on each data source they protect, and nothing else. The backup subscription grants the platform’s IaC identity only Backup Operator (start a backup or restore, never delete), and grants the AI agents nothing (section 6.1).
• Retention (the picture’s example, to be agreed per data class): acc keeps prd’s layout so the procedures can be rehearsed there; dev has none, because it holds synthetic data only.
• Keys of the backups: the backup vaults and the backup storage accounts use Microsoft-managed keys with infrastructure encryption, not customer-managed keys, so a Cross Region Restore never depends on a key whose region is lost; the data they hold is CMK-encrypted at its source.

Per data store

• Volume snapshots are not enough. AKS volume snapshots and the operational tier stay in the primary region and may be deleted with it, so only the vault tier counts for region recovery. AKS backups are crash-consistent, so a database inside the stateful cluster also needs its operator’s own application-consistent backup.
• Customer-managed keys. A restored disk or database is encrypted again under the paired region’s new keys (section 4.2). If backed-up data is encrypted under a key that the restore needs (for example SQL TDE with a customer-managed key), that key is in the Key Vault backup above and is restored first.
• Azure Policy enforces the backup posture: every SQL server, PostgreSQL server and Cosmos DB account has geo-redundant backup storage and its LTR / vault policy; every application storage account has versioning and an object replication policy to its zone’s backup account; every backup vault is locked immutable with Cross Region Restore. A data store without protection is non-compliant and blocks the promotion of its application to prd.
• Monitoring: failed backup jobs, replication lag and the age of the newest recovery point per data store are alerts in Grafana Cloud (section 6.2).

Region recovery

A region loss is declared by humans: the platform lead together with the business owner, following the incident runbook. No agent and no automation may start a region recovery (safe actions). The recovery rebuilds the environment in the paired region, prd first:

1. Bootstrap: GitHub-hosted runners with OIDC federation to the environment subscriptions run the IaC with region=<pair>. They create the two hubs, the management VNet with its pipeline agents, and the corporate connectivity. For prd the ExpressRoute circuit (or VPN) in the paired region is provisioned in advance and only connected in the recovery, because ordering a circuit takes days that no prd RTO can absorb; dev and acc build theirs in the recovery. Once the agents are up, the pipelines run from the management VNet as before.
2. Environment module: the zone and platform Key Vaults, managed identities, Azure Policy assignments, backend spoke, ACR and private endpoints. Names that must be globally unique (Key Vaults, storage accounts, ACR, SQL and PostgreSQL servers) get the region suffix -<pair>, because the originals still exist in the lost region - the naming policy’s {location_code} does that for the platform’s resources once its templates carry it. Applications get their names from the overlays and cluster-settings, so this is a variable change, not a code change.
3. Secrets and keys: restore the Key Vault backups into the new zone vaults, and the needed keys into the new platform vault. The renewal job issues the certificates again.
4. Data, per zone, in parallel: geo-restore (or restore from the immutable copy) the SQL and PostgreSQL servers, copy the blobs from stbkp<env><zone> into new application storage accounts, and restore Cosmos DB.
5. Clusters: create the stateless cells (Flux bootstrap) and, if one exists, the stateful cluster. For the stateful cluster, restore the persistent volumes first (Cross Region Restore of the namespaces’ volumes, restoring only PersistentVolumeClaims and their volumes, with the new zone StorageClasses). The pipeline then deploys the StatefulSets, which bind the restored claims by name.
6. Images: prd’s ACR replica in the paired region already holds every deployed digest, so the cells start their applications unchanged. In dev and acc, CI builds the images of the commits that the fleet repository deploys, pushes them to the new ACR, and the release agent opens the pull request that moves the digests.
7. Traffic: Traffic Manager is global and survives; the IaC points its endpoints at the new cells’ external gateways. The internal Private DNS zones are created in the new internal hub, and ExternalDNS fills them.

• RTO is measured, not assumed. It is the time of the rebuild drill below, and it is dominated by the database restores and the image builds. The target is set with the business.
• Address plan: the paired region reuses the environment’s own prefixes (address plan), so Firewall rules, NSGs and on-premises routes keep their meaning. That works only because the primary region’s spokes are gone; a partial outage is handled by waiting for the region, not by running both at once.
• Failback to the primary region is a planned migration later, done the same way in the other direction. It is not part of the recovery.

Tests

Added to the acceptance tests (section 0.3) and repeated every six months in acc:

• Immutability: as Owner of the backup subscription, try to delete a recovery point, delete a blob version, shorten a retention and disable soft delete. Every attempt is denied or blocked by multi-user authorization.
• Region recovery drill: treat the primary region as lost (no access to its resources at all), rebuild acc in the paired region from IaC, Git and the immutable copies only, and run the reference application’s tests. Record the RPO of every data store and the RTO end to end.
• Stateful volumes: Cross Region Restore of one zone’s persistent volumes into a rebuilt stateful cluster; data consistent, and the StatefulSets bind the restored claims.