6. Operations

6.1 AI-driven day 2 operations

Requirements: R30, R33.

This design was written by AI, and the platform it describes is built to be operated mostly by AI too: upgrades, capacity, cost, pod right-sizing, compliance, drift, incident triage and application onboarding are run by agents. Agents may change every environment, prd included, but only the way a human platform engineer would: with a pull request to the IaC or fleet repository. Humans no longer make most of the changes; they own the guardrails that decide which changes can be merged and by whom.

Rules

1. Git is the only write path: the existing delivery (section 3.3) applies every agent change.
2. Agents observe with read-only identities, act with pull requests, each with the least read access its task needs and no access to application data.
3. Guardrails are deterministic and owned by humans. What may be merged is decided by checks, limits and approval rules in a separate guardrails repository that agents can read but not change, not by an agent’s judgement – an agent cannot label its own change as low risk.
4. Risk decides the approvals, not the environment alone. A small change inside the limits merges automatically, also in prd; anything that touches identity, network, policy, data or the guardrails needs one or two humans.
5. The design documents are the specification. Agents read this repository as their context and cite the section a change follows; a change the design does not cover is a design change, made here first, by pull request.
6. The runtime guardrails stay the last line. Azure Policy, the admission policies of section 2.7, cell-by-cell rollouts with SLO gates and automatic rollback stop a wrong change that got merged, whoever wrote it.

Agents

One agent per responsibility, each with its own identity, tool allow-list and instructions, so that a confused or manipulated agent can only do what its role allows. Agent instructions (prompts, tools, allowed repositories and paths) are kept in the guardrails repository, versioned like code.

The reviewer agent never reviews its own work: it runs with a separate context, its own instructions and, where possible, a different model than the author, and it only sees the pull request, the plan output and the design documents. Human pull requests get the same review.

Every agent pull request has the same structure, enforced by the merge gate: intent (the issue or alert it answers), evidence (queries and links to metrics), the design section it follows, the plan (IaC plan / what-if and rendered manifest diff, attached by CI, not written by the agent), the rollback and the risk tier computed by the gate. Commits carry trailers Agent: <name> and Agent-Run: <id>, so every line in Git leads back to the agent run and its transcript.

Repositories and the merge gate

The examples use GitHub; Azure DevOps offers the same with branch policies, required reviewers and build validation.

• main is protected by an organisation ruleset that no repository admin or app can bypass: pull request required, linear history, signed commits, stale approvals dismissed on push, and the most recent push approved by someone other than its author.
• The merge gate is a required workflow defined in the guardrails repository, so an agent cannot change the check by changing the repository it is checked in. It:
  1. waits for CI to attach the plan – terraform plan / Bicep what-if as JSON for the IaC repository, rendered manifests per cluster and environment for the fleet repository;
  2. evaluates the plan, not the diff, against the policies and limits.yaml (below);
  3. computes the risk tier from what the plan does (resource types, actions, environments), never from a label set by the author;
  4. checks the approvals the tier requires: the reviewer agent, and humans from the CODEOWNERS teams. Agents’ apps are never members of those teams, so an agent approval can never count as a human one;
  5. enables auto-merge for tier low; for the others it waits for the approvals.
• Change windows and freezes are part of limits.yaml; outside a window a prd pull request waits, unless it is marked as an incident fix by a human.
• Rate limits per agent: open pull requests, merges per day and environment, and one prd cell per pull request – the gate rejects a pull request that changes more than one cell’s tag or version at a time, so the cell-by-cell rule of section 3.4 also holds for agents.

Risk tiers

limits.yaml holds the numbers the gate checks, for example:

environments:
  prd:
    changeWindows: ["Mon-Thu 08:00-15:00 Europe/Helsinki"]
    freezes: ["2026-12-18/2027-01-06"]
    maxMergesPerDay: { upgrade-agent: 2, capacity-agent: 3, platform-agent: 5 }
    nodePools:
      maxNodesPerPool: { soft: 30, hard: 60 }      # over soft = medium, over hard = rejected
      maxCpuPerZone: { soft: 960, hard: 1920 }
      maxSpotShare: 0.5
    allowedVmSizes: [Standard_D4ads_v6, Standard_D8ads_v6, Standard_D4pds_v6, Standard_D8pds_v6]
rightSizing:                                       # VPA objects and resource-only workload changes
  updateModes: ["Off", Initial, InPlaceOrRecreate] # stateful cluster: "Off" only
  controlledValues: [RequestsOnly]
  maxAllowed: { cpu: { soft: 4, hard: 16 }, memory: { soft: 16Gi, hard: 64Gi } }   # per container
  maxStepFactor: 2                                 # a request or limit moves at most ×2 per pull request
kubernetes:
  allowedMinors: { stateless: ["1.36"], stateful: ["1.36"] }   # 1.36 is the floor; newer minors added by the upgrade agent's PR, approved per tier
forbidden:
  - action: delete
    types: [Microsoft.KeyVault/vaults, Microsoft.ContainerRegistry/registries, Microsoft.Storage/storageAccounts, Microsoft.Sql/servers]
  - action: roleAssignment
    roles: [Owner, User Access Administrator, Role Based Access Control Administrator]

Policies in the gate cover what Azure Policy cannot see before deployment: deletions and replacements in the plan (a “replace” of a stateful resource is treated as a delete), blast radius (number of resources, number of cells), address spaces outside the IPAM allocation, Firewall rules to 0.0.0.0/0 or *, role assignments wider than a zone, and the rendered manifests against the same constraint templates as the clusters.

Identities and access rights

Agents run as container jobs in the management VNet (management subscription), the same network as the pipeline agents; their only egress is the model endpoint (private endpoint), the Git platform and an allow-list of documentation sites, through the internal hub’s Firewall (the management VNet is peered to both hubs). Each agent has, per environment, its own user-assigned managed identity with workload identity federation (no client secrets) and its own GitHub App with short-lived installation tokens. The identities and their role assignments are IaC in the guardrails repository.

Agents reach Azure and the clusters through tools, not shells: read-only wrappers around az and kubectl with a read-verb allow-list, or a tool server such as the AKS-MCP server run with its read-only access level and the agent’s own identity, next to tools for Grafana Cloud and Git. The tool server is a convenience and an audit point, not the security boundary: the read-only roles below are, so a tool server configured with a write level could still change nothing.

Azure (per environment subscription, read only)

No agent has a data-plane role on the PaaS services or the ACR: they do not need application data to run the platform, and a prompt injection through a log line or an issue then cannot exfiltrate it. Agents do read application logs and traces in Grafana Cloud, and the rule of what must not be sent keeps secrets out of them; their outbound access stays limited to the model provider and GitHub. Only the guardrails pipeline may create role assignments for agent identities; any other role assignment to one of them triggers an Activity Log alert, and a daily comparison against the guardrails IaC reports and removes it.

Git platform

Deployment identities (unchanged, not agents): the existing IaC pipeline, Flux and stateful pipeline identities are the only ones with write access. Their federated credentials accept only tokens for the protected main branch and the GitHub environment of the target (repo:<org>/<repo>:environment:prd), so a pull request branch – an agent’s or a human’s – can never deploy.

Humans: platform and security engineers approve pull requests and own the guardrails; their Azure access stays read-only plus break-glass via PIM (section 3.2). A break-glass action is followed by a pull request that brings Git back in line, opened by the operations agent from the drift it detects.

Safe actions without a pull request

Some incidents cannot wait for a pull request, but all of them are in the safe direction: stopping, or moving traffic away from a failing cell, not starting. They are parameterised pipelines in the runbooks repository, run with the existing deployment identities; the operations agent may only dispatch them:

Anything else, including undoing these actions, is a pull request. Every dispatch pages the human on-call, who can always reverse it.

Guardrails during deployment and at runtime

The pull request is the first guardrail, not the only one:

• Deployment: the cell-by-cell releases, SLO gates, soak times and automatic rollback of section 3.4 and section 3.5 apply to every merged change; the stateful pipeline keeps its kubectl diff and approval stage.
• Feedback: the operations agent verifies every merged change against the evidence in its pull request (did the alert clear, did cost drop) and reports the result on the pull request, so that humans can judge the agents by outcome.

Audit and kill switch

• Agent runs (instructions version, inputs, tool calls, outputs) are stored in an immutable storage account in the management subscription for a year; Git, the GitHub audit log and the Azure Activity Log link to them through the run ID.
• Kill switch: suspending an agent’s GitHub App installation stops its pull requests; agents.enabled: false in the guardrails repository makes the merge gate reject every agent pull request; removing the federated credentials stops its Azure reads. The platform keeps running – it is just changed by humans again.
• Start small: agents begin with a human on every tier above low in dev and above medium in acc and prd (minimumTier in the guardrails), and tiers are relaxed per agent and environment based on the measured outcome (merged without rework, rollbacks, incidents caused).