6. Operations

6.2 Observability with Grafana Cloud

Requirements: R14, R24, R31.

All telemetry goes to Grafana Cloud; the platform runs only the collector, Grafana Alloy.

Access rule: anyone who may sign in to a stack may see all of its data. Data inside a stack is not separated by isolation zone, country or application. This is a deliberate decision. The isolation zones (R1–R4) still separate the workloads, networks, nodes and secrets. Telemetry is not separated, because a cross-zone view is what makes diagnosing problems fast. The access rule decides who gets into a stack, and therefore what goes into it (below).

Stacks and access

• Sign-in with Entra ID (SAML single sign-on) through one enterprise application per stack. A user’s Grafana role comes from a group: Viewer for everyone in grp-grafana-<env>, Editor for the platform team in dev and acc, and Admin only through a PIM-activated group. Local Grafana users and basic auth are disabled. Grafana Cloud organisation administration (stacks, billing, access policies) also goes through PIM.

• Machine access uses access policies with the narrowest scopes:

  Token	Scopes	Stored in	Used by
  grafana-<env>-write	Fleet Management, metrics:write, logs:write, traces:write on its own stack	kv-<env>-platform	Alloy in every cluster of the environment
  grafana-<env>-opencost	metrics:read, limited by a label-based access policy to the cluster’s own series	kv-<env>-platform	OpenCost in every cluster of the environment
  grafana-<env>-read-<agent>	Viewer service account	The agent’s own secret store	One per AI agent and environment (section 6.1)
  grafana-<env>-gate	metrics:read	GitHub environment secret	Release and upgrade pipelines (SLO gates)
  grafana-<env>-iac	Stack administration	GitHub environment secret, main only	The Terraform pipeline that applies dashboards, alerts and SLOs

  Tokens expire after 90 days and are rotated by the IaC pipeline. The only read token in a cluster is OpenCost’s, limited by a label-based access policy to the cluster’s own series (cluster="<cluster>"), so a compromised cluster cannot read another cluster’s telemetry through it.

• Everything is code. Data sources, folders, dashboards, alert rules, contact points, notification policies, SLOs, synthetic checks and IRM schedules are managed by the grafana Terraform provider. The platform’s resources live in the IaC repository. An application team’s dashboards, alerts and SLOs live in apps/<app>/observability/ in the fleet repository, and the same pipeline applies them, dev → acc → prd. Changes made in the UI are allowed in dev only, and the next apply overwrites them.

Collection: Grafana Alloy, node-local

The k8s-monitoring Helm chart installs Alloy in the platform namespace infra-observability, configured from the stack with Grafana Cloud Fleet Management (remote configuration): the chart’s values only start the collectors and point them at Fleet Management; their pipelines – what is collected, how it is labelled and where it goes – are managed there by the platform team. Flux installs it in the stateless clusters and the pipeline in the stateful cluster, from the platform repository (section 3.3). Every collector works only on its own node, so telemetry never crosses from one isolation zone to another inside the cluster. A zone’s data leaves the cluster from that zone’s own nodes, through the Firewall rule of that zone:

• OTLP endpoint for applications: Service alloy-otlp.infra-observability.svc, ports 4317 (gRPC) and 4318 (HTTP), with internalTrafficPolicy: Local, so a pod always reaches the Alloy on its own node. An Azure Policy mutation injects OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_RESOURCE_ATTRIBUTES (deployment.environment, k8s.cluster.name, platform.zone) into every application container that does not set them itself. The platform’s Cilium baseline allows every application pod to reach alloy-daemon on these two ports, and alloy-daemon to reach the declared metrics port of the pods on its node, the same way it allows DNS. An application’s own egress policy needs no entry for telemetry.
• Common labels on every series, log line and span: env, cluster, cell (sl-az<N> or sf), zone (from the namespace’s platform/zone label), namespace, app and team (from the namespace labels set at onboarding). Alerts are routed and costs are attributed by these labels, not by access.
• Platform rules for Alloy: Alloy and node-exporter run as Grafana’s Fleet Management presets set them – privileged, on the host network and PID namespace, with the node’s logs, cgroups and storage – so Alloy can instrument applications with eBPF (Beyla) and read the node. Being on the host network, they are outside Cilium’s pod policies; the namespace’s other pods (kube-state-metrics, Beyla’s cache, OpenCost) keep a platform-owned policy. infra-observability is a platform namespace with a platform-owned egress policy observability (the Grafana Cloud ingest and Fleet Management host names, prices.azure.com for OpenCost, the API server’s name and the kubelets; layer 4 to world, the Firewall deciding by name).
• Write token: an ExternalSecret grafana-cloud in flux-system reads the secret flux-system--grafana-cloud (the Fleet Management URL, user and token as one JSON) from the platform vault through flux-system’s own prefix share, served by the platform’s External Secrets Operator in external-secrets (section 2.5); it lives in flux-system because the HelmRelease’s valuesFrom reads Secrets of its own namespace only.
• Buffering: Alloy keeps a write-ahead log on the node (an emptyDir), so a short Grafana Cloud or Private Link outage loses no data. A longer outage drops the oldest data first, and it never blocks the applications.
• OpenCost, part of the same chart, prices every node from Azure’s retail prices (spot nodes at the spot price) and splits each cell’s cost by namespace and pod; kube-state-metrics exports the namespaces’ platform/team, platform/cost-center and platform/zone labels, so cost is reported per team and cost centre (section 2.6). OpenCost reads the cluster’s usage back from the stack’s Prometheus API with a token of its own that has only metrics:read.
• Fleet Management matches its pipelines to the collector’s cluster attribute, the cluster’s name, so a renamed or rebuilt cell keeps its name or gets its matcher updated with it; otherwise it gets only Grafana’s default pipelines.
• Not used: Container Insights, the AKS cost analysis add-on and Azure Monitor managed Prometheus, so nothing is collected twice. The company’s security tool is security, not observability, and keeps its own agents.

Network: Grafana Cloud through Private Link

Telemetry never goes over the Internet. This matters most for the stateful cluster, which has no Internet at all (R19).

• The stacks are hosted on Azure, and Grafana Cloud accepts metrics, logs, traces and OTLP through Azure Private Link. The ingest host names are shared by all stacks in the same Grafana Cloud region, so there is one private endpoint per hub (snet-hub-pe of the internal and of the external hub) for all environments of that exposure. The stacks stay apart by tenant ID and token.
• DNS: for each ingest host name, the hub has a Private DNS zone named after the full host name (for example prometheus-prod-<n>-prod-<region>.grafana.net), with an apex A record pointing to the private endpoint. Only the ingest and query endpoints resolve privately. The Grafana UI (<org>-<env>.grafana.net) still resolves publicly.
• Firewall: one network rule per environment and exposure on the exposure’s Firewall, and the same for the management VNet.
• The stacks live outside the platform’s region: they are hosted in an Azure region other than the primary one (preferably the paired region of section 6.3). The private endpoint reaches them across regions, so losing the primary region does not take observability, its history or the alerting with it.
• Users sign in only with Entra ID, under a Conditional Access policy (compliant device, MFA) on each stack’s enterprise application.

What Grafana Cloud does

SLO gates. Releases (section 3.4) and cluster upgrades (section 3.5) are gated on these SLOs, per cell. The pipeline queries the burn rate of the cell’s applications from the stack’s Prometheus API, with the grafana-<env>-gate token, through the private endpoint. A missing signal counts as a failed gate: no data means no promotion.

Watchdog. The absent-data alert pages the platform on-call, so a broken collector is never mistaken for a quiet system.

Azure Monitor data, read in place

Some data is produced by Azure, not by the clusters. It stays where the SOC and its security tool expect it, in the environment’s Log Analytics workspace (log-<env>-platform), through diagnostic settings that Azure Policy enforces:

• Activity Log; AKS resource logs (kube-audit-admin and guard, more categories as a setting of the cluster IaC) into the environment’s workspace log-<env>-platform (1 GB a day, no public access); Traffic Manager, Key Vault, private endpoint and PaaS diagnostic logs; Azure Policy compliance. The Firewall’s log goes to the hub’s own workspace: the egress agent reads it raw, every other consumer its summaries.
• Grafana reads them, and Azure Monitor platform metrics (Firewall, Traffic Manager endpoint health, PaaS), through the Azure Monitor data source. It uses an app registration per environment with Monitoring Reader and Log Analytics Reader on the environment subscription, a certificate credential, and rotation by IaC. The access rule applies here too: every stack user can query these logs from Grafana.
• The security tool’s cloud and runtime alerts go to the SOC tooling as before, and are also shown in Grafana.

Security detections and forensics

Collecting everything is not the same as seeing what matters. Next to the SLO alerts, the platform defines what must never happen in it. Each detection has an owner who answers it:

The detections are Grafana alert rules on the Log Analytics data source and Sentinel analytics rules in the SOC tooling, kept as code. A detection that keeps firing without action is fixed or removed, so that the remaining ones are trusted.

Forensics. Metrics, logs and flow logs show symptoms; during an incident the platform team sometimes needs the kernel’s view: which process opened a socket, what a container executed. A kernel-level tracing tool (Inspektor Gadget, eBPF-based and Kubernetes-aware) is deployed only during an incident, through break-glass, as a privileged DaemonSet in a platform namespace. It is not installed by default, because a privileged, cluster-wide tracer is itself a high-value target.

What must not be sent

Because every stack user sees all data, nothing may be sent that a stack user is not allowed to see:

• No secrets, tokens or credentials in logs, span attributes or metric labels. Alloy drops Authorization, Cookie and Set-Cookie headers and redacts known token patterns (JWTs, connection strings, Key Vault secret values) before sending. This is a safety net; the application is responsible for not logging them.
• Personal data only where the stack’s users may see it: no request or response bodies, and no personal identifiers in metric labels. Logs may hold the personal data that the operations staff of every country are allowed to process.
• Retention (example, to be agreed): metrics 13 months; logs 30 days in prd and 14 days in dev and acc; traces 14 days. Kubernetes audit logs and Firewall logs stay in Log Analytics with the SOC’s retention; the Firewall log’s summaries are kept longer.