5. Egress control
5.1 Egress control and the company firewall
Who decides what an application may reach outside the landing zone: the hub Firewall, in every cell. Every cell runs Azure CNI Pod Subnet with the Cilium dataplane, so every pod keeps its own VNet address up to the Firewall (section 2.1), which decides every flow that leaves the cluster from rules written per application. Cilium decides only inside the cluster. This page states the model, why Cilium’s per-application FQDN policies are not used, and how the platform works with a company-wide firewall that another team owns.
The hub Firewall decides, Cilium inside the cluster
The Firewall sees a zone’s application pods by their own addresses, in the zone’s pod subnet, and the platform’s own components (gateways, ESO, Flux, external-dns) by the system subnets. Its rules name subnets or IP groups, so it opens a destination to a zone, not to one application - but its log names the pod, so every flow can be attributed. Every pod keeps its own address up to the hub Firewall (the platform configures AKS’s ip-masq-agent to translate nothing), and no egress policy exists in the application repositories:
- Cilium decides only inside the cluster, at layers 3 and 4: the zone baseline, namespace isolation and the
platform namespaces’ policies (section 2.6); the baseline’s DNS rule is plain
port 53 to kube-dns, and a platform policy per zone (
<zone>-firewall-egress, Flux Kustomizationfirewall-egress-<zone>) lets the zone’s application namespaces out to the world. Azure Policy’sapp-egressrule (section 2.7) keeps an application’s own policies inside the cluster, so the Firewall is the one decision out of it. - The isolation zone is the egress boundary. The Firewall decides everything that leaves the cluster from rules per application (one rule file per application namespace: the owner, the reason, the review) whose source is the zone’s IP group (its node and pod subnets in every cell): an allowed destination is open to every application of the zone, the rule is removed with the application that asked for it, and the Firewall’s log names the pod. A destination is any name the application calls as a client - an API on the Internet, a PaaS endpoint, or the published name of an application in another zone, whose gateway the same rule reaches through the Firewall (section 2.1); zones are never connected directly.
- An AI egress agent proposes the rules (section 6.1): it reads the flows the Firewall denied (Log Analytics), finds the pod behind each source address, checks the destination against the application’s configuration, and opens one pull request per application with the evidence. Humans of the platform and security merge it; nothing is allowed automatically.
- Nothing in 100.64.0.0/10 leaves the landing zone: the Firewall Policies’ SNAT private ranges are the landing zone’s own (the RFC 1918 ranges and 100.64.0.0/10), so every allowed flow to the Internet or the corporate network is translated to the Firewall’s address.
- Premium features on every flow: IDPS, threat intelligence, TLS inspection and URL filtering where agreed, FQDN tags; one log for security, in Log Analytics; one enforcement point for every flow that leaves the landing zone, from any compute, that nothing inside a cluster can change.
Making the shared destinations visible. The Firewall opens a destination to the whole zone - but the per-application rule files say which application asked for it, why, and the evidence. Section 5.2 generates from them, and from the applications’ own configuration, a picture per application and a picture per zone (the union the Firewall opens, and who brought each name in) - the review a zone-wide rule needs. An application that must not share its destinations with the rest of its zone - a regulated integration, a third-party API key that should only ever leave from one place - gets its own zone (adding a country adds a zone the same way).
Why not Cilium’s FQDN policies
AKS offers per-application egress by name through Advanced Container Networking Services
(ACNS): CiliumNetworkPolicy objects with toFQDNs, resolved by a DNS proxy, enforced at the source node; the
same add-on carries Hubble flow logs and eBPF host routing. The design does not use it:
- Cost. ACNS is billed per node and hour, so it scales with exactly the zone nodes the autoscaler creates on load. The hub Firewall is paid once per exposure whether the cells use it or not and is already the enforcement point for everything that is not a pod; letting it decide for the pods too adds data processing per GB and no fixed cost.
- What it would add is small here. Per-application granularity, which the per-application rule files and the traffic flow pages give at review time, and which a zone of its own gives at runtime; Hubble’s per-flow history, where the Firewall’s log already names the pod; and eBPF host routing’s few percent of node CPU, which Cilium’s kube-proxy replacement and a pod subnet that masquerades nothing have mostly taken already.
- What it would cost beyond the fee. A DNS proxy in every pod’s path and a policy per application for every team to keep (a mistake shows up as a broken application); no IDPS, threat intelligence or TLS inspection in that layer; incompatibility with LocalDNS; and, for host routing, a rebuild per cell and the loss of confidential VMs, pod sandboxing, Windows nodes and the Static Egress Gateway while it is on.
The dataplane baseline of the acceptance tests (section 0.3) is checked against the thresholds at which eBPF host routing would pay off: nodes spending more than 10 % of CPU in softirq, several hundred thousand packets per second per node, or a gateway-to-pod p99 budget the baseline misses - thresholds a cell of fewer than about 100 zone nodes does not reach.
Working with a company-wide firewall
In many companies the network or security team owns the firewalls, and the platform team does not. The platform’s egress rules are then requests to that team, in a form that team can take without learning Kubernetes: a rule file per application namespace (the zone, a contact, and per destination the FQDN, protocol, port, reason and evidence) plus the source IP groups the platform maintains (one per zone and one for the exposure’s system subnets). Nothing in a rule refers to pods or namespaces, and the egress agent’s pull requests and every human change go through the same review. The rules reach the company’s firewall as a child Firewall Policy under the company’s parent, on the platform’s own Firewall per exposure.
The Firewall’s log and its summaries
The hub Firewall writes its log to the hub’s own Log Analytics workspace: one line per connection with the source address, the destination (name or address) and port, the protocol, the action and the rule that decided it, or the default action. Denials are always logged, allowed flows only while an environment learns its rules. Two kinds of consumer read it:
- The egress agent reads the raw log (section 6.1): fresh within minutes, one flow at a time, and it finds the pod behind a source address at the time of the flow.
- Everything else reads summaries: the denial detection of section 6.2, the evidence of an egress review, the rule reviews and the trends per zone. Azure Monitor’s summary rules aggregate the raw log on a schedule into tables in the same workspace, so no consumer scans the raw log for the same questions again and the summaries outlive its retention.
Every rule attributes a source address to its environment and zone from the prefixes the Firewall Policies’ IP
groups hold: a zone’s node and pod slots in every cell, a cell’s system and shared slots as system and shared, the
management, VPN and hub networks by name, anything else other. The IaC generates the mapping with the IP groups, so
a new zone or cell is attributed from its first flow. The summaries keep the source address; they never name the pod.
| Rule | Bin | Table | Grouped by | Measures | Read by |
|---|---|---|---|---|---|
firewall-denied-flows |
1 hour | FirewallDeniedFlows_CL |
Environment, Zone, SourceIp, Category, Protocol, Destination (the FQDN, else the address), DestinationPort, Policy, RuleCollectionGroup, RuleCollection, Rule, ActionReason |
Count, FirstSeen, LastSeen |
The denial detection per zone; an egress review’s evidence (which source asked for what, and when); the zone isolation’s evidence |
firewall-rule-hits |
1 hour | FirewallRuleHits_CL |
Environment, Zone, Action, Category, Policy, RuleCollectionGroup, RuleCollection, Rule, ActionReason |
Count, Sources, Destinations (distinct) |
Rule hygiene: which rules each zone uses and how much; an application rule no zone has hit for 30 days is proposed for removal; which deny rules and default actions fire |
firewall-zone-traffic |
1 day | FirewallZoneTraffic_CL |
Environment, Zone, Action, Category, Protocol, DestinationPort |
Count, Sources, Destinations |
The trend per zone and day: capacity, the Firewall’s data processing cost, the weekly baseline of the denial detection, the SOC’s dashboards |
Retention (example, to be agreed, like section 6.2): the raw log with the SOC’s retention, the hourly tables 90 days, the daily table 13 months like the metrics.
The rules are monitored like a collector. The workspace’s diagnostic setting sends the rules’ run log to the workspace itself, one entry per bin: started, succeeded or failed. Azure Monitor retries a failed bin ten times within eight hours and then skips it; after eight consecutive failed bins it suspends the rule. A failed bin is a platform alert, so a gap in a summary is never read as a quiet zone.
What shapes the queries. A rule’s query has no time filter (the bin is its range, and a bin delay covers the log’s ingestion latency) and reads only its own workspace (no cross-workspace query), so every hub workspace, in every environment and exposure, has the same three rules. The results must be a tiny fraction of the source; they are billed as Analytics ingestion, and on the Analytics plan the query itself is free.