2. Inside a cluster

2.1 Network layout

Requirements: R2, R3, R4, R5, R6, R22, R29.

Every cell has its own VNet; the picture shows its slots, subnets and routes for an internal cell. A zone’s index, and so its slot, is the same in every cell of the exposure, so a new zone takes a free slot in every cell and nothing existing is re-addressed. The pods’ address space is a /16 of 100.64.0.0/10: about 4,000 pods per zone, handed to the nodes in batches of sixteen (an example is in the address plan). Private endpoints are not in the cells at all: they are reached through the hub firewall.

Zone isolation rests on every zone having its own nodes in its own subnets: a packet from a zone that leaves the cluster always carries an address of that zone’s slots. Every subnet has its own NSG (nsg-<cell>-<subnet>) and every node and pod subnet a route table (rt-<cell>, rt-<cell>-<zone>) with BGP propagation off, of the same shape for the system pool and for every zone - five routes, the Firewall being the exposure’s hub Firewall. The default route alone would not do: a packet to another slot of the cell matches the VNet’s own system route, which is more specific than 0.0.0.0/0, and would be delivered inside the VNet without any firewall seeing it. A user-defined route for the whole address space takes precedence over the system route for the same prefix, so what one slot sends to another leaves for the Firewall, which decides by rule and logs it: nothing is allowed between zones (R4, section 2.4), and between a zone and the system pool only the Kubernetes flows ① to ④ of section 2.4 - the kubelet, node metrics, Cilium’s health and the gateways from the system pool to a zone, cluster DNS from a zone to the system pool - as network rules of the exposure’s Firewall Policy. The own-slot exceptions are needed because Azure applies a route to the traffic within a subnet as well: the nodes and pods of one pool talk to each other directly. Two details keep the paths symmetric, which a stateful firewall needs. The API server’s private endpoint stays outside the Firewall’s view: a private endpoint ignores its subnet’s route table, and every subnet of the VNet carries a /32 system route to it that wins over the user-defined routes, so a zone’s nodes reach the API server directly and it answers them directly (making the endpoint honour the system subnet’s table was tried and withdrawn: its answers to the system pool’s own nodes and pods then left for the Firewall while their requests had stayed inside the subnet, and the system node’s kubelet lost the API server). And a zone’s own load balancer is reached through the Firewall because a request delivered inside the VNet would be handed to the gateway by the load balancer while the gateway’s answer, from the system pool, left for the Firewall. A new zone changes no table. So inside a cell three layers separate the zones, and the zones from the platform, in the order a packet meets them: the Firewall (L3-L7, logged), the destination subnet’s NSG and Cilium - and the Firewall is also the layer for everything that leaves the cell.

An application that must call an application of another zone - or its own zone’s published name - is a client of that zone’s entry point like any other, never a neighbour: its rule file names the published name (<app>.<zone>.<env>.example.com, resolved by the hub’s Private DNS to the zone’s gateway load balancer), exactly as it names an API on the Internet (section 5.1). The Firewall’s application rule proxies the flow from its own address, so the gateway - whose pods run on the system pool - answers through the hub by the system subnets’ default route, the path is symmetric, and the Firewall’s log names the calling pod. A network rule between two zones’ slots is never written: it would bypass the entry point. Applications of one zone call each other through their Kubernetes Services, inside the cluster, where Cilium decides (section 2.6).

What the NSGs allow on a node subnet, and from where:

• the kubelet port (TCP 10250) from the cell’s system subnet (the AKS tunnel); Cilium health (TCP 4240), ICMP and DNS (53) between the cell’s node subnets; the cell’s pod subnets between the node subnets; load balancer probes from AzureLoadBalancer; ICMP from the hub firewall’s address; traffic within the subnet. A pod subnet gets the rules of its pool’s node subnet;
• TCP 443 from the Internet on the system subnet (every zone’s gateway pods run there) and on an external zone’s node subnet; TCP 443 from VirtualNetwork on an internal zone’s node and load balancer subnets and on the system subnet - an NSG rule takes one service tag, and the corporate ranges arrive through the hub as VirtualNetwork.

Why subnets and not separate VNets? AKS requires all node pool subnets and the API server subnet of one cluster to be in the same VNet. Within one cluster, “network per country” is therefore implemented as a subnet slot per zone + NSG + Cilium, with the hub firewall between the slots and for everything that leaves the cell, which the security team accepts for countries. Internal and external are not separated this way: they are separate clusters in separate subscriptions behind separate hubs, so the picture shows an internal cell; an external cell’s VNet (vnet-prd-ext-stateless-az1) is the same with ext-fi and ext-se in its slots. Should a hard VNet boundary per country ever be required, the same step is taken once more – one cluster per zone – and the address plan stays the same.