Cloud networking

What you get

• An address space you choose (RFC 1918 normally), split into subnets.
• Route tables that decide where traffic leaves each subnet.
• Stateful filters on instances or subnets (security groups, NSGs, firewall rules).
• Gateways for the outside world: internet, NAT, VPN, private circuits, peering.

The provider runs the actual forwarding in its own software-defined fabric. There is no broadcast and no real layer 2: ARP is answered by the hypervisor, and multicast is not supported by default.

Scope differences

Why the GCP model matters

In GCP two VMs in different regions of the same VPC talk over Google's backbone with no peering and no gateway. In AWS or Azure the same design needs one VPC or VNet per region plus peering or a transit hub between them.

AWS

A subnet is tied to one AZ (us-east-1a, for example). A typical layout for two AZs:

Reserved in 10.10.10.0/24: .0 network, .1 VPC router, .2 Amazon DNS, .3 reserved for future use, .255 broadcast (broadcast is not supported, but the address is still held back). Subnet size is /16 to /28.

Azure

A subnet spans all zones in the region; you pick the zone per VM or per zonal resource. Azure reserves .0 network, .1 default gateway, .2 and .3 for Azure DNS mapping, and the last address. The smallest subnet is /29. Some services need a dedicated subnet with a fixed name, such as GatewaySubnet for VPN and ExpressRoute gateways and AzureFirewallSubnet for Azure Firewall.

GCP

Subnets are regional. GCP reserves the network address, the gateway (.1), the second-to-last address and the broadcast address in the primary range.

The three conditions for an instance to talk to the internet directly

1. Its subnet's route table has 0.0.0.0/0 (or ::/0) pointing to an igw- target.
2. It has a public IPv4 address or an Elastic IP (or a global IPv6 address).
3. Security group and network ACL allow the traffic.

Miss any one and it does not work. "Public subnet" is not a checkbox; it is a label people use for condition 1. The "auto-assign public IPv4" subnet setting only helps with condition 2.

Typical route tables

rtb-public                       rtb-private-a
10.10.0.0/16  local              10.10.0.0/16  local
0.0.0.0/0     igw-0c1d2e3f4a5b60789  0.0.0.0/0  nat-0f1e2d3c4b5a69788

Why private subnets

Databases and app servers should not be reachable from the internet at all. In a private subnet they can still start outbound connections (updates, APIs) through the NAT gateway, but nothing can open a connection to them from outside.

AWS

• Local route. 10.10.0.0/16 local is in every table. All subnets in the VPC can reach each other unless a security group or NACL stops it.
• Targets. igw-, nat-, pcx- (peering), tgw-, vgw-, eni- (an appliance), vpce- (gateway or GWLB endpoint), egress-only IGW for IPv6.
• Selection. Longest prefix match first. For identical prefixes, static routes beat propagated (BGP) routes.
• Propagation. A route table can learn routes from a virtual private gateway when route propagation is enabled for it.
• Gateway route tables. You can associate a route table with the internet gateway or virtual private gateway itself (edge association) to steer inbound traffic through a firewall appliance.

Azure

Every subnet gets system routes: the VNet range, peered VNets, 0.0.0.0/0 to Internet, and next hop None for RFC 1918 and 100.64.0.0/10 ranges that are not in the VNet. You attach a route table of user-defined routes (UDRs) to a subnet with next hop types such as VirtualAppliance, VirtualNetworkGateway, Internet or None. For the same prefix, a UDR beats a BGP route, which beats a system route.

GCP

Routes are VPC-wide objects. Subnet routes are created for you. A default route to the default internet gateway exists until you delete it. Custom static routes have a priority (lower wins on a tie) and can apply only to instances with a network tag. Cloud Router adds dynamic routes, either for its own region or for all regions depending on the VPC's dynamic routing mode.

Internet gateway (IGW)

• One per VPC, highly available by design, no bandwidth knob to size.
• Does static 1:1 NAT for instances with public IPv4 or Elastic IPs. The instance OS only ever sees its private address.
• For IPv6 there is no NAT. An egress-only internet gateway gives outbound-only IPv6, which is the IPv6 equivalent of a NAT gateway's protection.

NAT gateway

• Created in one subnet of one AZ. A public NAT gateway has an Elastic IP and must sit in a public subnet (route 0.0.0.0/0 to the IGW).
• Private subnets route 0.0.0.0/0 to it. It translates source address and port (PAT).
• Supports up to 55,000 simultaneous connections to each unique destination (IP, port, protocol).
• For AZ resilience you deploy one per AZ and point each private subnet at the NAT gateway in its own AZ. One shared NAT gateway is a single point of failure and adds cross-AZ data charges.
• A private NAT gateway has no Elastic IP and translates to its private address. It is used to reach other networks (for example through a transit gateway) when address ranges overlap.
• You cannot attach a security group to it. The subnet's network ACL still applies.

Equivalents

• Azure: NAT Gateway, a managed resource associated with subnets, using public IPs or prefixes. Each public IP gives 64,512 SNAT ports.
• GCP: Cloud NAT, configured on a Cloud Router. It is software-defined: there is no NAT box in the data path, the translation happens in the host networking stack.

How I use them

• Security groups do the real work. Reference groups instead of addresses: the app SG allows port 8080 from the web SG, the DB SG allows 5432 from the app SG. Instances can be replaced or scaled and the rules still hold.
• NACLs are a coarse guard rail. Block a known bad range, or enforce "this subnet never talks to that subnet" as a second layer. Keep them simple, because every rule needs a matching return rule.

The return traffic trap

A NACL that allows inbound 443 must also allow outbound to the client's ephemeral ports, and a subnet whose instances make outbound calls needs inbound rules for the replies. Linux uses 32768 to 60999, Windows uses 49152 to 65535, and load balancers can use anything from 1024. Allowing 1024 to 65535 is the usual answer.

How it works

The instance's ENI has 10.10.0.25. AWS maps 203.0.113.25 to it at the edge. Applications must not bind to the public address, and anything that embeds its own IP in the payload (SIP, FTP active mode) needs to be told the public address.

Auto-assigned public IP vs Elastic IP

Cost

Since February 2024 AWS charges an hourly fee for every public IPv4 address, in use or idle. That is one more reason to keep servers private behind load balancers and NAT, and to look at IPv6.

Azure and GCP

Same idea: Azure public IP resources attach to a NIC or a load balancer, and GCP external IPs (ephemeral or static) are attached to the VM's access config. The guest OS sees only the internal address in both.

How I choose

• Proof of concept or small office: VPN.
• Production data center with steady traffic or latency-sensitive apps: two circuits in two locations, with a VPN as a cheap extra backup.
• Many branches already on SD-WAN: extend the fabric with virtual appliances in a transit VPC or VNet, or the provider's native SD-WAN integration (AWS Transit Gateway Connect, Azure Virtual WAN).
• Compliance requires encryption on the circuit: MACsec where supported, or IPsec over the circuit.

BGP is the routing protocol on every one of these options; static routing exists for VPNs but loses automatic failover.

Rules in AWS

• One-to-one, between two VPCs, same or different account, same or different region.
• CIDRs must not overlap.
• No edge-to-edge routing: traffic must start or end in one of the two peered VPCs.
• Both sides must add routes to the other VPC's CIDR with the pcx- target, and security groups must allow the traffic. Peering by itself routes nothing.
• In the same region, security groups can reference a group in the peer VPC.
• No hourly charge. Data across AZs or regions is charged as usual.

The scaling math

Each one also needs route entries on both sides. That is where a transit gateway wins.

Other clouds

• Azure VNet peering is non-transitive too. The exceptions are gateway transit (spokes use the hub's VPN or ExpressRoute gateway) and routing through a network virtual appliance with UDRs.
• GCP VPC Network Peering is non-transitive. It exchanges subnet routes, and custom routes only if you enable import and export.

Pieces

• Customer gateway (CGW): a record of your public IP and BGP ASN.
• Virtual private gateway (VGW): attached to one VPC. Amazon side ASN defaults to 64512. Or terminate on a transit gateway to serve many VPCs.
• VPN connection: two tunnels, each to a different AWS endpoint. IKEv1 or IKEv2, PSK or certificates.

Inside addressing

Each tunnel gets a /30 such as 169.254.10.0/30. AWS uses the first host (169.254.10.1), your router uses the second (169.254.10.2). BGP runs between those addresses.

Behaviour worth knowing

• By default your device must start IKE; AWS waits. Tunnel options can make AWS initiate with IKEv2.
• A standard tunnel carries around 1.25 Gbps. A VGW will not load-share across tunnels. A transit gateway can ECMP across tunnels and across several VPN connections with BGP.
• Dead peer detection timeout defaults to 30 seconds.
• AWS-provided sample configs clamp TCP MSS on the tunnel interface to avoid fragmentation.

Equivalents

• Azure VPN Gateway (route-based) lives in GatewaySubnet. Active-standby by default with one public IP; active-active uses two. Default Azure ASN for BGP is 65515.
• GCP HA VPN has two interfaces with two external IPs. With tunnels from both interfaces and BGP on Cloud Router it carries a 99.99% SLA; Classic VPN is 99.9%.

Notes that come up

• ALB node addresses change. Partners who need a fixed IP get an NLB, or Global Accelerator in front.
• An NLB can be the front of a PrivateLink endpoint service; an ALB cannot by itself (you can put an ALB behind an NLB as a target).
• GWLB is used with GWLB endpoints placed as route targets in VPC route tables, so traffic is steered to the appliance fleet and back without changing addresses.

Other clouds

Policies

Health checks

• Checkers in several AWS regions probe the endpoint, every 30 seconds by default (10 seconds as an option), and mark it unhealthy after 3 failures by default.
• They run from the internet, so they cannot probe a private IP. For private endpoints, base the health check on a CloudWatch alarm.
• Failover is only as fast as the record TTL lets clients notice. Keep TTLs short (60 seconds or less) on records you expect to fail over.

Alias vs CNAME

• A CNAME cannot sit at the apex (example.com) because the apex already has SOA and NS records and a CNAME cannot coexist with other data.
• An alias is a Route 53 feature that returns the target's A or AAAA records directly, follows the target's address changes, and is free for queries to AWS targets such as ELB, CloudFront and S3 websites.

Private hosted zones

Answer only inside the VPCs they are associated with. Useful for internal names and split-horizon DNS.

Inside a node

Each pod has its own network namespace. The CNI plugin creates a veth pair: one end is eth0 in the pod, the other sits on the host, attached to a bridge or routed by the host. Containers in one pod share the namespace and talk over localhost.

Between nodes

Overlay works anywhere but adds encapsulation overhead and lowers the pod MTU. VPC-native avoids the overlay and makes pods first-class in security groups and flow logs, but uses VPC addresses fast.

What CNI is

The Container Network Interface is a spec and a set of plugins. The container runtime calls the plugin with ADD and DEL when a pod sandbox is created or removed. The plugin handles interface setup and IPAM. Network policy enforcement is a separate capability that some CNIs provide (Calico, Cilium) and others do not (Flannel on its own).

Three ranges to plan

Node addresses, the pod CIDR and the service CIDR. None of them may overlap with each other or with networks pods must reach.

Types

Data path

1. Pods matching the selector and passing readiness probes are listed in EndpointSlices.
2. kube-proxy watches them and writes rules: iptables DNAT chains, IPVS virtual servers, or nftables in newer releases. Cilium can replace kube-proxy with eBPF.
3. A packet to 10.96.40.12:80 is translated to a chosen pod, for example 10.244.3.17:8080.
4. CoreDNS answers web.shop.svc.cluster.local with the ClusterIP.

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: shop
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080

What an AZ is

One or more data centers in a region with separate power, cooling and networking, joined to the other AZs by low-latency links. AZ failures are rare but real; region failures are rarer.

Checklist

• Subnets: public, private and data subnets in each AZ.
• Compute: auto scaling groups across AZs, with enough headroom that two AZs carry the load of three.
• Load balancers: enabled in every AZ that has targets.
• NAT: one NAT gateway per AZ. A single NAT gateway makes every AZ depend on one.
• Data: Multi-AZ databases, replicated caches.
• Hybrid: VPN tunnels and transit gateway attachments already span AZs; make sure the subnets chosen for attachments cover every AZ you use.

AZ names are per account

us-east-1a in one account is not necessarily us-east-1a in another. AWS publishes AZ IDs such as use1-az1 that are the same everywhere; use them when lining up shared subnets or partner connections.

Azure and GCP

Azure regions with zones expose zones 1, 2 and 3; resources are zonal or zone-redundant. GCP zones such as us-central1-a and regional managed instance groups play the same role.

Azure NSG

• Custom rules use priority 100 to 4096. Default rules sit at 65000 and up: allow VNet to VNet, allow Azure Load Balancer probes inbound, allow internet outbound, then deny all.
• Service tags such as VirtualNetwork, Internet, AzureLoadBalancer and Storage stand in for address lists Microsoft maintains.
• Application security groups (ASGs) group NICs by role, so a rule says "web ASG to app ASG on 8080", close to AWS security group referencing.
• With NSGs on both subnet and NIC: inbound is checked at the subnet first, then the NIC; outbound at the NIC first, then the subnet. Both must allow.
• Watch out: VirtualNetwork includes peered VNets and on-prem ranges learned through a gateway, so the default allow is wider than people think.

GCP VPC firewall rules

• Defined on the VPC, applied in every region. Direction is ingress or egress.
• Priority 0 to 65535, default 1000. Implied rules at the lowest priority deny all ingress and allow all egress.
• Targets: all instances, instances with a network tag, or instances running as a service account. Service accounts are stronger because changing them needs IAM permission, while tags are editable by anyone who can edit the VM.
• Hierarchical and network firewall policies add organization-wide rules above the VPC rules.

AWS

Allow-only, per ENI, all rules evaluated, other security groups as sources. Network ACLs are the place for explicit deny.

Principles

• Unique everywhere. Anything that may ever be peered, routed over a transit hub or reached from on-prem must not overlap. Overlap is cheap to avoid on day one and very expensive to fix later.
• Hierarchical. For example 10.64.0.0/10 for AWS, split by region (10.64.0.0/13 us-east-1, 10.72.0.0/13 eu-west-1), then a /16 to /20 per VPC. Summaries per region keep the on-prem and transit route tables short.
• Size for growth. Kubernetes and serverless consume IPs quickly. A /16 VPC is not wasteful if the block is planned; a /24 VPC that has to be rebuilt is.
• Leave room in each VPC. Do not carve the whole VPC into subnets on day one. You can add subnets but you cannot resize existing ones.
• Use a tool. AWS VPC IPAM or NetBox as the source of truth, so teams request ranges instead of guessing.

Ranges to avoid

Kubernetes ranges count too

Pod and service CIDRs that are not VPC addresses still must not overlap anything pods need to reach. EKS, for example, picks 10.100.0.0/16 or 172.20.0.0/16 for services if you do not set one.

The meters that surprise people (us-east-1 list prices, they change, always check)

Design habits

• Gateway endpoints for S3 and DynamoDB. They are route table entries, free, and remove that traffic from the NAT gateway. A backup job pushing terabytes to S3 through NAT is a classic bill shock.
• NAT gateway per AZ. Besides resilience, it avoids paying inter-AZ charges on every outbound byte.
• AZ affinity. Keep app-to-cache and app-to-database traffic in the same AZ when the service supports it; replicate across AZs on purpose, not by accident.
• CDN in front of public content. CloudFront or another CDN offloads repeated downloads.
• Direct Connect for heavy hybrid traffic. Data transfer out over Direct Connect is billed at a lower rate than internet egress.
• Measure. VPC Flow Logs plus Cost Explorer by usage type (DataTransfer-Regional-Bytes, NatGateway-Bytes) shows who is moving data.

Other clouds

Azure and GCP follow the same shape: free ingress, paid internet and inter-region egress. GCP also charges for traffic between zones in a region.

VPC settings

• enableDnsSupport turns the Amazon resolver on (default true).
• enableDnsHostnames gives instances DNS hostnames (default false in custom VPCs). Private hosted zones need both set to true.
• Private hosted zones are associated with specific VPCs, so the same name can resolve differently inside and outside (split-horizon).

Hybrid with Route 53 Resolver

on-prem DNS --- conditional forward aws.corp.example --> inbound endpoint IPs (ENIs in the VPC)
VPC resolver --- rule: forward corp.example --------> outbound endpoint --> on-prem DNS 192.168.10.53

Each endpoint has ENIs in at least two AZs. Forwarding rules can be shared to other accounts with AWS RAM and associated with many VPCs.

Azure and GCP

• Azure: the platform resolver is 168.63.129.16. Private DNS zones link to VNets. Azure DNS Private Resolver gives inbound and outbound endpoints plus forwarding rulesets, the same pattern as Route 53 Resolver. Before it existed, people ran DNS forwarder VMs.
• GCP: Cloud DNS private zones, forwarding zones to on-prem, and an inbound server policy that gives on-prem a forwarding target inside the VPC.

Building blocks

• Attachments: VPC (one subnet per AZ, best as a small dedicated /28), site-to-site VPN, Direct Connect gateway (through a transit VIF), TGW peering (between regions, static routes only), and Connect (GRE plus BGP for SD-WAN appliances).
• Route tables: a TGW can have many. Association says which table an attachment uses when its traffic enters the TGW (exactly one). Propagation says which tables learn that attachment's routes (zero or more).
• Static routes can be added, including blackhole routes that drop traffic.
• VPC side: the VPC's own route tables still need routes such as 10.0.0.0/8 to tgw-....

Packet walk

VPC A instance -> VPC A route table (10.0.0.0/8 to tgw) -> TGW attachment A
  -> lookup in the route table associated with A -> attachment B -> VPC B local route -> instance

Things to know

• It is a regional resource. Inter-region needs TGW peering with static routes.
• It charges per attachment hour and per GB processed, so it is not free like peering.
• With BGP VPNs it can ECMP across many tunnels, which is how you get past the per-tunnel bandwidth.
• Amazon side ASN defaults to 64512 unless you set it at creation.

Equivalents

• Azure Virtual WAN: a Microsoft-managed hub with transitive routing between VNets, VPN, ExpressRoute and point-to-site. The do-it-yourself version is a hub VNet with Azure Firewall or an NVA, spokes peered to it, and UDRs pointing at the firewall.
• GCP Network Connectivity Center: a hub with VPC spokes and hybrid spokes (VPN, Interconnect, router appliances).

AWS side (traffic from the VPC to on-prem)

1. Most specific prefix wins. Advertising 10.20.0.0/16 over DX and 10.0.0.0/8 over VPN keeps traffic on DX until the /16 is withdrawn.
2. Same prefix: Direct Connect BGP, then VPN static, then VPN BGP.
3. Between VPN BGP routes: shortest AS path, then lowest MED.

To make tunnel 2 the backup, advertise the same prefixes on it with a prepended AS path:

route-map AWS-TUN2-OUT permit 10
 set as-path prepend 65010 65010
!
router bgp 65010
 address-family ipv4
  neighbor 169.254.20.1 route-map AWS-TUN2-OUT out

Your side (traffic from on-prem to AWS)

Your routers choose. Set a higher local preference on routes learned over DX, and keep the VPN routes at the default 100 so they are used only when DX disappears.

Transit gateway is different

A TGW can ECMP across VPN tunnels when the paths are equal and ECMP is enabled. That is a feature for bandwidth, and a trap when you wanted active-standby: then make the paths unequal on purpose.

Connection types

• Dedicated: a cross-connect to an AWS router port in a colocation facility. You order it, get a Letter of Authorization (LOA-CFA), and have the cross-connect run.
• Hosted: a partner already connected to AWS provisions a VLAN for you at a fixed bandwidth, from as little as 50 Mbps.

Virtual interfaces

A Direct Connect gateway is a global object: one private or transit VIF can reach VGWs or TGWs in many regions. It does not route VPC to VPC through itself.

Rules that cause outages

• BGP is mandatory on every VIF.
• A private VIF accepts at most 100 prefixes from you. Advertise more and the BGP session goes down. Summarize.
• Direct Connect is not encrypted. MACsec is available on some dedicated connections; otherwise run IPsec over it.

Redundancy

One connection is a single point of failure. AWS's guidance for critical workloads is connections at two separate Direct Connect locations, each on separate devices, with BFD for fast failover and a VPN as an optional last resort.

Circuit anatomy

• Ordered at a peering location with a bandwidth (provider circuits from 50 Mbps up to 10 Gbps; ExpressRoute Direct gives you 10 or 100 Gbps ports).
• Two links, two BGP sessions per peering. For private peering you supply two /30 subnets (one per link), a VLAN ID and your ASN.
• Private peering: to VNets. Needs an ExpressRoute virtual network gateway in GatewaySubnet. FastPath can send data around the gateway for high-throughput cases.
• Microsoft peering: to public Microsoft services, with public IPs and route filters.
• Global Reach: connects two of your circuits so two on-prem sites talk over Microsoft's backbone.
• Premium add-on: reach VNets in other geopolitical regions and a larger route limit.

Not encrypted

Same as Direct Connect. MACsec is available on ExpressRoute Direct; otherwise IPsec over private peering.

GCP Cloud Interconnect

• Dedicated Interconnect: your own 10 or 100 Gbps circuits into a Google colocation.
• Partner Interconnect: through a provider, smaller sizes.
• VLAN attachments connect to a Cloud Router, which runs BGP and installs routes in the VPC. Google publishes the topologies (two metros, four connections) needed for the 99.99% SLA.

Mapping across clouds

Gateway endpoint (S3, DynamoDB only)

• Adds a prefix list route (pl-...) to the route tables you choose.
• No hourly charge, no data processing charge.
• Cannot be used from on-prem, a peered VPC or through a transit gateway.

Interface endpoint (most AWS services)

• One ENI per chosen AZ, with a security group.
• Private DNS option makes the normal service name (for example sqs.us-east-1.amazonaws.com) resolve to the endpoint IPs inside the VPC.
• Reachable over VPN, Direct Connect, peering and transit gateway.

Your own service with PrivateLink

consumer VPC (10.0.0.0/16)            provider VPC (10.0.0.0/16, same range)
app -> interface endpoint 10.0.5.20 -> PrivateLink -> NLB -> service targets

The provider creates an endpoint service from an NLB and allows principals. The consumer creates an interface endpoint. Only the consumer can start connections. The provider sees the NLB's address as the source unless it enables proxy protocol v2.

Azure and GCP

• Azure Private Endpoint: a NIC with a private IP mapped to one specific resource (one storage account, one SQL server). Service endpoints are older: the service still has a public IP, but the subnet's identity is used to allow access, and on-prem cannot use them. Private Link service publishes your own service behind a Standard Load Balancer.
• GCP Private Service Connect: an internal IP endpoint for Google APIs or for services published by producers with a service attachment. Private Google Access lets VMs without external IPs reach Google APIs.

Peering settings, set on each side

Spoke-to-spoke routing

Spoke 1 and spoke 2 are peered only with the hub. To connect them:

• Route table on spoke 1 subnets: 10.2.0.0/16 next hop VirtualAppliance at the firewall IP (for example 10.0.1.4).
• Route table on spoke 2 subnets: 10.1.0.0/16 to the same firewall.
• Firewall policy allows the flow. Forwarded traffic allowed on the peerings.

Gateway route propagation

A route table has a "propagate gateway routes" setting. Turning it off on spoke subnets stops on-prem routes learned by the gateway from bypassing the firewall; a UDR then sends on-prem traffic to the firewall instead.

Health checks

• Probes come from the load balancer nodes' private IPs. The target security group must allow them, ideally by referencing the load balancer's security group.
• Point the check at a real health endpoint (/healthz) that tests dependencies the instance needs, not just "the web server answers".
• Success codes default to 200 for an ALB; you can widen it, for example 200-399.
• Fail open: if every target in a target group is unhealthy, the load balancer sends traffic to all of them anyway. A bad health check path does not cause an outage, but it hides the real health.

Deregistration delay

During scale-in or deploys, a target stops getting new connections but keeps existing ones for up to 300 seconds. Short-lived HTTP services often lower it to 30 seconds to speed up deploys.

Cross-zone

With cross-zone off, each AZ's load balancer node only uses targets in its own AZ. If AZ a has 2 targets and AZ b has 8, each AZ still gets half the traffic, so the 2 targets in AZ a run four times hotter.

By load balancer

Side effects of preserving the client IP

• The target's security group must allow the real client addresses, not just the load balancer.
• Replies must go back through the load balancer path. With a passthrough design, a server whose default route points elsewhere breaks the flow.
• Hairpin: with NLB client IP preservation, a target that connects to its own NLB and is balanced to itself sees a packet from its own IP to its own IP, and the connection fails.

Kubernetes

A LoadBalancer Service with externalTrafficPolicy: Cluster lets kube-proxy send traffic to a pod on another node, and it SNATs the packet, so the pod sees a node IP. externalTrafficPolicy: Local keeps traffic on nodes that run a pod and preserves the client IP, at the cost of possibly uneven spread.

Ingress needs a controller

The Ingress object does nothing on its own. The controller watches it and configures a proxy:

• Self-hosted proxies (NGINX-based, Traefik, HAProxy, Envoy-based) running as pods behind one LoadBalancer Service.
• Cloud-managed: the AWS Load Balancer Controller builds an ALB from an Ingress; GKE builds a Google Application Load Balancer.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  namespace: shop
spec:
  ingressClassName: alb
  tls:
    - hosts: [shop.example.com]
      secretName: shop-tls
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80

With the AWS controller, TLS is normally done with an ACM certificate annotation rather than a secret.

Gateway API

Split by role: the platform team owns GatewayClass and Gateway (listeners, certificates), app teams own HTTPRoute objects that attach to it. It supports header matching, traffic splitting and more protocols without controller-specific annotations.

Default deny plus an allow

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: shop
spec:
  podSelector: {}
  policyTypes: [Ingress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-from-web
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: web
      ports:
        - protocol: TCP
          port: 8080

The AND vs OR trap

# One element: namespace AND pod must match
- from:
    - namespaceSelector: {matchLabels: {team: web}}
      podSelector: {matchLabels: {app: web}}
# Two elements: namespace OR pod matches
- from:
    - namespaceSelector: {matchLabels: {team: web}}
    - podSelector: {matchLabels: {app: web}}

One dash changes the meaning and can open a namespace wide.

Enforcement

Calico, Cilium, the AWS VPC CNI with its network policy agent, and Azure's options enforce policies. Flannel alone does not, and the API server accepts the objects anyway, so nothing warns you.

Order of checks

1. Routing, both directions. The return route is the one people forget.
2. Security groups. Inbound on the destination, outbound on the source.
3. Network ACLs. Both subnets, both directions, including ephemeral return ports.
4. Host. ss -ltnp to see the listener, iptables or Windows Firewall, the app's bind address.

Flow logs

2 123456789012 eni-0a1b2c3d4e5f60718 10.10.10.25 10.20.10.40 49152 5432 6 10 840 1731230000 1731230060 REJECT OK

Version, account, ENI, source, destination, ports, protocol (6 is TCP), packets, bytes, window start and end, action, log status. A REJECT points at a security group or NACL.

Not captured: traffic to the Amazon DNS resolver, the instance metadata service, the Amazon Time Sync service, DHCP, and Windows license activation. Records are aggregated over an interval (10 minutes by default, 1 minute as an option), so they are not real time.

Tools by cloud

Option 1: re-IP

Painful, but it removes the problem forever. Often the right answer for a small VPC or a new acquisition.

Option 2: PrivateLink (service-level connectivity)

The provider puts the service behind an NLB and publishes an endpoint service. The consumer creates an interface endpoint, which takes an IP from the consumer's own subnet. Neither side ever routes to the other's range, so overlap does not matter. Limits: one direction only (consumer starts connections), and it exposes services, not whole networks.

Option 3: NAT with a private NAT gateway (AWS pattern)

1. Add a small, unique secondary CIDR to each VPC (the routable range), for example 100.64.10.0/24.
2. Keep workloads in the overlapping, non-routable range.
3. Put a private NAT gateway in a routable subnet. Outbound traffic to the other network is SNATed to it and sent via the transit gateway.
4. For inbound, place an internal ALB or NLB in the routable subnets with targets in the overlapping range.

Option 4: NAT on the edge

For on-prem overlap, translate on the VPN or firewall: Azure VPN Gateway supports NAT rules, and a firewall or router appliance in a transit VPC can do static or dynamic NAT. Every translated service needs DNS that returns the translated address.

What I tell the interviewer

Overlap is a planning failure; NAT is a workaround that adds state, breaks protocols that embed addresses, and complicates troubleshooting. I would choose PrivateLink for a few services and re-IP for broad connectivity.

Route tables

Prod and dev never learn each other's routes, so there is no path. That is routing-based isolation; security groups still apply inside.

Blackhole routes

If a spoke table has 0.0.0.0/0 to a central egress VPC, I add a blackhole route for the internal supernet (for example 10.0.0.0/8). Traffic to an internal range nobody has propagated is dropped at the TGW instead of wandering toward the NAT gateways. Longer propagated routes still win by longest prefix match.

Inspection VPC

• Spoke tables have 0.0.0.0/0 (or the internal supernet) to the inspection VPC attachment.
• The inspection VPC sends traffic through AWS Network Firewall endpoints or a Gateway Load Balancer with third-party firewalls, then back to the TGW.
• The inspection attachment is associated with a table that knows all spoke routes.

Why appliance mode

By default a TGW keeps traffic in the AZ where it entered. If the source is in AZ a and the destination in AZ b, the request enters the inspection VPC in AZ a and the reply in AZ b. Two different firewalls see the two halves; the stateful one that sees the reply has no session and drops it. With appliance mode enabled on the inspection VPC attachment, the TGW picks one AZ for the flow and uses it for both directions.

Azure example

mystore.blob.core.windows.net
  CNAME mystore.privatelink.blob.core.windows.net
    inside a VNet linked to privatelink.blob.core.windows.net  -> A 10.50.2.4
    anywhere else (public DNS)                                 -> public IP

1. Create the private DNS zone privatelink.blob.core.windows.net, link it to the hub VNet, and let the private endpoint register its record there.
2. Deploy Azure DNS Private Resolver with an inbound endpoint in the hub (or a DNS forwarder VM that uses 168.63.129.16).
3. On-prem DNS: conditional forwarder for blob.core.windows.net to the inbound endpoint IP. Microsoft recommends forwarding the public zone name, not the privatelink name, so the CNAME chain is followed inside Azure.

168.63.129.16 itself is only reachable from inside Azure, which is why on-prem cannot forward to it directly.

AWS equivalent

An interface endpoint with private DNS creates a hidden private hosted zone for the service name in that VPC. On-prem forwards the service domain to a Route 53 Resolver inbound endpoint. For endpoints shared from a central VPC, turn private DNS off on the endpoint, create your own private hosted zone for the service name with an alias to the endpoint, and associate that zone with every spoke VPC.

Design rule

One private DNS zone per privatelink name for the whole organisation, linked to the VNets or VPCs that need it. Duplicate zones in different hubs give different answers depending on where you ask.

The pieces

• Network account: TGW per region (shared via RAM), TGW peering between regions, Direct Connect gateway with transit VIFs, VPN backup.
• IPAM: pools per region and environment so every VPC gets a unique, summarizable range.
• Inspection VPC: AWS Network Firewall or GWLB with third-party firewalls, for east-west and egress.
• Egress VPC: NAT gateways per AZ. Spokes have no IGW or NAT of their own.
• Shared services VPC: interface endpoints shared through private hosted zones, Route 53 Resolver endpoints and rules shared with RAM, directory services.
• Guard rails: service control policies that stop spoke accounts from creating IGWs or peering on their own.

Alternative: VPC sharing

AWS RAM can share subnets of one VPC with many accounts. Accounts launch resources in shared subnets, the network team owns routing. Fewer VPCs and attachments, but a bigger blast radius per VPC.

Trade-offs to state out loud

• Central egress saves NAT gateway hours across many VPCs but adds TGW processing charges per GB.
• Central inspection adds a hop and cost; inspect what policy requires, not everything by default.
• One TGW per region is a shared dependency; change control on its route tables matters.

Other clouds

• GCP Shared VPC: a host project owns the VPC and subnets; service projects use them. Combine with NCC or VPN and Interconnect in the host project.
• Azure: a hub VNet per region with Azure Firewall, gateways and DNS resolver, spokes per subscription, policy to enforce UDRs. Virtual WAN is the managed version.

Why it runs out

• Max pods per node is bounded by ENIs times IPs per ENI. The classic formula is ENIs x (IPs per ENI - 1) + 2. An m5.large has 3 ENIs with 10 IPs each, so 3 x 9 + 2 = 29 pods.
• The CNI pre-allocates: by default it keeps a whole spare ENI's worth of addresses warm on every node. Fifty nodes can reserve hundreds of idle addresses.

Fixes

Custom networking sketch

1. Add a secondary CIDR to the VPC, for example 100.64.0.0/16.
2. Create pod subnets per AZ from it.
3. Set AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true and create one ENIConfig per AZ.
4. Pods take addresses from the large range; nodes keep their routable addresses. Pod traffic leaving the VPC is SNATed to the node IP by default, so the 100.64 range does not need to be routable on-prem.

AWS numbers

Where it hurts

• An instance with MTU 9001 sends large segments to a host across the transit gateway or the internet. If ICMP type 3 code 4 is blocked, the sender never learns to shrink, and transfers stall after the handshake.
• On-prem over VPN: the tunnel overhead leaves less than 1500 for the inner packet. Clamp MSS on the tunnel interface of your router.

What I do

• Allow ICMP "destination unreachable, fragmentation needed" inbound in security groups for paths that cross gateways.
• For instances that mostly talk off-VPC, set the interface MTU to 1500:

  ip link show eth0
  sudo ip link set dev eth0 mtu 1500

• Clamp MSS on VPN routers. AWS's sample Cisco config uses ip tcp adjust-mss 1379.

GCP

VPC networks default to an MTU of 1460 and can be raised up to 8896. The guest OS must match the VPC setting.