Starphix
Network Engineering / Sources
The Lab / Sources / Network Engineering / Cloud-Modern-Networking/Cloud-Networking-Patterns.md

Cloud Networking Patterns

The primitives are the same in every cloud — networks, subnets, routes, gateways, policy — but the defaults differ enough to cause real incidents when a team moves between providers.

The translation table

Concept AWS Azure GCP
Isolated network VPC Virtual Network (VNet) VPC network
Subnet scope One AZ Whole VNet Region
Routing table Route table per subnet Route table per subnet Routes per network
Internet egress Internet Gateway + NAT Gateway Internet outbound / NAT Gateway Cloud NAT
Private service access VPC endpoints / PrivateLink Private Endpoint / Service Endpoint Private Service Connect
Policy (instance) Security Group (stateful) NSG (stateful) Firewall rules (stateful, VPC-level)
Policy (subnet) NACL (stateless) (NSG can attach to subnet) Hierarchical firewall policies
Peering VPC peering (non-transitive) VNet peering (non-transitive) VPC peering (non-transitive)
Hub-and-spoke Transit Gateway Virtual WAN / hub VNet Network Connectivity Center
Private DNS Route 53 private hosted zone Private DNS zone Cloud DNS private zone

Three differences cause most of the pain:

  1. Subnet scope. In AWS a subnet lives in one AZ; in Azure a subnet spans the VNet; in GCP a subnet is regional. Designs copied between clouds produce subnets in the wrong places.
  2. Statefulness. AWS SGs are stateful and NACLs are not; Azure NSGs are stateful at both levels. Port-opening advice from one cloud is wrong in the other.
  3. Transitivity. Peering is non-transitive everywhere, so a hub is mandatory beyond two networks — and the hub has a cost and a bandwidth ceiling.

The address plan is the design

Cloud makes address planning harder, not easier:

  • A VPC CIDR is difficult to change after resources exist.
  • Peered networks must not overlap, ever.
  • Kubernetes clusters add pod and service CIDRs, which must also not overlap.
  • Vendor SaaS private endpoints consume addresses in your space.

Plan for the next three years, not the next sprint:

10.0.0.0/8              # the whole estate, if you can get it
  10.10.0.0/16          # production region A
    10.10.1.0/24        # public subnets
    10.10.10.0/24       # application
    10.10.20.0/24       # data
    10.10.100.0/24      # management
  10.20.0.0/16          # production region B
  10.30.0.0/16          # non-production
  10.40.0.0/16          # shared services / hub
  100.64.0.0/10         # carrier-grade NAT space, reserved for expansion

Egress: the pattern people forget

Inbound gets designed; outbound gets discovered during the first incident.

  • NAT gateway per AZ for resilience, and it is billed per GB.
  • Egress filtering — allow the destinations services actually need, or a compromised workload exfiltrates freely.
  • Private endpoints for cloud services, so traffic does not leave the VPC.
  • A predictable egress IP for partners that allowlist, which means a stable NAT EIP.

Hybrid connectivity

Option Latency Cost Use
Site-to-site VPN Variable, internet-dependent Low Small sites, backup path
Dedicated interconnect Predictable High Production, regulated workloads
SD-WAN overlay Policy-managed Medium Many branches
Cloud WAN / Transit Managed Medium-high Multi-VPC, multi-region

Design for the case where the primary path fails: a VPN that has never carried production traffic will not carry it well on the day it must.

Where it goes wrong

  • Overlapping CIDRs between on-prem and cloud. Blocks the VPN entirely; the fix is renumbering one side, which is a project.
  • Non-transitive peering assumed transitive. Spoke A cannot reach spoke B through the hub until routes and gateways are actually configured.
  • Security groups referenced by IP instead of by group. Autoscaling changes the IPs and the rules rot.
  • NAT gateway in one AZ. The AZ fails, and so does all egress.
  • Logging off. VPC flow logs disabled to save cost, then no forensic trail during an incident.
  • Public subnets by default. The default VPC in AWS has an internet gateway and a permissive security group; people build in it and inherit the exposure.
  • Forgotten IPv6. Dual-stack subnets with no IPv6 security group rules are a wide-open door with no logs.

From the Network Engineering research vault · CC BY 4.0 · all notes · raw markdown