Firewalls, ACLs and Security Groups

Every network has at least three policy layers, and they fail in different ways. Knowing which one dropped the packet is most of firewall troubleshooting.

The layers

A connection has to pass all of them. When it does not, the first question is not "which rule is wrong" but "which layer dropped it" — and the answer comes from where the packet stopped.

Stateless vs stateful, and why it bites

• Stateless (router ACL, cloud NACL) evaluates each packet independently. You must permit the return direction explicitly, and for NACLs that means ephemeral ports 1024–65535.
• Stateful (firewall, security group) remembers the flow and allows the reply automatically.

The classic cloud bug: an inbound rule permits tcp/443, the NACL permits tcp/443 in and not the ephemeral range out — so the TLS handshake completes one way and dies. The security group looks perfect.

Rule order and the implicit deny

ACLs are evaluated top-down, first match wins, and end with an implicit deny any unless a permit-all exists at the bottom.

permit tcp any host 203.0.113.10 eq 443     # allow
deny   ip  any any                          # everything else

Two mistakes dominate:

1. A permit above a deny that makes the deny useless — common after "temporary" additions.
2. A permit ip any any at the end that makes the whole list decorative.

Rule hygiene that survives audits: numbered rules, a comment on every entry, an owner, a review date, and a monthly count of zero-hit rules for deletion.

Zones, not IPs

Mature designs write policy between zones (users, servers, guests, OT, management) rather than between hosts. Writing "permit 10.1.5.23 → 10.2.9.14:5432" produces thousands of rules and no understanding; "servers may reach the payment zone on 5432" produces a policy a human can review.

What to log, and what logging costs

• Log denies at the perimeter (the interesting ones), sample allows.
• Log new flows, not every packet.
• Ship logs off-box; a firewall whose logs are on the firewall is compromised by definition.
• Keep the retention period the incident response plan requires, not the vendor default.

Where it goes wrong

• Debugging the wrong firewall. The cloud security group permits it, the host nftables drops it. tcpdump on the host shows whether the packet arrived.
• Asymmetric routing through a stateful device. Outbound via A, return via B; the state table never sees the reply, and the connection resets.
• NAT and policy in the wrong order. The policy references the pre-NAT or post-NAT address depending on platform and direction. One vendor matches before translation, another after.
• Rules with any source "temporarily". The change ticket says two days; the rule outlives the employee.
• Deny-all egress. Good practice, and the reason a new service cannot reach its dependency. Egress rules need the same documentation as ingress.
• Silent drops. A DROP gives a timeout; a REJECT gives a connection refused. Choosing one without thinking makes every future debugging session longer.

Related Notes

• NAT And PAT — order of translation vs policy
• Network Security Fundamentals — the threat model these rules serve
• Zero Trust And Segmentation — zones taken seriously
• Ports And Services — what the numbers mean