# Firewalls, ACLs and Security Groups

Every network has at least three policy layers, and they fail in different ways. Knowing which one dropped the packet is most of firewall troubleshooting.

## The layers

| Layer | Stateful? | Applied to | Typical failure |
|---|---|---|---|
| **Router ACL** | No | Interface, direction | Wrong order, implicit deny |
| **Firewall / security appliance** | Yes | Zone pair | Wrong zone, missing NAT rule |
| **Host firewall** (nftables, ufw, Windows Firewall) | Yes | The host | Rule exists, service bound elsewhere |
| **Cloud security group** | Yes (stateful) | Instance/ENI | Missing egress rule, wrong port |
| **Cloud NACL** | No (stateless) | Subnet | Return traffic blocked |
| **Service mesh policy** | Yes (mTLS identity) | Workload | Identity mismatch |

A connection has to pass all of them. When it does not, the first question is not "which rule is wrong" but **"which layer dropped it"** — and the answer comes from where the packet stopped.

## Stateless vs stateful, and why it bites

- **Stateless** (router ACL, cloud NACL) evaluates each packet independently. You must permit the return direction explicitly, and for NACLs that means **ephemeral ports 1024–65535**.
- **Stateful** (firewall, security group) remembers the flow and allows the reply automatically.

The classic cloud bug: an inbound rule permits `tcp/443`, the NACL permits `tcp/443` in and **not** the ephemeral range out — so the TLS handshake completes one way and dies. The security group looks perfect.

## Rule order and the implicit deny

ACLs are evaluated top-down, first match wins, and end with an implicit `deny any` unless a permit-all exists at the bottom.

```text
permit tcp any host 203.0.113.10 eq 443     # allow
deny   ip  any any                          # everything else
```

Two mistakes dominate:

1. **A permit above a deny** that makes the deny useless — common after "temporary" additions.
2. **A `permit ip any any` at the end** that makes the whole list decorative.

Rule hygiene that survives audits: numbered rules, a comment on every entry, an owner, a review date, and a monthly count of zero-hit rules for deletion.

## Zones, not IPs

Mature designs write policy between **zones** (users, servers, guests, OT, management) rather than between hosts. Writing "permit 10.1.5.23 → 10.2.9.14:5432" produces thousands of rules and no understanding; "servers may reach the payment zone on 5432" produces a policy a human can review.

## What to log, and what logging costs

- Log **denies** at the perimeter (the interesting ones), sample allows.
- Log **new flows**, not every packet.
- Ship logs off-box; a firewall whose logs are on the firewall is compromised by definition.
- Keep the retention period the incident response plan requires, not the vendor default.

## Where it goes wrong

- **Debugging the wrong firewall.** The cloud security group permits it, the host `nftables` drops it. `tcpdump` on the host shows whether the packet arrived.
- **Asymmetric routing through a stateful device.** Outbound via A, return via B; the state table never sees the reply, and the connection resets.
- **NAT and policy in the wrong order.** The policy references the pre-NAT or post-NAT address depending on platform and direction. One vendor matches before translation, another after.
- **Rules with `any` source "temporarily".** The change ticket says two days; the rule outlives the employee.
- **Deny-all egress.** Good practice, and the reason a new service cannot reach its dependency. Egress rules need the same documentation as ingress.
- **Silent drops.** A `DROP` gives a timeout; a `REJECT` gives a connection refused. Choosing one without thinking makes every future debugging session longer.

## Teaching hook

**The building-security analogy.** The lobby guard checks who enters (edge firewall); each floor has a key card (segmentation); each office door has its own lock (host firewall). If a visitor cannot reach the fourth floor, the answer is not "the lobby guard is broken" — it is "find which door did not open".

Demo: `nc -vz host port` from three places (same host, same subnet, another subnet) and correlate each result with `tcpdump` and the relevant rule set. Refused vs timeout becomes tangible.

## Related Notes

- [[Network-Services/NAT-And-PAT.md]] — order of translation vs policy
- [[Security/Network-Security-Fundamentals.md]] — the threat model these rules serve
- [[Security/Zero-Trust-And-Segmentation.md]] — zones taken seriously
- [[Core-Protocols/Ports-And-Services.md]] — what the numbers mean