Octet

Two instances in AWS cannot talk. How do you troubleshoot it, and what do flow logs not show you?

Show the answer

Say this first

I follow the path both ways: route tables at the source and at the destination, then security groups, then network ACLs in both directions, then the OS firewall and whether the app listens. Reachability Analyzer checks the configuration hop by hop without sending packets. VPC Flow Logs show ACCEPT or REJECT decisions by security groups and NACLs, but they do not log packets dropped for lack of a route, and they skip some traffic such as Amazon DNS and instance metadata.

Go deeper

Order of checks

  1. Routing, both directions. The return route is the one people forget.
  2. Security groups. Inbound on the destination, outbound on the source.
  3. Network ACLs. Both subnets, both directions, including ephemeral return ports.
  4. Host. ss -ltnp to see the listener, iptables or Windows Firewall, the app's bind address.

Flow logs

Output
2 123456789012 eni-0a1b2c3d4e5f60718 10.10.10.25 10.20.10.40 49152 5432 6 10 840 1731230000 1731230060 REJECT OK

Version, account, ENI, source, destination, ports, protocol (6 is TCP), packets, bytes, window start and end, action, log status. A REJECT points at a security group or NACL.

Not captured: traffic to the Amazon DNS resolver, the instance metadata service, the Amazon Time Sync service, DHCP, and Windows license activation. Records are aggregated over an interval (10 minutes by default, 1 minute as an option), so they are not real time.

Tools by cloud

AWSAzureGCP
Reachability AnalyzerNetwork Watcher IP flow verify, next hop, connection troubleshootConnectivity Tests
VPC Flow LogsVNet flow logsVPC Flow Logs, firewall rules logging
Traffic MirroringPacket capturePacket Mirroring

Traps

  • Treating the absence of a REJECT as proof the network is fine. Routing drops never show.
  • Checking only the source side. Most cloud connectivity bugs are on the return path.
How did you do?