Write web and app security groups plus a private subnet NACL in Terraform

Tasks

1. Create the web security group allowing HTTPS from anywhere, with egress only to app on 8080.
2. Create the app security group allowing 8080 only from web, and HTTPS out.
3. Create a NACL for the private subnets with rules for the app port, HTTPS out and ephemeral return traffic.
4. Run a plan, apply, and verify the rules in AWS.

Solution

1. 1Security groups

   resource "aws_security_group" "web" {
     name        = "web"
     description = "Internet-facing load balancer"
     vpc_id      = var.vpc_id
   }
   
   resource "aws_security_group" "app" {
     name        = "app"
     description = "App servers behind the load balancer"
     vpc_id      = var.vpc_id
   }

2. 2Rules that reference groups instead of addresses

   resource "aws_vpc_security_group_ingress_rule" "web_https" {
     security_group_id = aws_security_group.web.id
     cidr_ipv4         = "0.0.0.0/0"
     ip_protocol       = "tcp"
     from_port         = 443
     to_port           = 443
   }
   
   resource "aws_vpc_security_group_egress_rule" "web_to_app" {
     security_group_id            = aws_security_group.web.id
     referenced_security_group_id = aws_security_group.app.id
     ip_protocol                  = "tcp"
     from_port                    = 8080
     to_port                      = 8080
   }
   
   resource "aws_vpc_security_group_ingress_rule" "app_from_web" {
     security_group_id            = aws_security_group.app.id
     referenced_security_group_id = aws_security_group.web.id
     ip_protocol                  = "tcp"
     from_port                    = 8080
     to_port                      = 8080
   }
   
   resource "aws_vpc_security_group_egress_rule" "app_https_out" {
     security_group_id = aws_security_group.app.id
     cidr_ipv4         = "0.0.0.0/0"
     ip_protocol       = "tcp"
     from_port         = 443
     to_port           = 443
   }

   Terraform removes the allow-all egress rule AWS adds to a new group, so egress is exactly what you write. The load balancer's health checks also come from the web group, so app_from_web covers them.

3. 3Network ACL for the private subnets

   resource "aws_network_acl" "private" {
     vpc_id     = var.vpc_id
     subnet_ids = var.private_subnet_ids
   }
   
   # Inbound: app port from inside the VPC (load balancer nodes live in the public subnets)
   resource "aws_network_acl_rule" "in_app" {
     network_acl_id = aws_network_acl.private.id
     rule_number    = 100
     egress         = false
     protocol       = "tcp"
     rule_action    = "allow"
     cidr_block     = "10.10.0.0/16"
     from_port      = 8080
     to_port        = 8080
   }
   
   # Inbound: replies to our outbound HTTPS calls arrive on ephemeral ports
   resource "aws_network_acl_rule" "in_ephemeral" {
     network_acl_id = aws_network_acl.private.id
     rule_number    = 110
     egress         = false
     protocol       = "tcp"
     rule_action    = "allow"
     cidr_block     = "0.0.0.0/0"
     from_port      = 1024
     to_port        = 65535
   }
   
   # Outbound: HTTPS to the internet through the NAT gateway
   resource "aws_network_acl_rule" "out_https" {
     network_acl_id = aws_network_acl.private.id
     rule_number    = 100
     egress         = true
     protocol       = "tcp"
     rule_action    = "allow"
     cidr_block     = "0.0.0.0/0"
     from_port      = 443
     to_port        = 443
   }
   
   # Outbound: replies to the load balancer go to its ephemeral ports
   resource "aws_network_acl_rule" "out_ephemeral" {
     network_acl_id = aws_network_acl.private.id
     rule_number    = 110
     egress         = true
     protocol       = "tcp"
     rule_action    = "allow"
     cidr_block     = "10.10.0.0/16"
     from_port      = 1024
     to_port        = 65535
   }

   A custom NACL denies everything not allowed, through its final * rule. Because NACLs are stateless, every flow needs a rule in each direction. The in_ephemeral rule also admits new inbound connections on high ports from anywhere; the app security group is what actually blocks those.

Verify

terraform plan
Plan: 11 to add, 0 to change, 0 to destroy.

aws ec2 describe-security-group-rules --filters Name=group-id,Values=sg-0a2b3c4d5e6f70819 \
  --query "SecurityGroupRules[].[IsEgress,IpProtocol,FromPort,ToPort,CidrIpv4,ReferencedGroupInfo.GroupId]" \
  --output text

False   tcp     8080    8080    None         sg-0f1e2d3c4b5a69788
True    tcp     443     443     0.0.0.0/0    None

APP1# curl -s -m 5 -o /dev/null -w "%{http_code}\n" https://checkip.amazonaws.com
200

Mistakes people make

• Forgetting the inbound ephemeral rule on the NACL. Outbound calls leave and the replies are dropped at the subnet edge.
• Allowing 8080 from the load balancer's IP addresses instead of its security group. ALB node addresses change.
• Assuming Terraform keeps the default allow-all egress rule. It removes it, so nothing leaves until you add egress rules.
• Mixing inline ingress blocks in aws_security_group with separate rule resources for the same group, which makes Terraform fight itself.
• Forgetting that this NACL, like any custom NACL, drops ICMP. If large transfers stall, add an inbound rule for ICMP type 3 code 4 (fragmentation needed) so path MTU discovery works.

On other platforms

az network asg create -g rg-lab -n asg-web
az network asg create -g rg-lab -n asg-app
az network nsg create -g rg-lab -n nsg-app
az network nsg rule create -g rg-lab --nsg-name nsg-app -n allow-web-to-app \
  --priority 100 --direction Inbound --access Allow --protocol Tcp \
  --source-asgs asg-web --destination-asgs asg-app --destination-port-ranges 8080
az network vnet subnet update -g rg-lab --vnet-name vnet-lab -n snet-app \
  --network-security-group nsg-app