11 min read
Auto Scaling a Two-Tier App on AWS: ALB, ASG, and systemd the Hard Way
Deploying HashiCorp's demo-consul-101 app across private subnets with public/internal ALBs, dedicated systemd service users, and target-tracking Auto Scaling Groups — and every mistake I made getting there.
Quick navigation#
- Overview
- Architecture
- Why a NAT Gateway and a jump host
- systemd services with dedicated non-root users
- Security groups: SG-to-SG only
- Baking the AMI (and everything that went wrong)
- Wiring up the ALBs and target groups
- Auto Scaling with request count, not CPU
- Load testing and watching it actually scale
- Hitting a real AWS account limit
- Tearing it all down
- Lessons learned
Overview#
I deployed HashiCorp's demo-consul-101 app — a dashboard-service and a counting-service, the classic "counter" tutorial app — onto plain EC2 instances, without Consul yet (that comes later in the course this was for). No Docker, no Kubernetes. Just:
- Two Go binaries running as systemd services
- Both tiers spread across two Availability Zones, entirely in private subnets
- A public-facing ALB in front of the dashboard, an internal ALB in front of the counting service
- Auto Scaling Groups for both, scaling on real request volume
- A single bastion host for SSH access
heyfor load testing to actually trigger and watch the scaling happen
It sounds straightforward. It was not straightforward. This post is mostly about the things that broke.
Architecture#
Users
|
Internet Gateway
|
Public ALB (dashboard)
|
dashboard-service ASG (private subnets, 2 AZs)
|
Internal ALB (counting)
|
counting-service ASG (private subnets, 2 AZs)
Both ASGs run in private subnets only. A NAT Gateway gives them outbound internet access (for pulling release binaries at boot / bootstrap time), and a single jump host in a public subnet is the only way in via SSH.
Why a NAT Gateway and a jump host#
These solve two completely different problems, and it's worth being explicit about it because it's easy to conflate them:
- NAT Gateway — outbound only. Lets private instances reach the internet (in this case, to
curlrelease binaries from GitHub) without being reachable from it. - Jump host — inbound only. A bastion in a public subnet that SSH traffic goes through to reach the private instances. Has nothing to do with NAT.
I used a regional NAT Gateway rather than the older per-AZ zonal pattern — one logical resource that automatically expands to whichever AZ has running workloads, instead of manually provisioning and wiring up a NAT Gateway plus route table per AZ. Fewer moving parts, same effect.
One jump host, not one per AZ. It's an admin access path, not something serving user traffic — losing it temporarily doesn't take the app down, so it doesn't need the same redundancy as the actual service tiers.
systemd services with dedicated non-root users#
Both services run under their own dedicated system accounts, not the ubuntu login user:
useradd --system --no-create-home --shell /usr/sbin/nologin counting-service-runner
--system gives it a system UID, --shell /usr/sbin/nologin means it can never be SSH'd into or used interactively. The point is separation: the account you log in with (ubuntu) is not the account the process runs as. If the app is ever compromised, it doesn't inherit your login user's sudo access.
The unit file:
[Unit]
Description=Consul Demo Counting Service
After=network.target
[Service]
Type=simple
User=counting-service-runner
Group=counting-service-runner
WorkingDirectory=/opt/counting-service
Environment="PORT=9000"
ExecStart=/opt/counting-service/counting-service
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
Security groups: SG-to-SG only#
Every rule references a security group as the source, not a CIDR block, so nothing needs updating as the Auto Scaling Groups scale in and out:
| Security Group | Inbound |
|---|---|
dashboard-lb-sg | 80 from 0.0.0.0/0 (intentional — public) |
dashboard-sg | 8080 from dashboard-lb-sg, 22 from jump-host-sg |
counting-lb-sg | 80 from dashboard-sg |
counting-service-sg | 9000 from counting-lb-sg, 22 from jump-host-sg |
One deliberate detail: counting-service-sg allows traffic from counting-lb-sg, not from dashboard-sg directly. On the wire, the actual hop is dashboard → internal ALB → counting — the ALB's security group is the correct source, not the caller two hops away.
Baking the AMI (and everything that went wrong)#
The plan: build one instance by hand, verify the service runs correctly, then bake it into an AMI so the Auto Scaling Group launches clean, pre-configured instances with nothing to download or configure at boot. In theory this removes a whole class of boot-time failure. In practice, I found every way to still fail.
Bug 1 — unzip not installed. The base Ubuntu AMI doesn't ship unzip. My bootstrap script died at unzip -o counting-service_linux_amd64.zip with command not found, and because the script used set -e, everything after that line — including writing the systemd unit — never ran. Fix: apt-get install -y unzip before the download.
Bug 2 — the actual bug that cost the most time. After fixing the user account setup, I rewrote the systemd unit and ran:
sudo systemctl daemon-reload
sudo systemctl restart counting-service
restart only restarts a service that's already enabled. It does not set the "start on boot" flag. The service ran fine on that one instance — but the moment I imaged it into an AMI, every new instance the ASG launched from it booted with the service present but never started. Health checks failed, the ASG "helpfully" kept replacing the broken instance with more broken instances from the same AMI, and I spent a long time chasing what looked like a networking problem before realizing the fix was one word: enable --now, not restart.
Bug 3 — the release zip's internal filename. dashboard-service_linux_amd64.zip doesn't extract to dashboard-service — it extracts to dashboard-service_linux_amd64. My script's next line, chmod +x dashboard-service, failed on a file that didn't exist, and set -e killed the script again before the systemd unit ever got written. Fix: rename right after unzip.
unzip -o dashboard-service_linux_amd64.zip
mv -f dashboard-service_linux_amd64 dashboard-service
chmod +x dashboard-service
Bug 4 — sudo cat > file does not do what it looks like it does. While patching a broken unit file by hand over SSH, I tried:
sudo cat > /etc/systemd/system/dashboard-service.service <<EOF
...
EOF
This fails on a root-owned path, and the reason trips a lot of people up: the > redirection is performed by your own shell, before sudo ever runs cat. sudo only elevates the command, not the redirection target. The fix is sudo tee:
sudo tee /etc/systemd/system/dashboard-service.service > /dev/null <<'EOF'
...
EOF
Bug 5 — quoting WorkingDirectory. I wrote WorkingDirectory="/opt/dashboard-service" with quotes, copying the pattern from Environment=. systemd's Environment= directive strips quotes; WorkingDirectory= does not. The literal string "/opt/dashboard-service" — quote characters included — doesn't start with /, so systemd rejected it: WorkingDirectory= path is not absolute. Only Environment= wants quotes; plain paths like WorkingDirectory= and ExecStart= should be bare.
Bug 6 — stale references after rebuilding the VPC. After tearing down and rebuilding the whole environment, the launch template still pointed at a security group ID and an AMI ID that no longer existed — both destroyed with the old VPC. Nothing about the launch template's content was wrong; the IDs it referenced simply didn't exist anymore. Worth remembering: a launch template doesn't validate that its references still exist until you actually try to launch from it.
Wiring up the ALBs and target groups#
Two more mistakes here worth calling out:
Target group port is immutable. I initially created the target group on port 80 (copying the listener port) instead of port 9000, where the app actually listens. aws elbv2 modify-target-group has no --port flag — port, protocol, and target type are all set at creation and can't be changed after. The fix was delete-and-recreate, not modify.
Listener port and target port are two different things. The ALB listener is what clients connect to (port 80, standard HTTP, no port in the URL). The target group port is what the ALB uses to reach the actual app process on each instance (9000, or 8080 for the dashboard). They don't need to match, and conflating them is an easy mistake the first time you set one of these up.
Auto Scaling with request count, not CPU#
I deliberately skipped CPU-based target tracking. Both services are trivially lightweight — increment a counter, marshal a small JSON blob, done. Pushing meaningful CPU load onto a t3.micro running an app like that requires an unrealistic amount of concurrency, and t3's burstable-credit model adds a confounding variable on top of that.
Instead, I used ALBRequestCountPerTarget, which ties the scaling decision directly to what a load test actually generates:
aws autoscaling put-scaling-policy \
--auto-scaling-group-name dashboard-asg \
--policy-type TargetTrackingScaling \
--target-tracking-configuration '{
"PredefinedMetricSpecification": {
"PredefinedMetricType": "ALBRequestCountPerTarget",
"ResourceLabel": "app/dashboard-lb/<lb-id>/targetgroup/dashboard-tg/<tg-id>"
},
"TargetValue": 20.0
}'
One gotcha here too: the ResourceLabel has to point at the ASG's own ALB and target group. I once had it pointing at the wrong service's target group entirely (an ASG that had briefly been attached to the wrong target group during setup, with the scaling policy never updated to match after I fixed the attachment) — meaning it was scaling based on someone else's traffic, not its own. put-scaling-policy on the same policy name overwrites it in place, no need to delete first.
Load testing and watching it actually scale#
hey -z 3m -c 50 http://dashboard-lb-xxxxx.us-east-1.elb.amazonaws.com
A plain hey <url> with no flags only sends a quick default burst and finishes in under a second — nowhere near enough to trigger a target-tracking alarm, which needs the metric to stay above threshold across multiple consecutive one-minute evaluation periods, not just a brief spike. -z 3m keeps it sustained.
Watching this actually work end-to-end was the payoff for all of the above:
RequestCountPerTargetclimbs well past the target value- The CloudWatch alarm flips to
ALARMafter catching up (there's a real minute-or-two lag between the metric updating and the alarm re-evaluating — don't panic if it looks stuck) - The ASG's desired capacity increases, new instances launch, register with the target group, and pass health checks
- Once the load test finishes and request volume drops, the ASG scales back down on its own
That last point matters more than it sounds like it should. It's easy to demo the scale-out and call it done, but the scale-in is what proves the whole loop — the same mechanism that adds capacity under load also removes it once the load is gone, without anyone touching a console.
Hitting a real AWS account limit#
Partway through testing, new instance launches started failing with:
You have requested more vCPU capacity than your current vCPU limit of 8 allows
This is a real, per-region EC2 service quota — "Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances" — and it's set per account, independent of available billing credits. At 2 vCPUs per t3.micro, an 8 vCPU limit caps you at 4 instances running simultaneously, total, across the whole account: bastion plus both service tiers combined.
I tried switching to t2.micro to halve the per-instance vCPU cost, which would have doubled the effective headroom under the same quota. That hit a second, unrelated wall: the account was on an AWS "Free Plan" tier that only allows t3 instances — the entire t2 family was blocked outright, independent of the vCPU quota.
The actual fix ended up being a mix of patience and math: filing a quota increase request (which eventually got approved, though it initially required manual AWS Support review rather than the instant auto-approval smaller accounts sometimes get), combined with working out exactly how much headroom existed at any given min/max setting so I could still demonstrate both services scaling — including simultaneously — without exceeding it.
Tearing it all down#
Once done, tearing down cleanly matters as much as building it, mostly because AWS keeps billing you for anything left running. Order matters — delete in reverse dependency order or you'll hit DependencyViolation errors:
# ASGs first — terminates instances as a side effect
aws autoscaling delete-auto-scaling-group --auto-scaling-group-name dashboard-asg --force-delete
# ALBs and target groups
aws elbv2 delete-listener --listener-arn <arn>
aws elbv2 delete-load-balancer --load-balancer-arn <arn>
aws elbv2 delete-target-group --target-group-arn <arn>
# NAT Gateway, then release its EIP
aws ec2 delete-nat-gateway --nat-gateway-id <id>
aws ec2 release-address --allocation-id <id>
# Security groups need the right order too — anything still
# referenced by another group's rules will refuse to delete
aws ec2 delete-security-group --group-id <id>
# Subnets, IGW, then the VPC itself
aws ec2 delete-subnet --subnet-id <id>
aws ec2 delete-internet-gateway --internet-gateway-id <id>
aws ec2 delete-vpc --vpc-id <id>
Security groups that reference each other in their rules need to be deleted in dependency order too — delete whichever group nothing else still points to first, then work backward. Trying to delete them all in a single unordered loop just produces a string of DependencyViolation errors until the graph resolves itself.
Lessons learned#
enable --now is not the same as restart, and the difference only shows up later. A service that's running but not enabled looks completely fine right up until you image it or reboot the box — then it silently doesn't start, and the failure looks like anything but a systemd config issue at first.
sudo only elevates the command, never the shell's own redirection. sudo cat > /etc/some/root/path fails for a reason that isn't obvious the first time you hit it. sudo tee is the actual fix, every time.
Quoting rules in systemd unit files are directive-specific, not universal. Environment= wants quotes; most path-like directives don't. Assuming consistency across directives is how you get "path is not absolute" for a path that looks completely absolute to you.
A target group's port is permanent. Get it right at creation, because "just update it" isn't an option — you're deleting and recreating either way.
Pick the scaling metric that matches what you're actually testing. CPU is the default reflex, but for a lightweight service under a synthetic load test, request count per target is a far more honest signal of whether autoscaling is actually working.
Service quotas are per-account and per-region, independent of your bill. Having credits doesn't raise a vCPU limit. Knowing the exact math — instance type × vCPU count × max instances — turned a blocking wall into something I could work around while a support case processed in the background.
Tear down in the reverse order you built. Every dependency that blocked creation blocks deletion too, just backward. Skipping the order just means doing the same deletions twice.