Why I Intentionally Broke My Production System (And You Should Too)

Over the weekend I happily went and crashed my production system deployed on AWS. Born not out of rage, but out of sober muscle memory from a long past Azure AKS story I will cover on a later post.

This time, I forced a CloudWatch alarm into ALARM on Ceiba's production account. The CloudWatch event published to SNS, which then invoked a Lambda, that then called StopInstances on the EC2 host serving api.useceiba.com and app.useceiba.com. I did it three times across two days. Yes, there was no staging copy involved, because there isn't one for now.

The reason was a billing guardrail I had built during the AWS migration and never watched run.

The problem with a guardrail nobody has fired#

The idea was that spend gets capped by an automatic shutdown before it reaches the imposed ceiling, and yes, that could change overtime if Ceiba's destiny is to scale. I will share on a later post on why I've been obsessing about FinOps stuff for the past few months.

So Ceiba runs under a hard cost ceiling of $80 a month, and the guardrail is a CloudWatch alarm on estimated charges with a threshold of $70. When it fires, it publishes to an SNS topic, and a Lambda subscribed to that topic calls StopInstances on the application host.

All of that has been configured and applied, but none of it had ever executed end to end.

On its own that is a normal state of affairs for a project this size, and I would not have lost a weekend over it. However, what made it worth testing was a change I had made to the Lambda's IAM permission. Its ec2:StopInstances grant had been narrowed from a wildcard down to a single instance ARN, which is the correct thing to do and also the kind of change that can break the automation it is attached to.

The chain would still look healthy from the outside and the alarm would fire and record a successful action. SNS would deliver the message and the Lambda would be invoked and would start running. Failing on the StopInstances call with AccessDenied, that error would appear exactly in the Lambda's own CloudWatch log stream and the alarm history would not show it. The SNS metrics would not show it either, and it would've remained silent wiht nothing paging me.

In a few words, the instance would keep running and the bill would keep climbing if the scoping were wrong, even after the alarm fires and lambda runs.

Chaos engineering when you are the only operator#

Chaos engineering is usually discussed at a scale I am not operating at, with fault injection tooling, scheduled game days, and teams whose job is to break things carefully. I have one production instance and one person to run it, AND guardrails that I hope will work when needed.

The core idea still applies. A safety mechanism that has never been triggered is just a claim about behavior, not a verified behavior. Reading the Terraform, checking the IAM policy, and reasoning through the sequence tells you what should happen. But it does not tell you what actually happens.

At some point the only remaining way to close that gap is to trigger it deliberately.

What this does require is knowing the blast radius first. Before starting I confirmed that real estimated charges were well below the $70 threshold to proof. That check would separate between a drill and an incident on top of it.

What the drill did#

The trigger was set-alarm-state on the billing alarm itself. Publishing a test message directly to the SNS topic would have been faster and would also have been the wrong test, because the Lambda does not inspect the message payload. Any message on that topic stops production, including one that says "ignore this." Driving the alarm is the only way to exercise the real path.

From there the sequence ran on its own: alarm to SNS, SNS to Lambda, Lambda to StopInstances, the instance stopping, the two readiness alarms noticing the outage, a manual start-instances to bring it back, and then recovery of everything downstream without touching the host.

There were three runs. The first two established the baseline. The third, two days later, closed a measurement gap the first two had left open.

The results were consistent across all of them:

• Alarm to StopInstances took 4 seconds, with no AccessDenied. The narrowed IAM permission, which was the main thing I wanted to test, worked every time.
• Both readiness alarms detected the outage and returned to OK on their own once the service recovered, firing their recovery notifications without intervention.
• The application reconnected to Postgres unattended using its restricted database credential rather than the admin one. That credential split had been made a few weeks earlier and had never been exercised by a restart.
• The Elastic IP was unchanged across the stop and start, so DNS required no intervention.
• The containers came back on their own through restart: unless-stopped, with no manual docker compose up at any point.

None of those were verified before this. All of them had been assumed and I'm glad I did find some peace of mind on the setup.

The gap found -> not measuring when the instance stopped#

After the Lambda called StopInstances, I wanted the exact time the instance finished stopping. That turned out to be harder than expected.

Polling describe-instances to observe the shutdown didn't do it, as the call returns the current state at the moment you ask, not a history of transitions.

On the second run I polled 19 seconds after the stop call and the instance already read stopped. The transition had happened somewhere inside that window and no polling interval I was willing to run would reliably catch it.

CloudTrail does not answer it either, though it looks like it should. CloudTrail recorded the StopInstances call, the time it was made, and the identity that made it. That is attribution, not state timing. It tells you who asked and when they asked, but nothing about when the instance actually changed state.

After some investigation, the mechanism that does answer it is EventBridge. AWS emits a state-change event for each EC2 transition at the moment it occurs, with the timestamp in the event itself, independent of anyone polling. I added a rule that captures those events for pending, running, stopping, stopped, shutting-down, and terminated, and writes them to a CloudWatch Logs group.

I deliberately did not filter that rule to a specific instance ID. This account runs exactly one EC2 instance, so filtering changes nothing about which events arrive, and of course, the unfiltered rule would keep working if the instance is ever replaced, instead of going quiet when pinning it to a specific instance id.

That case has already happened here once. In August, a terraform apply intended only to adjust an IAM policy replaced the production instance and destroyed the deployment environment file along with it. Working out what had happened, weeks later and from indirect evidence, took real effort. A lifecycle log would have shown the old instance terminating and a new one starting, with timestamps, in one place.

And that is the actual justification for building this. Better drill measurements ended up being a secondary benefit.

The third run had the rule live, and it produced what polling could not:

• running to stopped: 19 seconds
• pending to running: 4 seconds

The stopping event arrived in the same second as the Lambda's stop response, which means the API call and the state transition are now correlated by evidence instead of by assumption. The total outage on that run was about 2 minutes and 32 seconds.

The cost of running this is close to zero. EventBridge does not charge for AWS service events on the default bus. Looking at the numbers, CloudWatch Logs ingestion is $0.50 per GB, and these events are roughly 1 KB each. Storage at a full year of retention is $0.03 per GB-month. There is no added EC2, RDS, or networking cost, and does not move the billing alarm it was built to help validate.

What came out of it#

The first is that an untested control is an assumption. Every part of this chain had been written and reviewed carefully, and none of that told me whether the narrowed IAM permission still allowed the stop. Only firing it did.

The second is that instrumentation gaps do not show up until something is actually moving. I did not know that polling could not capture a sub-20-second transition until I needed it to. Reading the configuration would never have surfaced that.

The third is that tightening the IAM scope and keeping the automation working were not in conflict, which is worth knowing rather than hoping. Narrowing the permission was correct on paper, and now there is evidence it was also correct in practice.

There is also a fourth point that is easy to miss if guardrails are filed mentally under cost control. The same drill exercised the readiness alarms, which have nothing to do with billing. They watch for outages, and this was the first real outage either of them had ever seen. They caught it in 4 minutes and 2 seconds, measured from the stop to the first alarm entering ALARM. That number is the current gap between production breaking and the monitoring system knowing about it, and I would not have had it without running this.

What is still open and where this leaves it#

The drill also did not cover everything. Things like load behavior during the outage, a degraded-but-running failure that stops short of the instance going down, and anything on the database side were all out of scope for this pass.

The guardrail cost nothing to build and the drill cost nothing to run beyond a weekend of attention (EventBridge update a few days later).

Just believing the $80 ceiling holds is not the simply not the same as having evidence that it does, including a 4 second path from alarm to shutdown + the email notification on my phone while I' could be taking a nap.

The alternative was finding out the same thing later, from a bill I could of course spend responsibly cgh cgh...

If you have a guardrail you built, reviewed, and have never actually watched fire, it is worth asking what you would learn by firing it on purpose, at a time you choose.