Moving Ceiba to AWS: A Small Deployment on a Real Budget

It's been sometime since my last post where I made plans to Running Ceiba on a Single Node First, where I shared on how a single node wasn't the final architecture destination, just a disciplined place to test things out, and find out whether Ceiba's boundaries were real before adding more infrastructure to hide behind.

That question got answered somehow with other workloads (projects) in mind that I could use to work on such homelab project (more on that on another post). Ceiba's production stack is on AWS now, app.useceiba.com and api.useceiba.com are serving real traffic over HTTPS, and the move happened without needing to touch the boundaries the single-node phase was there to prove. Control plane, runtime, and data tier moved as the same three things they already were. Very straight forward.

Now, what actually shaped this AWS deployment environment wasn't the cloud part. It was the budget.

The number that decided everything#

So I set myself a hard ceiling of $80 a month to start off, with a target baseline of $30-40. That came out not as a rough guess, but actually the number came from every single infrastructure decision that got checked against, before it got checked against anything else.

That constraint is more useful than it sounds. You don't want to learn the hard way as I did when I got billed over $800 just for playing around with AKS (Azure Kubernetes Service) for testing and development purposes in the end (yes amateurish).

Yes, a blank check tends to produce a fleet of managed services because each one is individually reasonable, but
a real ceiling forces you to ask, for every piece, whether it's paying for something you actually need yet. So instead of a reasonable bet, this is what I figured after actual investigation on the matter.

Here's where the $34/month baseline actually goes:

• Compute — EC2 t4g.small (Graviton), running control plane, runtime, and Redis: ~$12.26
• Database — RDS Postgres db.t4g.micro, single-AZ: ~$12.41
• EBS root volume — 30 GB gp3: ~$2.40
• DB storage — 20 GB gp3: ~$2.30
• Public IPv4 — 1 address: ~$3.65
• DNS — Route 53 hosted zone: ~$0.50
• Cache, TLS, observability, backups — Redis on-instance, Caddy, CloudWatch/CloudTrail (free tier), S3: ~$0.25
• NAT Gateway — skipped entirely: $0

Roughly 42% of the ceiling. Two of those lines are worth their own paragraph:

Graviton over x86. t4g.small and db.t4g.micro instead of the t3/m5 equivalents, purely for price-per-performance at this size. Nothing exotic, just the arm64 tax discount AWS has been offering for years that a lot of small deployments still don't bother collecting.

No NAT Gateway. This one is usually ease to get wrong by default, because "private subnet" and "NAT Gateway" get bundled together in most tutorials as if the second is required by the first. It isn't. A NAT Gateway exists so something in a private subnet can reach out to the internet. RDS never does that. It only accepts inbound connections from the app's security group. So there's a second, empty private subnet in a different AZ that costs nothing and does nothing except satisfy AWS's requirement that a DB subnet group span two Availability Zones, and the real private subnet holding the database has no NAT Gateway at all. That single skip is the largest fixed cost this deployment doesn't pay: roughly $33/month, for a gateway that would have sat there unused. And of course, Multi-AZ and Auto-scaling at the current scale are simply out for the moment, implying an instance failure as downtime, not failover right now. That's a deliberate cost decision at pre-revenue traffic, not an oversight.

So what's actually running?#

So we got a single EC2 instance running Caddy for TLS termination in front of the control plane and runtime, both in containers, alongside a self-hosted Redis. An RDS Postgres instance, single-AZ, in a private subnet that can't be reached from outside the app's own security group. Everything committed with Terraform, applied by hand, and reviewed before every apply, never through the console.

The terraform plan runs in CI on every pull request but never applies. No automated actor touches this infrastructure. If something's wrong, it's proudly my fault.

The part I didn't expect to have opinions about: how CI/CD authenticates#

As we understand, GitHub Actions needs to be able to push images and run commands on the production host. The easy way to wire that up is a long-lived AWS access key stored as a repo secret. However, I wanted to find a way to avoid that, sitting around waiting to leak.

So it came up to OIDC federation. GitHub Actions in this case authenticates to AWS by proving, cryptographically, which repo and which workflow it's running as, and AWS hands back short-lived credentials scoped to that specific claim. Avoding a stored key to rotate, and nothing useful to steal from a compromised dependency.

The part I'd actually flag to anyone setting this up is the trust policy as it has to be scoped to a specific branch, not repo:OWNER/REPO:*. That wildcard would let any branch in the repo assume a role that can push images and run shell commands on the production host, including a branch from a pull request. Two separate roles exist here, one read-only role trusted only for terraform plan on pull requests, and a deploy role trusted only for a pinned branch ref in the two app repos. Getting that scoping right the first time mattered more than getting OIDC working at all.

Cost guardrails, in two layers#

As mentioned earlier, the hard ceiling came as critical requirement in the begining, and a single budget alert emailing me (or someone) is a suggestion, not the type of control I wanted. So there are two layers.

AWS Budgets — one monthly cost budget at $80, alerting at 50%, 80%, 100% of actual spend, plus a forecasted-to-exceed alert that catches a trend before it becomes a bill.

An auto-shutdown circuit breaker — a CloudWatch alarm on the EstimatedCharges billing metric, threshold set at $70 (below the $80 ceiling, on purpose, to leave room to react), publishing to an SNS topic with two subscribers: an email address, and a Lambda with policy permission to stops the non-critical compute.

There are some caveats to this. It will not catch an instant spend spike, since billing metrics update every few hours, not in real time. This is a safety net for a slow leak, not a compromised-credential-spinning-up-dozens-of-instances scenario, which is exactly why the IAM scoping above matters more than this alarm does. And it deliberately does not touch RDS. Stopping an RDS instance only pauses billing for up to seven days, after which AWS silently restarts it. A cost-driven database pause has to be a deliberate, watched action, no place for the circuit breaker.

The bug that reframed how I think about CI reliability#

The CD pipeline for ceiba-control-plane and ceiba-runtime was crashing intermittently on the arm64 build step. First fix was a retry loop. It worked, in the sense that builds eventually succeeded, so I moved on.

Then it happened three times in a row with the exact same crash signature. Not flaky, deterministic. The retry loop hadn't fixed anything, it had just been paying a timing tax on every single deploy and calling the eventual success a fix.

The actual problem was that QEMU arm64 build emulation step on x86 runners. The fix was switching to native ubuntu-24.04-arm runners instead of emulating the architecture. In my opinion, a purely novice move if I had investigated GitHub runners from the beginning, but now we know.

So real before/after numbers, from actual run IDs, (not a synthetic benchmark): ceiba-control-plane went from roughly 21-22 minutes to 2 minutes 46 seconds. ceiba-runtime went from roughly 7.75 minutes to 2 minutes 11 seconds. Zero crashes on either repo's first native run.

A simingly obvious lesson here if you ask me, is that a failure that survives one retry attempt deserves root-causing before a second retry, not just a bigger timeout. I'd already broken that rule for testing ground once before I caught it.

What's still genuinely open#

The billing guardrail has never fired. The CloudWatch --> SNS --> Lambda chain is configured and applied. Nobody has ever driven it to ALARM and watched the instance actually stop. The Lambda's ec2:StopInstances permission was also recently narrowed from a wildcard to a single instance ARN, and if that scoping is wrong, the guardrail fails silently at the exact moment it's needed. Until the drill runs, this is a design, not a proven control.

Single-AZ, single instance. Clearly a deliberate cost decision at pre-revenue traffic, not exactly oversight, and there's a priced Phase 2 alternative and explicit reversal criteria for it. However, AZ failure or an instance failure is downtime, it's still true today.

Terraform state is local. Only one operator, one machine, no locking, no remote backup. The conditions that should trigger a move to S3 + DynamoDB are written down in advance, specifically so this doesn't quietly become permanent by default.

What I'd already do differently at scale#

Some of this is a trade-off I'd revisit on purpose, not a shortcut I'd be embarrassed by. Worth naming both kinds separately.

I'd stop hand-pinning a single AMI ID. An unrelated terraform apply once silently replaced the running production host, because the AMI reference was resolving to most_recent on every apply instead of a fixed ID. Pinning it fixed the real incident, but it's a single-instance answer. The moment there's a second instance or an autoscaling group, a launch template with a controlled rollout replaces this.

I'd add a synthetic check that runs independent of the deploy pipeline. Every readiness check this stack has only runs during a deploy. An outage happened between deploys, with nothing watching in the gap, because CD's own health checks had nothing to fail against when CD wasn't running. A scheduled external check, even something as blunt as a five-minute cron hitting /ready, is the obvious next layer. I haven't built it yet, and should be added deliberately at some point.

I'd reconsider self-hosted Redis before it becomes a second incident. It already caused one: an unhandled Redis error crashed the entire runtime process, not just the rate-limiting feature that depended on it. The fix made the self-hosted version safer. It didn't make it the right long-term answer. ElastiCache is the priced Phase 2 upgrade for exactly this reason.

Where this leaves things#

Moving from the single-node phase proving if Ceiba's boundaries were real, AWS is where I found out the answer was mostly yes, at the cost of a few things I hadn't priced correctly yet, and one CI assumption that got my head spinning until investigation rendered its fruits.

Moving from the architecture questions, I'll share on another post more on my experiences understanding traffic.

Thanks for reading up to here, until next time!