Scalability Lessons Learned From Years of Running on AWS
We've written about why we default to AWS. This is the harder follow-up: what actually broke, what we assumed wrong, and what we'd tell ourselves before the first traffic spike caught us off guard.
We've written before about why we default to AWS for client infrastructure. This is the harder, more honest follow-up: the scalability lessons that only came from getting things wrong first.
Auto Scaling Isn't "Set It and Forget It"
Our first real traffic spike taught us that auto scaling groups don't magically solve capacity problems — they solve them on a delay. Scaling policies react to metrics that have already crossed a threshold, which means there's a real window between "traffic is spiking" and "capacity has caught up" where things are genuinely strained. For a gradual traffic ramp, this delay barely matters. For a sudden spike — a launch, a viral moment, a marketing push landing all at once — that delay is exactly when things break.
The fix wasn't a smarter scaling policy. It was pre-scaling ahead of known traffic events (we know when a client is launching a campaign; we don't have to wait for the metric to catch up) and setting more aggressive scaling triggers than felt intuitively necessary, because the cost of over-provisioning briefly is trivial compared to the cost of a launch-day outage.
Database Connections Run Out Before Compute Does
The first time we hit a real scaling wall, it wasn't CPU or memory — it was database connections. Every application server we auto-scaled added more simultaneous connections to the database, and the database has a hard connection limit that doesn't scale the same way compute does. We had healthy-looking application servers timing out on database calls because the database itself had become the bottleneck, silently, while every dashboard we were watching showed compute metrics that looked fine.
The lesson: scalability planning has to include the database layer specifically, not just the application layer. Connection pooling (PgBouncer or similar, sitting between your application and your database) solves this directly, and we now treat it as a default component rather than something we add after hitting the wall once.
Cost Doesn't Scale Linearly With Traffic (In Either Direction)
We assumed early on that infrastructure cost would track roughly proportionally with traffic. It doesn't, in both directions. Some costs are step-function — you don't pay incrementally more as you approach a capacity threshold, you pay a full additional unit the moment you cross it, whether you needed 10% more capacity or 90% more. Other costs (data transfer, certain managed service tiers) scale in ways that are much steeper than traffic growth, and you don't notice until the bill arrives.
The fix is treating cost monitoring as an ongoing practice, not a one-time budgeting exercise — reviewing the cost breakdown by service regularly enough to catch a cost curve that's diverging from the traffic curve before it becomes a large, surprising number.
The Cache Invalidation Problem Is Real, Even at Small Scale
"There are only two hard problems in computer science: cache invalidation and naming things" is a joke that stops being funny the first time a caching layer serves stale data to a user in a way that actually matters — an out-of-date price, a stale inventory count, a permission that should have been revoked. We added caching early to handle load, and underestimated how much complexity the invalidation logic would actually require to be correct, not just fast.
The lesson: caching buys you real performance, but the invalidation logic is not an afterthought you bolt on later. Design it at the same time you design the cache, and be honest about which data can tolerate staleness and which genuinely can't.
What We'd Tell Ourselves Before the First Spike
Load test before you need to, not after something breaks. Assume the database is your first real bottleneck, not your compute layer. Watch cost curves as closely as you watch performance metrics, because a cost surprise is still a business problem even when nothing technically failed. And build in headroom for the traffic event you can predict (a launch, a campaign) rather than relying on reactive scaling to handle it in real time.
None of this is exotic. It's the boring, specific stuff that only becomes obvious after you've hit it once — which is exactly why it's worth writing down.