The Architecture Decisions That Haunt You at Scale
Every backend works fine with 100 users. The problems show up at 10,000. And by then, fixing them is expensive.
Here are the lessons we've learned from scaling backends in production.
Lesson 1: Your Database Will Be the Bottleneck
It's almost always the database. Not the application server. Not the network. The database.
The most common issues we see:
- Missing indexes on frequently queried columns
- N+1 query problems (fetching related data in a loop)
- No connection pooling (each request opens a new DB connection)
- Storing large blobs in the database instead of object storage
The Fix
- Add indexes on every column you filter or sort by
- Use an ORM that supports eager loading (Prisma's include, Django's select_related)
- Use PgBouncer or a managed connection pooler
- Move files to S3 or Cloudflare R2
Lesson 2: Cache Aggressively, Invalidate Carefully
Redis is the single best investment you can make in backend performance. But caching introduces a new problem: stale data.
Our rule: cache anything that is read frequently and changes infrequently.
- User profiles: cache for 5 minutes
- Product catalog: cache for 1 hour
- Real-time inventory: don't cache
Lesson 3: Design for Failure
"We had 99.9% uptime until we didn't. And we had no plan for the 0.1%."
Every external service you depend on will fail at some point. Your architecture should handle this gracefully.
- Use circuit breakers for external API calls
- Implement retry logic with exponential backoff
- Have fallback behavior when non-critical services are down
- Test your failure scenarios before they happen in production
Lesson 4: Observability Is Not Optional
You cannot fix what you cannot see. Before you scale, instrument your application:
- Structured logging (JSON logs that can be queried)
- Distributed tracing (to find slow operations across services)
- Metrics and alerting (to know before your users do)
We use Datadog for most production systems. For smaller budgets, Grafana + Prometheus is excellent.
Lesson 5: Horizontal Scaling Is Easier Than You Think
Most developers think scaling means rewriting everything. It doesn't.
If your application is stateless (no in-memory sessions, no local file storage), you can run multiple instances behind a load balancer with almost no code changes.
The key is making your application stateless from day one.
The Architecture That Scales
For most applications, this stack handles 100,000+ users without heroics:
- Next.js or Node.js application (stateless, horizontally scalable)
- PostgreSQL with read replicas
- Redis for caching and sessions
- S3 for file storage
- CloudFront CDN for static assets
- GitHub Actions for CI/CD
Simple. Proven. Scalable.