System Design · Lesson 2 of 8 · Updated ·
Scaling and Load Balancing — Vertical vs Horizontal, Stateless Servers
Your app runs fine for 100 users. Then a college fest promotes it and 10,000 students sign up in an hour. Scaling is how you survive that.
1. Vertical vs horizontal
| Vertical (scale up) | Horizontal (scale out) | |
|---|---|---|
| How | bigger machine (more CPU, RAM) | more machines |
| Limit | the biggest machine available | practically none |
| Failure | one machine = single point of failure | one dies, others continue |
| Complexity | simple | needs load balancing, stateless design |
| Cost | gets expensive fast | grows gradually |
Most systems start vertical (simple) and go horizontal when needed.
2. Load balancers
A load balancer (LB) sits in front of servers and spreads requests among them. Examples: nginx, HAProxy, AWS ALB.
Algorithms
| Algorithm | How it picks a server |
|---|---|
| Round robin | 1, 2, 3, 1, 2, 3… |
| Weighted round robin | bigger servers get more requests |
| Least connections | the server with the fewest active requests |
| IP hash | the same user's IP always goes to the same server |
Health checks
The LB pings each server (e.g. GET /health every few seconds). A server that fails is removed from rotation automatically until it recovers. This is how one crashed server doesn't take the site down.
L4 vs L7: a layer-4 LB routes by IP and port (fast); a layer-7 LB understands HTTP and can route by URL path (/api to one group, /images to another).
3. Stateless servers
For horizontal scaling, any server must be able to handle any request. So servers shouldn't keep user state in their own memory.
- ❌ Session stored in server 1's RAM → the next request goes to server 2 → "please log in again".
- ✅ Store sessions in Redis or the database, or use signed tokens (JWT).
- ✅ Store uploaded files in object storage (S3), not the server's disk.
(Sticky sessions — always sending a user to the same server — work, but make scaling and failures messier.)
4. Auto-scaling
Cloud platforms can add servers when CPU or traffic is high and remove them when it drops — you pay only for what you use. Example rule: if average CPU > 70% for 5 minutes, add 2 servers.
5. Remove single points of failure
- Run at least two of everything important (servers, load balancers).
- Use multiple availability zones (separate data centres).
- Database: a replica ready to take over (next lessons).
Common mistakes
- Storing sessions or files on one app server's disk.
- One load balancer with no backup.
- Scaling app servers when the real bottleneck is the database.
Quick recap
- Vertical = bigger machine; horizontal = more machines + load balancer.
- LB algorithms: round robin, weighted, least connections, IP hash; health checks remove dead servers.
- Stateless app servers: sessions in Redis/DB/JWT, files in object storage.
- Auto-scale with demand; remove single points of failure.