AdvancedSystem Design · Lesson 12 of 12 · Updated ·
Design a Unique ID Generator — Auto-Increment, UUID and Snowflake IDs
Every order, message, payment and short URL needs an ID. On one database, AUTO_INCREMENT gives 1, 2, 3… and the problem looks solved. But a chat system writing 70,000 messages a second across hundreds of servers cannot send every insert to one database just to get the next number. This lesson compares the standard answers and builds the one most interviewers expect: the Snowflake ID, first described by Twitter.
1. What do we need from an ID?
Clarify these first — they decide the design:
| Requirement | Why it matters |
|---|---|
| Unique across all servers | two orders with the same ID is a serious bug |
| Generated without coordination | asking a central server for every ID adds latency and a single point of failure |
| Roughly sorted by time | "latest 50 messages" becomes a simple range scan; database indexes stay compact |
| Fits in 64 bits | a BIGINT column is smaller and faster to index than a 128-bit value |
| Not easy to guess (sometimes) | sequential public IDs let anyone count your orders or scrape them |
2. The options
Auto-increment in one database. Simple and perfectly ordered. But every insert depends on that one database, and it does not work once data is sharded across many databases.
Auto-increment with steps. With N databases, database k hands out k, k+N, k+2N… (with 3 databases: 1, 4, 7… / 2, 5, 8… / 3, 6, 9…). No clashes — but adding a fourth database changes the step for everyone, and IDs are no longer in time order across databases.
ID ranges (ticket server). A small central service hands each app server a block of IDs, say 1,000,000 at a time. The server then hands them out from memory. Only one network call per million IDs, so the central service is not a bottleneck. If a server crashes, the unused part of its block is lost — usually acceptable. The URL shortener lesson used this idea.
UUID v4. A 128-bit random value like 9f1c2e7a-4b3d-4f8e-9a61-2c7d5e0b8a43. Any server can create one with no coordination, and the chance of two clashing is so small that it is ignored in practice. But it is twice the size of a 64-bit ID, and because it is random, new rows land all over the database index instead of at the end, which makes inserts slower on large tables.
UUID v7. A newer UUID version (standardised in RFC 9562, 2024) that puts a millisecond timestamp at the front, so UUIDs are sorted by time while staying globally unique. A good choice when 128 bits is acceptable.
Snowflake. 64 bits, time-sorted, created locally on each server with no network call. Let's build it.
3. The Snowflake layout
A Snowflake ID packs three numbers into one 64-bit integer:
| Part | Bits | What it gives |
|---|---|---|
| sign | 1 | always 0, so the ID is a positive signed 64-bit number |
| timestamp | 41 | milliseconds since a custom epoch you choose; 2⁴¹ ms ≈ 69.7 years |
| machine id | 10 | 2¹⁰ = 1,024 generators (often split as 5 bits data centre + 5 bits worker) |
| sequence | 12 | 2¹² = 4,096 IDs per millisecond on each machine |
Because the timestamp sits in the highest bits, a larger ID always means a later millisecond. Sorting by ID is sorting by creation time.
Uniqueness comes from the combination: two machines never share a machine id, and within one machine and one millisecond the sequence number is different for every ID.
Maximum rate: 4,096 × 1,000 = about 4 million IDs per second per machine — far more than one server usually needs.
4. Building it in Python
We pass the current time in as an argument so the output is the same every time you run it. A real generator would read the clock itself.
EPOCH = 1735689600000 # 1 Jan 2025 00:00 UTC, in milliseconds
class Snowflake:
def __init__(self, machine_id):
assert 0 <= machine_id < 1024
self.machine_id, self.seq, self.last_ms = machine_id, 0, -1
def next_id(self, now_ms):
if now_ms < self.last_ms:
raise RuntimeError("clock moved backwards")
if now_ms == self.last_ms:
self.seq = (self.seq + 1) & 4095 # 12 bits: 0..4095
if self.seq == 0:
raise RuntimeError("4,096 IDs used this ms - wait for the next ms")
else:
self.seq = 0
self.last_ms = now_ms
return ((now_ms - EPOCH) << 22) | (self.machine_id << 12) | self.seq
def decode(snowflake_id):
return {"ms_since_epoch": snowflake_id >> 22,
"machine": (snowflake_id >> 12) & 1023,
"seq": snowflake_id & 4095}
gen = Snowflake(machine_id=7)
t = EPOCH + 86_400_000 # exactly one day after the epoch
for now in [t, t, t, t + 1]:
i = gen.next_id(now)
print(i, decode(i))
362387865628672 {'ms_since_epoch': 86400000, 'machine': 7, 'seq': 0}
362387865628673 {'ms_since_epoch': 86400000, 'machine': 7, 'seq': 1}
362387865628674 {'ms_since_epoch': 86400000, 'machine': 7, 'seq': 2}
362387869822976 {'ms_since_epoch': 86400001, 'machine': 7, 'seq': 0}
Read the output line by line:
- The first three IDs come from the same millisecond, so only the sequence changes (0, 1, 2).
- The fourth ID is one millisecond later: the timestamp part goes up by one, and the sequence resets to 0.
- Every ID is larger than the one before it — time order is preserved.
86,400,000ms is one day, which is exactly what the decoder reads back.
The shifts do all the work: << 22 moves the timestamp above the 10 + 12 lower bits; << 12 moves the machine id above the 12 sequence bits; & 4095 and & 1023 keep only the lowest 12 or 10 bits when decoding.
5. The hard parts interviewers ask about
Assigning machine ids. Each generator needs a unique 0–1023 number. Options: a config value per server, or a coordination service such as ZooKeeper or etcd that leases ids to servers as they start. In Kubernetes, the ordinal of a StatefulSet pod (gen-0, gen-1…) is a common trick.
Clocks going backwards. Server clocks are corrected by NTP (Network Time Protocol) and can jump back a few milliseconds. If the generator kept going, it could repeat an old timestamp and sequence — a duplicate ID. Our code refuses instead. Real generators usually wait until the clock passes last_ms again, and alert if the jump is large.
Sequence overflow. If one machine needs more than 4,096 IDs in one millisecond, it must wait for the next millisecond. In practice this is rare.
The 69-year limit. Pick an epoch close to your launch date, not 1970, so you do not waste decades of the range.
Guessability. Snowflake IDs reveal creation time and roughly how busy you are. For public URLs where that matters, show a separate random token instead of the raw ID.
6. Choosing
- Small app with one database: auto-increment is fine.
- Many writers, 128 bits acceptable: UUID v7 (or v4 if order does not matter).
- Many writers, compact sorted 64-bit keys at high volume (messages, events, orders): Snowflake.
Common mistakes
- Using
MAX(id) + 1from application code — two requests read the same maximum and create the same ID. - Using random UUID v4 as the primary key of a huge, insert-heavy table without thinking about index fragmentation.
- Giving two servers the same machine id (for example by copying a config file), which silently creates duplicates.
- Ignoring clock changes on the generator servers.
Quick recap
- IDs should be unique, created without coordination, ideally time-sorted and compact.
- Auto-increment needs one database; ID ranges reduce coordination to one call per block.
- UUID v4 is random 128-bit; UUID v7 adds a time prefix so it sorts by time.
- Snowflake = 41-bit time + 10-bit machine + 12-bit sequence in 64 bits: about 69.7 years, 1,024 machines, 4,096 IDs per ms each.
- Watch for duplicate machine ids and clocks moving backwards.