Skills101

AdvancedSystem Design · Lesson 18 of 19 · Updated ·

Distributed Transactions — Two-Phase Commit, the Saga Pattern and the Transactional Outbox

In a single database, a transaction is all-or-nothing: either every change in it is saved, or none is. That is the A (atomicity) of ACID, and it makes "move ₹500 from Asha to Ravi" safe. Now split the app into microservices, each with its own database: a payment service, a restaurant service and a delivery service. Placing one food order touches all three. There is no single database transaction that covers them. If the rider service fails after the payment succeeded, the customer has paid for food that will never come. This lesson covers how real systems handle that.

1. The problem: an order that is half done

Placing an order needs three steps in three services:

  1. Payment service: charge ₹450 to the customer.
  2. Restaurant service: the restaurant accepts the order.
  3. Delivery service: assign a rider.

Each step commits in its own database. If step 3 fails (no rider nearby), steps 1 and 2 are already committed. Something must undo them, or the system is left in a state nobody wants: money taken, kitchen cooking, no rider. Network calls make it worse: a request can time out after the other side did the work, so the caller cannot even be sure whether a step happened.

2. Two-phase commit (2PC)

Two-phase commit makes several databases act like one transaction, using a coordinator:

  • Phase 1, prepare: the coordinator asks every participant "can you commit?" Each one does the work, locks the rows, writes it durably, and votes yes or no.
  • Phase 2, commit or abort: if all voted yes, the coordinator tells everyone to commit. If any voted no (or did not answer), everyone aborts.

Two-phase commit: the coordinator sends prepare to the payment, restaurant and delivery databases, all three vote yes, then the coordinator sends commit; any no vote would make it send abort to all

2PC gives real atomicity, but it has serious costs:

ProblemWhy
Blockingparticipants hold locks between prepare and commit; if the coordinator crashes in between, they may wait with rows locked
Slowat least two network round trips to every participant, while holding locks
Availabilityevery participant must be up; one slow service slows every order
Supportmany modern services (most NoSQL stores, message brokers, third-party payment APIs) do not take part in 2PC

2PC is used inside some databases and in tightly controlled systems, but between microservices most teams avoid it.

3. The saga pattern

A saga splits the big transaction into a sequence of local transactions, one per service. Each step commits on its own. For every step there is a compensating action that semantically undoes it: refund a payment, cancel a restaurant order. If a step fails, the saga runs the compensations for the completed steps in reverse order.

def run_saga(steps, fail_at=None):
    done = []
    for name, action, compensate in steps:
        if name == fail_at:
            print(f"FAILED: {name}")
            for prev_name, undo in reversed(done):
                print(f"  compensate: {undo}")
            return "order cancelled, customer refunded"
        print(f"ok: {action}")
        done.append((name, compensate))
    return "order confirmed"

steps = [
    ("payment", "charge Rs 450", "refund Rs 450"),
    ("restaurant", "restaurant accepts order", "cancel restaurant order"),
    ("delivery", "assign rider", "release rider"),
]

print(run_saga(steps))
print()
print(run_saga(steps, fail_at="delivery"))
ok: charge Rs 450
ok: restaurant accepts order
ok: assign rider
order confirmed

ok: charge Rs 450
ok: restaurant accepts order
FAILED: delivery
  compensate: cancel restaurant order
  compensate: refund Rs 450
order cancelled, customer refunded

Saga with compensation, traced from the Python run: payment and restaurant steps succeed, the delivery step fails, so the saga cancels the restaurant order and then refunds the payment, in reverse order

Important properties of sagas:

  • No global locks. Other users can see intermediate states ("payment done, waiting for restaurant"). This is called the lack of isolation. Design for it, for example with an order status like PENDING that the app shows honestly.
  • Compensation is not a rollback. A refund is a new transaction; the charge still appears in the history. Some actions cannot be undone at all (an SMS already sent), so put them last, after the steps that may fail.
  • Every step and compensation must be idempotent (safe to run twice), because messages get retried (see the API design lesson).

4. Orchestration vs choreography

There are two ways to drive a saga:

OrchestrationChoreography
Who decides the next stepa central orchestrator (an "order saga" service or a workflow engine)each service reacts to events from the others
Floworchestrator: "payment, charge" → "restaurant, accept" → …PaymentDone event → restaurant listens and acts → OrderAccepted event → …
Good forcomplex flows, many steps, clear visibilitysimple flows with few services
Riskthe orchestrator is extra code to runthe flow is spread over many services and hard to follow

Workflow engines such as Temporal, AWS Step Functions and Camunda are often used as orchestrators because they persist the saga's state and retry steps for you.

5. The transactional outbox

Sagas and event-driven services depend on one small but tricky step: save to my database AND publish an event. If the service saves the order and then crashes before publishing OrderCreated, the other services never hear about it. If it publishes first and then the save fails, they react to an order that does not exist. This is the dual-write problem.

The outbox pattern fixes it: write the event into an outbox table in the same local transaction as the business data. A separate relay process reads unsent outbox rows, publishes them to the message broker (such as Kafka or RabbitMQ), and marks them sent.

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE orders (id INTEGER PRIMARY KEY, item TEXT, status TEXT)")
db.execute("CREATE TABLE outbox (id INTEGER PRIMARY KEY, event TEXT, sent INTEGER DEFAULT 0)")

def place_order(item, crash=False):
    try:
        with db:                                  # one local transaction
            cur = db.execute("INSERT INTO orders (item, status) VALUES (?, 'PENDING')", (item,))
            if crash:
                raise RuntimeError("server crashed")
            db.execute("INSERT INTO outbox (event) VALUES (?)", (f"OrderCreated:{cur.lastrowid}",))
    except RuntimeError as e:
        print("place_order failed:", e)

def relay():                                      # runs separately, every second or so
    for row_id, event in db.execute("SELECT id, event FROM outbox WHERE sent = 0").fetchall():
        print("publish to broker:", event)
        db.execute("UPDATE outbox SET sent = 1 WHERE id = ?", (row_id,))
    db.commit()

place_order("paneer biryani")
place_order("masala dosa", crash=True)
relay()
print("orders:", db.execute("SELECT id, item FROM orders").fetchall())
print("unsent events:", db.execute("SELECT COUNT(*) FROM outbox WHERE sent = 0").fetchone()[0])
place_order failed: server crashed
publish to broker: OrderCreated:1
orders: [(1, 'paneer biryani')]
unsent events: 0

The crashed order left no order row and no event: both were rolled back together. The successful order and its event were saved together, and the relay published it. If the relay crashes after publishing but before marking the row sent, it publishes the event again on restart. That is why consumers must be idempotent: delivery is at least once, not exactly once.

6. What to say in an interview

  • "Each service owns its database, so I cannot use one ACID transaction across them."
  • "I'll avoid 2PC between services because it blocks and needs every participant up."
  • "I'll use an orchestrated saga: charge, accept, assign rider; on failure, compensate in reverse: release, cancel, refund."
  • "Each service writes its state change and its event in one transaction using an outbox; consumers are idempotent because delivery is at-least-once."
  • "The user sees PENDING until the saga finishes, and irreversible steps like the SMS come last."

Common mistakes

  • Saving to the database and publishing to the queue as two separate steps (the dual-write problem).
  • Forgetting to design compensating actions, or assuming every action can be undone.
  • Non-idempotent steps that charge twice when a message is retried.
  • Using 2PC across microservices and third-party APIs that cannot take part in it.
  • Hiding intermediate states from users instead of showing a clear PENDING status.

Quick recap

  • Microservices with separate databases lose cross-service ACID transactions.
  • 2PC: prepare, then commit or abort; atomic but blocking, slow and fragile across services.
  • Saga: local transactions plus compensating actions in reverse order; no isolation, so design for visible intermediate states.
  • Orchestration (central coordinator) vs choreography (services react to events).
  • Outbox: business row + event row in one local transaction; a relay publishes; consumers are idempotent.

Practice what you learned

📝 Quiz: Distributed Transactions and Sagas5 questions · instant answers & explanations · +10 XP · counts towards your free certificateTake the quiz →