🔔 Event-driven: CQRS, Outbox & CDC

Prerequisites: Stream Processing, Queues & Brokers | You'll be able to: (1) explain how event-driven systems decouple services and derive read models from an event log; (2) spot the dual-write hazard that silently loses or fabricates events; (3) choose between the outbox pattern and change data capture to emit events atomically.

🧨 The problem (why this exists)

Two systems must agree on what just happened, and there is no safe moment to tell them both.

A customer places an order. Your service writes the order row to its database. Then it publishes an OrderPlaced event so the warehouse can pick the goods, the email service can send a receipt, and the analytics pipeline can count the sale. That is two operations: a database commit and a message publish. They are not one atomic act. So ask the uncomfortable question: what happens if the process crashes between them?

If the database commit succeeds and the publish never happens, the order exists but no warehouse, no email, and no analytics ever hear about it — a lost event. Flip the order of operations and it gets worse: publish first, then commit, and a crash leaves a OrderPlaced event racing through your system for an order that was never saved — a phantom event. Either way, no error is raised. The order looks fine in the database. The bug surfaces days later as an unshipped package or a revenue number that doesn't reconcile.

This is the dual-write problem, and it is the reason a whole family of patterns exists.

The synchronous alternative — have the order service call the warehouse, the mailer, and analytics directly over HTTP and wait for each — trades the dual-write hazard for tight coupling: now a slow mailer stalls checkout, and any downstream outage takes the whole flow down. Event-driven architecture, event sourcing, CQRS, the outbox pattern, and change data capture are all answers to the same underlying question: how do you reliably turn one thing that happened into a stream of facts that many independent systems can consume, without losing events, fabricating them, or coupling everyone together?

💡 Intuition first

Start from a single idea that reframes everything: an event is a fact about something that already happened, written down once and never changed. "Seat 14C was booked." "The cart had item X removed." "Payment of $30 was captured." Past tense, immutable, self-contained.

Now the trick. Instead of storing current state and mutating it in place, you store the sequence of events and treat that log as the single source of truth. Current state is not something you keep — it is something you compute by replaying the events from the beginning:

state = fold(events)   # apply each event in order to an accumulator

Your bank balance is not a number the bank "has"; it is the sum of every credit and debit ever applied to your account. Available seats in a theater are the capacity minus every processed booking plus every cancellation. This is exactly the perspective the Data Models lesson introduced as event sourcing, and it is the same shape as the operation log behind a collaborative editor like Google Docs: the document is the fold of every keystroke operation, not a blob you overwrite.

Once the log is the source of truth, a second idea falls out for free.

If you can build one view by folding events, you can build many. A booking dashboard, a search index, a per-user cache, a badge printer — each is its own fold of the same log, shaped for the way it will be read. You stop asking "how do I store this data?" and start asking "which events happened, and what views do I want from them?"

That reframing is the heart of everything below. Emit events. Build views from them. The three hard problems — decoupling, splitting reads from writes, and getting the events out of the database safely — are all consequences of taking this idea seriously.

⚙️ How it works

📢 Event-driven architecture: react, don't call

In a request/response system, service A knows about service B: it calls B's API and waits. In an event-driven system, A knows nothing about who cares. It emits an event to a durable log — Kafka, Kinesis, a log-based broker (see Queues & Brokers) — and moves on. B, C, and D each subscribe and react on their own schedule.

The payoff is loose coupling in two directions.

At the system level, the log buffers for a slow or crashed consumer, so a failing analytics job cannot stall checkout and producers keep running — a local fault stays local [DDIA2 p. 549]. At the human level, teams own their consumers behind a well-defined event contract and evolve independently [DDIA2 p. 550]. The price is eventual consistency: consumers lag the producer, so for a moment the warehouse has not yet heard about an order the database already committed. And debugging gets harder — there is no single call stack tracing a request end to end; a business action becomes a diffuse ripple across many asynchronous consumers.

🪵 Event sourcing: the log is the source of truth

Event-driven systems emit events but often still keep mutable state as their system of record. Event sourcing goes further: the append-only event log itself is the source of truth, and application state is strictly derived [DDIA2 p. 102]. Events are named in the past tense and are never updated or deleted — a cancellation is a new event appended after the booking, not an erasure of it [DDIA2 p. 103]. This buys reproducibility (delete a buggy view and recompute it from the log), a built-in audit trail valued in regulated industries, and clear intent — "booking was cancelled" says more than a mutated row [DDIA2 pp. 103–104]. Mutable state and an append-only log "are two sides of the same coin" — state is always the result of folding a sequence of events [DDIA2 p. 509].

✂️ CQRS: split the write model from the read models

Here is where event-driven design meets the scaling ladder. In the Scaling Reads lesson you added replicas and caches; in Scaling Writes you denormalized so reads wouldn't need expensive joins. CQRS — Command Query Responsibility Segregation — is the natural endpoint of that ladder: stop forcing one representation to serve both writes and every query. Write commands in the form that is easy and correct to write (an event log, or a normalized write model); derive one or more read models, each optimized for a specific query, and keep them up to date by consuming the log [DDIA2 p. 102].

The classic diagram: one write model feeds an event log, and several read models fan out from it.

The crucial caveat: read models are updated asynchronously, so they are eventually consistent with the write model. A user may not immediately read their own write — CQRS's biggest downside [DDIA2 p. 511]. Every materialized view must also process events in exactly the same order as the log, or two views will disagree [DDIA2 p. 105].

⚠️ The dual-write problem, precisely

Now back to the hazard from the opening. The naive way to emit events is a dual write: the application writes to the database and separately publishes to the broker. This fails two ways. First, partial failure — one write succeeds, the other doesn't, and there is no error [DDIA2 p. 502]. Second, a race condition — two clients' writes can reach the database and the broker in opposite orders, leaving the two systems permanently inconsistent with nobody in charge of ordering [DDIA2 pp. 501–502].

The fix everyone reaches for first — "just publish inside the transaction" — doesn't work, because the message broker is a different system that cannot join the database transaction. What you can do is make both writes land in the same database, so a single transaction covers them atomically. That is the outbox pattern.

📤 The outbox pattern: one transaction, then a relay

Write the event into an outbox table in the same transaction as the state change. Because both rows commit together, you can never have the order without its event or vice versa — the atomicity you wanted, achieved entirely inside one database [DDIA2 pp. 507–508]. A separate relay (a message relay / publisher) then reads unpublished outbox rows and pushes them to the broker, marking them sent.

The relay guarantees at-least-once delivery: if it crashes after publishing but before marking the row sent, it re-publishes on restart. Consumers must therefore be idempotent (dedupe by event ID) — the same requirement you'll meet again in exactly-once processing. The outbox's other quiet benefit: the outbox table has its own schema, decoupling your internal database columns from the public event contract, so an internal column rename doesn't break downstream consumers [DDIA2 pp. 507–508].

🛠️ Hands-on: run the outbox

A runnable implementation lives at _proof-of-concepts/06-patterns/08-event-driven-cqrs-outbox-cdc/ in the repo root — a real Postgres (via ./run), a Unit of Work that writes the order and its event in one transaction, a relay that polls the outbox with FOR UPDATE SKIP LOCKED, and an idempotent consumer.

cd _proof-of-concepts/06-patterns/08-event-driven-cqrs-outbox-cdc
./run            # start Postgres + run the walkthrough
./run test       # mypy --strict + smoke (all three guarantees)
./run stop       # tear down

The walkthrough is precisely the diagram above, made to fail on purpose: five orders write atomically; a rejected order writes neither row nor event; then the relay is crashed after publishing two events but before marking them sent — so the retry re-publishes them and the broker ends up with 9 messages including 2 duplicates, which the idempotent consumer absorbs to apply exactly 7 effects. The atomic write and the SKIP LOCKED poll run against real Postgres; the broker and consumer are simulated in-process so the duplicate is countable — the README maps exactly what is real versus simulated.

🎣 Change data capture: no dual write at all

The outbox still asks the application to write an extra row.

Change data capture (CDC) removes even that. Every database already keeps a replication log — an ordered stream of every committed write, used to feed replicas (see Replication). CDC taps that log directly: a connector tails the commit stream and turns each row change into an event [DDIA2 p. 503]. The database is the single leader whose commit order is authoritative; every derived system becomes a follower applying changes in that same order [DDIA2 pp. 503–504]. There is no second write to lose, because there is no second write.

CDC is almost always asynchronous — the source database commits without waiting for consumers — so all the usual replication-lag caveats apply [DDIA2 p. 504]. Bootstrapping a brand-new consumer needs an initial snapshot consistent with a known log offset, after which it applies the ongoing stream; Debezium uses Netflix's DBLog watermarking to take that snapshot without stopping writes [DDIA2 pp. 504–505].

CDC versus event sourcing are often confused. CDC extracts low-level row changes from the replication log after the fact, and the application may not even know it's happening — easy to bolt onto an existing database [DDIA2 p. 507]. Event sourcing captures high-level application intent ("SeatBooked", not "row 14C updated") and is a deliberate architectural commitment where the log is the system of record [DDIA2 pp. 506–507].

⚖️ Trade-offs

The recurring decision is how to get events out of the database: outbox or CDC. Both fix the dual-write problem; they differ in effort, coupling, and fidelity.

A second axis worth naming — event-driven vs request/response — sits underneath the whole family: asynchronous events give loose coupling, buffering, and replay at the cost of eventual consistency and harder debugging; synchronous calls give immediate read-your-write and a simple mental model at the cost of tight coupling and cascading failures. Neither wins outright; you pick per interaction.

🔢 Numbers that matter

• Log throughput. Log-based brokers achieve millions of messages/second per topic via sharding, despite writing every message to disk, and get durability from replication [DDIA2 p. 497]. Appends are sequential, so an event log absorbs write bursts better than a random-access database and lets downstream views catch up at their own pace [DDIA2 p. 104].
• How much history the log buffers. A single 20 TB HDD writing sequentially at ~250 MB/s takes about 22 hours to fill [DDIA2 p. 499]. That is the floor: a disk-backed log can buffer at least a day of events, and real deployments keep days to weeks — which is also how long a consumer can be down before it misses data.
• Replication / consumer lag. CDC and CQRS read models are asynchronous, so budget for lag measured in milliseconds-to-seconds under health, and much longer while a consumer catches up after an outage. Never assume a read model is current the instant a write commits.
• End-to-end freshness target. Pushing a state change all the way from one device through logs and processors to another device's UI can be done in under one second end-to-end [DDIA2 p. 559] — the yardstick for "real-time" event-driven UX.

🏭 In production

CDC tooling. Debezium is the de facto open-source CDC platform, with connectors for MySQL, PostgreSQL, Oracle, SQL Server, Cassandra and more; it parses each database's replication log and emits change events, typically onto Kafka via Kafka Connect [DDIA2 p. 504]. Kafka's protocol has become a de facto event-stream standard, and Kafka Connect sinks export the stream into search indexes, caches, and warehouses [DDIA2 pp. 504, 550]. Netflix's DBLog algorithm (used by Debezium) is what lets a fresh consumer take a consistent initial snapshot without pausing production writes [DDIA2 pp. 504–505].

Event logs as product infrastructure. Stripe exposes an event log as a first-class API: every state change to a charge or subscription is an immutable Event object that merchants consume via webhooks — the same event-sourcing shape used to keep external systems in sync, and the backbone of the Stripe payments design. The ad-click aggregator is a canonical event-driven analytics pipeline: clicks are events streamed into windowed aggregations. In a news feed, CDC-style change events drive cache invalidation and materialized-timeline maintenance — a post event fans out to every follower's read model.

Read-model / IVM engines. Keeping CQRS read models current is materialized-view maintenance, and a class of streaming databases now automate it: Kafka Streams and ksqlDB maintain views from Kafka topics, while incremental view maintenance engines — Materialize, RisingWave, ClickHouse, Feldera — convert SQL queries into operators that recompute only the changed data instead of periodically rebuilding the whole view [DDIA2 pp. 516–517]. Dedicated event-sourcing stores exist too (EventStoreDB, Marten on PostgreSQL, Axon), though a plain Kafka log with stream processors is the common substrate [DDIA2 p. 105].

🪤 Pitfalls & interview traps

• Reading your own write from an eventually-consistent read model. After a command commits, the CQRS read model has not caught up yet. UI that immediately re-queries the read model shows stale data and users think their action failed. Mitigations: read the write model for the just-written entity, echo the command's result optimistically, or wait for the read model to advance past the event's offset — but never assume the read model is instantly current [DDIA2 p. 511].
• Ordering only holds within a shard. A log-based broker guarantees total order within a partition, not across partitions [DDIA2 pp. 496–497]. If related events (e.g., all events for one account) land on different partitions, consumers can apply them out of order. Route causally related events to the same partition by keying on a natural ID (user ID, account ID) [DDIA2 p. 498].
• Reprocessing replays side effects. Rebuilding a read model by replaying the log is a superpower — until an event handler sends an email or charges a card. Replaying then resends every confirmation and double-charges. Keep externally-visible side effects out of pure projections, or guard them with idempotency keys [DDIA2 p. 104].
• Immutable logs and the right to be forgotten. Event sourcing's "never delete" collides with GDPR erasure. A per-user log can be dropped wholesale, but a mixed log cannot; the usual escape is to store personal data outside the event or crypto-shred it (encrypt, then discard the key), which complicates recomputing derived state [DDIA2 pp. 104, 512].
• At-least-once needs idempotent consumers. Outbox relays and CDC connectors both redeliver on failure, so every consumer must dedupe — commonly by storing the processed event's offset or ID alongside its write and skipping duplicates [DDIA2 p. 528].

✅ Check yourself

🔬 PoC — Proof of concepts

Run it yourself. Transactional outbox — a real Postgres where the business row and the outbox row commit in one transaction, then a relay publishes and marks them sent; watch it stay consistent across a simulated crash mid-publish. From _proof-of-concepts/06-patterns/08-event-driven-cqrs-outbox-cdc/, run ./run.

Study real implementations.

• Transactional Outbox pattern — Chris Richardson's canonical description of exactly this pattern and why it beats a dual write.
• Debezium — production change-data-capture: tail the database log and emit change events, the CDC half of this lesson, used to drive the outbox at scale.
• Apache Kafka — the log these events flow onto, and (with Kafka Connect) the standard transport from a Debezium source to downstream read models.

📚 Sources

DDIA2 ch. 12 pp. 500–508 (CDC, event sourcing, dual writes, outbox) · pp. 495–499 (log-based brokers, retention) · DDIA2 ch. 13 pp. 541–559 (derived data, systems of record, write/read paths) · DDIA2 ch. 3 pp. 101–105 (event sourcing & CQRS) · [web: debezium.io], [web: stripe.com] (tooling references)