⏱️ Faults, Clocks & Time

Prerequisites: Networking Essentials, Nonfunctional Requirements | You'll be able to: reason about why a timeout tells you almost nothing, explain why you must never order events across nodes by wall-clock time, and design a lease + fencing-token scheme that survives a paused leader.

🧨 The problem (why this exists)

On a single machine, software is comfortingly deterministic. Run the same function on working hardware and you get the same answer; when the hardware genuinely breaks, the machine usually crashes cleanly — a kernel panic, a blue screen — rather than quietly handing you a wrong result [i]. This is deliberate: a computer is designed to fail loudly rather than lie, because wrong results are far harder to cope with than a clean crash [i].

Wire a few of those machines together with a network and that comfort evaporates.

Now some parts of the system can be broken while others keep humming along, and — this is the sharp part — you often cannot tell which. DDIA calls this a partial failure, and it is nondeterministic: the same multi-node operation may work now and fail a moment later, and you may never learn whether it actually succeeded [i]. "The fact that such partial failures can occur is the defining characteristic of distributed systems." [i]

That single fact reshapes everything downstream.

You cannot trust silence from a node, trust which of two cross-machine events came first, or assume a thread that checked its watch a moment ago still holds a valid lease. This is the humility lesson — why the network lies, why clocks lie, and the one mechanism (fencing tokens) that lets you build something correct anyway. Get it wrong and you get the split-brain corruption bugs that haunt half the case studies in this book.

💡 Intuition first

Three things conspire against you, and it helps to name them in plain language before any formalism.

The network can lose or delay anything, silently. When you send a request and wait for a reply, the request might be lost, or stuck in a queue, or delivered to a node that has died, or delivered to a node that is merely frozen, or the reply might be lost, or the reply might just be slow [i]. From where you sit, all six look identical: no response yet [i]. There is no packet that arrives saying "your earlier packet definitely died."

Every machine keeps its own time, and every clock is a little wrong. Each node has a cheap quartz oscillator that drifts, nudged periodically toward the truth by NTP over the same unreliable network [i]. So "now" on machine A and "now" on machine B disagree — usually by a little, occasionally by a lot — and neither knows by how much.

A process can freeze at any instant, for a long time, without noticing. A garbage-collection pause, a hypervisor suspending your VM to migrate it, the OS paging your memory to disk — any of these can stop your thread mid-function for hundreds of milliseconds or, historically, minutes [i]. During that freeze the rest of the world moves on. It may decide you are dead, elect a replacement, and hand your job to someone else — and you will not find out until you wake up and, perhaps, do something catastrophic.

The takeaway a beginner should leave with: the network lies, and so do clocks, and your own process can black out. Everything else is engineering around those three truths.

⚙️ How it works

📡 The network: no response means nothing

Datacenter and internet networks are asynchronous packet networks — send a packet and the network makes no promise about when, or whether, it arrives [i]. The usual defense is a timeout: wait some interval, then give up and assume the worst. But a timeout does not tell you the node is dead. On timeout you still don't know if the remote node received your request — it may be sitting in a queue and get processed later, long after you moved on [i].

This ambiguity is not academic.

Network faults are common even inside one company's datacenter — one medium-sized study found roughly 12 network faults per month, half cutting off a single machine and half cutting off an entire rack [i]. Redundant switches help less than you'd hope, because they don't guard against the human misconfiguration behind many outages [i]. And delays get wild: across cloud regions, round-trip times of up to several minutes appear at high percentiles, and even inside one datacenter a packet can be delayed more than a minute during a topology reconfiguration [i]. Assume any message can be delayed arbitrarily.

Doesn't TCP fix this?

TCP is "reliable" in that it retransmits dropped packets, reorders them, and checksums for corruption [i]. But it sits on top of an unreliable network and does not remove it: TCP infers loss from a missing ACK, can't tell whether your packet or the ACK was lost, can't guarantee a resend gets through, and eventually gives up and hands your app an error [i]. Worse, a delivery ACK only means the remote kernel received the bytes — the application may have crashed before processing them. A definitive "yes, I did it" requires a positive response from the application itself, not from TCP [i].

Here is what that ambiguity looks like when you send one request:

⌛ Why there is no good timeout value

If networks had a maximum packet delay d and servers had a maximum handling time r, every successful request would return within 2d + r, and you could set your timeout just above that [i]. Real asynchronous networks have neither bound, so 2d + r doesn't hold. That leaves you tuning a timeout by feel, and every choice hurts:

• Too short, and you declare a merely-slow node dead. That's dangerous — an in-flight action (sending an email, writing a row) may be performed twice when a replacement takes over [i]. And declaring a node dead shifts its load onto others; under high load, premature death declarations can trigger a cascading failure where every node declares every other node dead and the whole system stops [i].
• Too long, and you wait ages before reacting to a genuinely dead node, hurting availability.

Because the variability comes mostly from queueing — at congested switches, busy destination CPUs, the sender's own TCP rate-limiter, and noisy neighbors in a multitenant cloud [i] — the practical answer is to measure, not guess: track the round-trip-time distribution and adapt. Production systems use accrual failure detectors (the Phi Accrual detector in Cassandra and Akka) that output a suspicion level from observed jitter rather than a hard yes/no [i].

🕰️ Clocks: which one, for which job

Every modern machine actually has two clocks, and confusing them is a classic bug [i]:

• A time-of-day clock (CLOCK_REALTIME, System.currentTimeMillis) tells you the calendar date and time, synced by NTP so a timestamp means roughly the same thing across machines. But it can jump backward when NTP yanks it into line, jump for a leap second, or jump for DST — which makes it useless for measuring how much time elapsed [i].
• A monotonic clock (CLOCK_MONOTONIC, System.nanoTime) only ever moves forward, like a stopwatch. Its absolute value is meaningless (nanoseconds since some arbitrary point), so comparing monotonic readings across machines is nonsense — but for measuring a duration on one machine, it's exactly right [i].

The rule: durations → monotonic clock; points in time → time-of-day clock; ordering events across nodes → neither.

🎣 Why wall-clock ordering is a trap

It is tempting to order writes across nodes by comparing their time-of-day timestamps — Last-Write-Wins. It is also silently wrong. Quartz clocks drift (Google assumes up to 200 ppm), NTP over the internet manages a minimum error around 35 ms and can spike to a full second, and the reading you get is a false-precision point, not a range [i]. With multi-leader replication and LWW, a causally later write can carry a smaller timestamp and get dropped — silently losing data — even when the clock skew between nodes is under 3 ms [i]. NTP simply cannot be made accurate enough to prevent this, because its accuracy is bounded by the network round-trip time it rides on [i]. The honest way to order events across nodes is logical clocks — counters that capture happened-before relationships — which we build in Linearizability & Ordering.

The one honest exception is to stop pretending a clock reading is a point and treat it as a confidence interval [earliest, latest]. Google Spanner's TrueTime and Amazon ClockBound expose exactly that [i]. Spanner orders transactions by waiting out the interval: if interval A ends before interval B begins, B definitely happened after A. To keep the wait short it keeps uncertainty tiny — a GPS receiver or atomic clock per datacenter syncs to within ~7 ms — then deliberately waits the interval's width before committing so later readers never overlap it [i]. The lesson isn't "buy atomic clocks"; it's that being honest about uncertainty is what makes clock-based ordering safe — and most APIs (clock_gettime) never tell you whether your error is 5 ms or 5 years [i].

🧊 Process pauses: "I checked 10 ms ago" means nothing

Suppose a node holds a lease — a lock with a timeout that it must renew before expiry, so a dead node's lease can be reassigned. A naive check looks safe:

while (true) {
  request = getNextRequest();
  if (lease.isValid()) {   // checked... it's valid!
    process(request);      // ...so process it
  }
}

This is buggy in two ways [i]. It compares a lease expiry set on another machine against the local clock, so clock skew breaks it. And it assumes no meaningful time passes between isValid() returning true and process() running. But a process can pause right there, between those two lines, for 15 seconds. During the pause the lease expires, another node takes over — and the paused thread, on waking, sails straight into process() believing it still holds a lease it lost long ago [i].

These pauses are not exotic.

Historically, stop-the-world garbage collection could freeze a JVM for several minutes; modern collectors usually keep it to a few ms, but the tail is still there [i]. A hypervisor can suspend an entire VM for live migration at any moment for any length of time; the OS can page your memory to disk so an innocent memory access blocks on I/O; a context switch or SIGSTOP can descheduled you [i]. The uncomfortable conclusion: a node must assume it can be paused at any point, even mid-function, and resume without realizing time has passed [i]. "Checked 10 ms ago" tells you nothing about now.

🎟️ The payoff: knowledge, truth, and lies

Given all that, a node can know nothing for certain about any other node — it can only guess from messages it did or didn't receive [i]. Two defenses make this workable.

The majority rules. A node cannot trust its own judgment (it may be the one that's partitioned or paused), and the system cannot trust any single node (it may fail at any moment). So decisions — including "is node X dead?" — are made by a quorum, a vote among nodes. The usual quorum is an absolute majority (more than half), which is safe because two conflicting majorities cannot exist at once, and it tolerates a minority of failures: 3 nodes tolerate 1 fault, 5 tolerate 2 [i]. If a majority declares a node dead, it must step down. This is the seed of consensus, developed in Consensus & Coordination.

Fencing tokens. Leases enforce "only one of some thing" — one leader per shard, one writer per file — but a paused zombie (a former leaseholder that hasn't learned it was replaced) can still issue a write, and even a crashed leader's delayed write can arrive minutes late after someone else took over [i]. Shutting the zombie down (STONITH) doesn't help: it can't catch a request already delayed in the network, and by the time you detect the zombie the damage may be done [i]. The robust fix: the lock service returns a fencing token — a number that increases on every grant — and every write to storage must carry the client's current token. Storage rejects any write whose token is lower than one it has already accepted [i]. The zombie's stale token is smaller than the new leader's, so its write bounces harmlessly off.

This is the mechanism half the case studies in this book quietly depend on. Real systems already hand you the token: ZooKeeper's zxid or node cversion, etcd's revision number, Chubby's sequencers, Kafka's epoch numbers, and — in consensus — Raft's term or Paxos's ballot [i]. Where a separate lock service is overkill, storage that supports a conditional atomic write (succeeds only if unmodified since you read it) gives you the same protection — S3 conditional writes, Azure Blob conditional headers, GCS preconditions [i].

🎭 A note on Byzantine faults

Fencing tokens stop a node that errs honestly — slow, silent, or out of date, but truthful if it answers. They do not stop a node that lies: sends fake tokens, casts contradictory votes, deliberately subverts the protocol. That is a Byzantine fault, and tolerating it (the Byzantine Generals Problem) is expensive — most BFT algorithms need a supermajority of more than two-thirds correct [i]. For interview systems you can almost always assume nodes are unreliable but honest — they're your own machines in your own datacenter [i]. Byzantine tolerance is for aerospace (radiation flipping bits) and trustless multi-party systems (blockchains); note it and move on.

⚖️ Trade-offs

🔢 Numbers that matter

Keep these in your head — they're the back-of-envelope figures that justify "assume the network and clocks are hostile." (For the estimation toolkit, see Estimation & Numbers.)

🏭 In production

Real systems live and die by these mechanisms. ZooKeeper, etcd, and Chubby exist largely to hand out leases and monotonic fencing tokens (zxid, revision numbers, sequencers) so leader-elected workers can't corrupt shared state when one pauses [i]. Kafka attaches epoch numbers to partition leaders for the same reason [i]. The pattern recurs across this book's case studies: the Job Scheduler leases each job to exactly one worker (fencing stops a paused worker's late completion from double-running); Ticketmaster holds seats under short-lived locks a zombie must not re-grant; and Uber leans on leader ownership for dispatch regions.

On the clock side, Cassandra and ScyllaDB default to client-clock LWW and inherit the silent-data-loss hazard above — an operational footgun that pushes careful teams toward version vectors [i]. Spanner and YugabyteDB take the opposite bet: invest in tight clock sync (GPS/atomic clocks, or AWS ClockBound) and use the confidence interval to get global ordering [i]. And on failure detection, Cassandra and Akka ship the Phi Accrual detector precisely because a single hard timeout is too brittle against real-world jitter [i]. Operationally, if your software requires synchronized clocks, you must monitor clock offsets across all machines and eject any node that drifts too far — because a bad clock fails silently, drifting your data into subtle loss rather than crashing loudly [i].

Teams prove these designs survive failure with fault injection: Netflix's Chaos Monkey (chaos engineering), Jepsen for testing distributed databases, and deterministic simulation testing at FoundationDB and TigerBeetle, which mock the clock and network so a failing run can be replayed exactly [i].

🪤 Pitfalls & interview traps

• "Just use a distributed lock." The senior follow-up is always: what happens when the lock holder pauses? If your answer isn't "fencing tokens," you've shipped a split-brain corruption bug [i].
• Trusting a delivery ACK. TCP delivery means the kernel got the bytes, not that the app processed them; and reconnect-and-retransmit can duplicate data. Real confirmation needs an application-level response and idempotency [i].
• Using the time-of-day clock for timeouts. It can jump backward and make a duration negative. Use the monotonic clock [i].
• Assuming redundant hardware removes faults. It doesn't touch human misconfiguration, a leading cause of outages [i].
• Reaching for Byzantine fault tolerance. In a normal datacenter your nodes are honest; BFT's >2/3 supermajority cost is almost never worth it — say so and move on [i].

✅ Check yourself

🔬 PoC — Proof of concepts

Why "just use a timestamp" is a trap, and what production systems do instead:

• The trouble with timestamps — Kyle Kingsbury on how last-write-wins silently loses data when a wall clock steps backward; the concrete version of this lesson's warning.
• Spanner & TrueTime — Google's answer: don't pretend the clock is exact, expose its uncertainty and wait it out.
• AWS ClockBound — the same idea as an open-source daemon: a bounded clock-error interval you can query, so ordering decisions are made with the error, not despite it.

📚 Sources

• DDIA2 ch. 9 pp. 345–347 (faults & partial failure; reliable systems from unreliable components)
• DDIA2 ch. 9 pp. 347–352 (unreliable networks; six failure modes; TCP's limits; network faults in practice; fault detection)
• DDIA2 ch. 9 pp. 352–355 (timeouts; the 2d + r bound; queueing; Phi Accrual)
• DDIA2 ch. 9 pp. 358–360 (unreliable clocks; monotonic vs time-of-day)
• DDIA2 ch. 9 pp. 360–366 (clock drift 200 ppm; NTP ~35 ms error; MiFID II 100 µs; confidence intervals; Spanner TrueTime ~7 ms)
• DDIA2 ch. 9 pp. 362–364 (why wall-clock ordering loses writes; logical clocks)
• DDIA2 ch. 9 pp. 366–370 (process pauses; the buggy lease loop; GC pauses)
• DDIA2 ch. 9 pp. 371–377 (majority quorums; leases; zombies; STONITH; fencing tokens)
• DDIA2 ch. 9 pp. 377–379 (Byzantine faults; unreliable-but-honest assumption)
• DDIA2 ch. 9 pp. 385–387 (fault injection; Chaos Monkey; Jepsen; deterministic simulation testing)