The homelab case study

You'll be able to: reconstruct an incident from timestamps rather than memory; tell a single point of failure apart from a component that merely failed; and evaluate mitigations by which failure mode each one actually addresses.

The cluster

Four nodes joined into one Kubernetes cluster over a WireGuard mesh — one rented VPS at the edge, three machines at home behind a domestic router.

The last column is the story of this chapter. Three nodes have been up for weeks or months. wk-1 came up today, and this is not the first time.

Utilisation, measured at the same moment:

Capacity is not the constraint and is nowhere near becoming one. Measured under load, one replica sustains 15,440 req/s on less than one of eight cores — enough for millions of monthly readers even with the CDN switched off (the arithmetic). Every real limit in this chapter is about availability, not throughput — which is the opposite of what capacity planning usually assumes.

Where the pieces run

Note what happened here without anyone deciding it: wk-1 has 18 cores and 32 GB, four times the memory of any other node, so the scheduler put the resource-hungry things there. The most capable node accumulated the most critical state — and it is also the least reliable one. Nothing chose that; it is what a default scheduler does with an uneven cluster.

The outage, from the timestamps

Today the platform was down for roughly an hour and a half. Reconstructed entirely from what the cluster recorded:

Two facts make this worth studying.

The application was not on the node that failed. It ran on wk-2 throughout. wk-2 was healthy the entire time. The application still could not serve, because the database it depends on was on wk-1.

The application behaved exactly as designed. It fail-fasts when Postgres is unreachable rather than serving a half-working service with a lying readiness probe. Every one of those 32 restarts was the intended behaviour, and the error message names the cause precisely.

So this is not a bug report. It is the moment where a correct design decision and a bad availability outcome are the same event — and pretending otherwise is how systems stay fragile.

Single points of failure

The outage makes the real dependency graph visible:

The genuinely load-bearing observation is the fourth row. wk-1 is not one component's problem — it is the node where everything write-related lives. A single reboot removes the database and the sandbox together, and the application's fail-fast turns that into a total outage even though its own node is fine.

The one bright spot: because reads are edge-cached, a reader browsing already-cached lessons during the outage would have noticed nothing. The architecture's read-path decision paid out precisely when it mattered.

Why wk-1 keeps rebooting

Three causes, none of them exotic:

1. Unattended kernel upgrades reboot the machine, at whatever hour the timer fires.
2. After a power interruption it did not come back — the BIOS was set to stay off when power returned, so a brief cut meant the node stayed down until someone noticed.
3. A hung kernel had nothing to recover it — no watchdog, so a hang meant an indefinite outage rather than an automatic reset.

The second and third are the expensive ones, because they turn a transient fault into an outage that lasts until a human intervenes.

What was actually done

Each targets a specific failure mode, which is the standard a mitigation should be held to. Note that the watchdog could not be enabled on wk-2 — its firmware does not expose the device. An honest mitigation table records where the mitigation does not reach.

None of these prevent today's outage class. They shorten unattended downtime; they do not remove the dependency on a single node holding the only copy of the database. That fix is different in kind, and it is discussed — with its real costs — in Trade-offs.

The cheapest genuine improvement is not replication but placement: anti-affinity so the database and the sandbox do not share a node, and so neither shares a node with the thing most likely to reboot. That converts one reboot from a total outage into a partial one, at the cost of a scheduling constraint.

What this costs, honestly

The last row is the real availability limit. There is no on-call rotation and no paging. Mean time to repair is bounded below by how long it takes me to look — which is why the mitigations that matter most are the ones that recover without a human.