⏱️ Latency, Throughput & Percentiles

Prerequisites: Nonfunctional Requirements | You'll be able to: define response time, latency, service time, and throughput precisely enough to catch misuse; compute and read p50/p95/p99 from raw data and explain what each extra "nine" costs; quantify tail-latency amplification in a parallel fan-out and write a percentile-based SLO.

🧨 The problem (why this exists)

Your team ships a product-page service.

The dashboard reports an average response time of 87 ms — comfortably inside the "it should be fast" line someone wrote in the requirements document. Then the support tickets arrive: checkout sometimes hangs for three seconds, and the complaints are coming disproportionately from your biggest accounts. The dashboard is still green. Nobody is lying. The average is simply doing what averages do: blending millions of quick requests with a slow minority until the slow minority disappears from view.

This is not a hypothetical embarrassment.

• Amazon observed that the slowest requests to its internal services were systematically the ones from customers with the most data on their accounts — in other words, from the customers who had bought the most, the most valuable ones [i].
• The average said the system was fast. The people generating the revenue experienced something else.

The previous lesson argued that "fast" and "reliable" are empty words until you define them. This lesson supplies the definitions for "fast":

what response time, latency, and throughput actually mean; why response time is a distribution rather than a number; how percentiles read that distribution; how tails get amplified when one request fans out to many backends; and how these numbers become promises — SLOs and SLAs.

By the end, "it should be fast" becomes one precise sentence: "p99 response time under 1 second, measured at the client."

🛒 Intuition first

Stand in a supermarket queue. Three quantities describe your afternoon:

• Throughput is the store's number: how many customers per hour get through the checkouts. It says nothing about any individual's experience.
• Response time is your number: the clock from the moment you join the queue to the moment you walk away with a receipt.
• Service time is the cashier's number: the seconds actually spent scanning your items. Everything else — standing behind other people — is waiting. Time spent waiting rather than being served is latency in the strict sense: your request exists, but nothing is working on it.

The shopper and the store manager care about different numbers, and both are right.

• You feel response time.
• The manager buys checkout lanes based on throughput, because throughput determines how much hardware — how much money — the operation needs [i].
• A design conversation that doesn't separate the two goes in circles: "make it faster" might mean shorten each visit or push more people through per hour, and the fixes are different.

Now the queue teaches its second lesson.

It's nearly closing time. One checkout lane is open, and the person ahead of you has a mountain of groceries. Your basket holds two items, so your service time will be ten seconds — and your response time will be eight minutes.

One slow customer at the front delays everyone behind them, and that is head-of-line blocking. Now notice what the cashier sees. Their own records — "time spent per customer" — look perfectly healthy, because scanning your two items really did take ten seconds. The delay is invisible from behind the counter and obvious from where you are standing [i].

Finally, ask a hundred shoppers how long their trip took.

Most say two or three minutes; a few say fifteen. "The average was 3.5 minutes" describes almost nobody.

So line all hundred up instead, sorted from fastest to slowest. The person in the middle is the median — half did better, half did worse. The person 95th from the front marks the 95th percentile: nineteen of twenty shoppers did at least this well.

A percentile is nothing more mysterious than a position in that sorted line. Unlike an average, it is a statement about how many people had a given experience.

⚙️ How it works

🕰️ Four clocks, one request

DDIA2 pins the vocabulary down like this [i]:

• 📡 Response time — what the client observes: from sending the request to receiving the answer, including every delay anywhere in the system [i].
• ⚙️ Service time — the duration a service spends actively processing the request [i].
• 🚏 Queueing delay — time spent waiting rather than being processed. It occurs at several points: waiting for a free CPU core, waiting to buffer a response packet onto the network, waiting behind other requests [i].
• 💤 Latency — strictly, a catchall for time in which the request is latent: it exists but is not being worked on. Network latency is the slice spent traveling through the network [i].

In everyday engineering speech, "latency" and "response time" are used interchangeably. That is usually harmless, but this book flags the distinction wherever it changes a decision — and saying "service time" when you mean it is a quiet signal of precision in an interview.

The client's clock sees ① through ⑤ added together; the server's own instrumentation sees only ③ (and whatever slice of ② happens inside the process).

That gap is why two identical requests can have wildly different response times even when the code path is identical. The randomness comes from mundane places: a context switch to a background process, a lost TCP packet waiting for retransmission, a garbage-collection pause, a page fault forcing a disk read — even mechanical vibration in the server rack [i]. Jitter: Variation contributed by the network is called jitter [i].

Queueing is the biggest of these variance sources, and it compounds:

• for CPU-bound work, a server can only process about as many things in parallel as it has CPU cores; an I/O-bound server can hold far more requests in flight than it has cores, because most of them are blocked waiting on the network or disk rather than running [web: Eli Bendersky — Concurrent Servers, Part 2: Threads]. Either way, it takes just a few slow requests at the head of the line to hold up everything behind them — head-of-line blocking at datacenter scale.
• The delayed requests feel slow to their clients while looking fast in the server's own accounting, which is exactly why response time must be measured on the client side [i].

📉 Throughput, and the cliff at the edge of capacity

Throughput is the count: requests per second processed, or bytes per second [i].

For a given hardware footprint there is a maximum throughput, and response time and throughput are coupled by queueing: at light load, response time is roughly flat; as demand approaches capacity, arriving requests increasingly find the CPUs busy and wait, and queueing delay rises sharply — the familiar hockey-stick curve [i]. A system running near its capacity gets slow before it gets unavailable. That is also why a scalable system is defined through this lens: one whose maximum throughput can be raised significantly by adding resources [i].

It gets worse past the knee.

• When a near-overloaded system slows down, clients time out and retry, adding load to a system that was already drowning — a retry storm — and the system can lock itself into an overloaded, so-called metastable state until something resets it [i].
• The standard defenses — exponential backoff with jittered retry intervals, circuit breakers, token buckets, load shedding, backpressure [i] — each get proper treatment later in the book; for now, the takeaway is that the right-hand end of the throughput curve is not a place to operate.
• A practical trigger for app servers: start scaling when CPU utilization sits above ~70–80%.

Throughput is also how you describe load in the first place:

• requests per second, gigabytes of new data per day, checkouts per hour — sometimes as a peak rather than an average (simultaneously online users), plus shape parameters like the read/write ratio, cache hit rate, or data items per user [i].
• Two questions then frame every scaling discussion: with fixed resources, how does performance degrade as load grows? And to hold performance constant, how much must resources grow [i]?

One formula ties throughput, latency, and concurrency together: Little's Law.

• Little's Law says that the average number of requests a system holds in flight at any instant equals the arrival rate multiplied by the average time each request spends in the system: L = λ × W [web: J.D.C. Little — A Proof for the Queuing Formula L = λW].
• Worked example: a service handling λ = 2,000 requests/second with an average response time W = 60 ms (0.06 s) holds L = 2,000 × 0.06 = 120 requests in flight at any moment — that is the minimum thread-pool, connection-pool, or event-loop capacity the service needs before adding any safety margin.
• The same formula is why a growing tail is a capacity problem, not just a user-experience one: if p99 grows from 60 ms to 600 ms while throughput holds steady, the number of requests the system must hold in flight grows tenfold too, and undersized pools start rejecting or queueing work that used to fit comfortably.

📊 Percentiles: reading the distribution

Response time is a distribution:

• measure the same request a thousand times and you get a thousand different numbers, most clustered, a few far out [i].
• The arithmetic mean has a legitimate job — estimating throughput limits and capacity — but it is a poor answer to "how long do users typically wait?", because it doesn't tell you how many users experienced any given delay [i].

Percentiles answer that question directly.

• Take all response times in some window and sort them.
• The median (p50) is the halfway point: half of requests finish faster, half slower [i].
• Higher percentiles read the tail: p95, p99, and p999 are the thresholds that 95%, 99%, and 99.9% of requests beat.
• A p95 of 1.5 s means 95 out of 100 requests finish in under 1.5 seconds, and 5 take longer [i].
• Tail latencies: These high percentiles are called tail latencies, and they hit user experience directly [i].

Feel the difference on a toy dataset — ten measured response times, already sorted, in ms:

32   38   41   45   52   58   71   105   260   1900

The median is ~55 ms (between the 5th and 6th values). The p90 is 260 ms. The mean? 2,602 ÷ 10 = 260 ms — dragged by a single outlier all the way up to the 90th percentile.

Report the mean and you claim a "typical" experience that nine out of ten requests beat handily. Report p50/p95/p99 and the shape of reality survives.

Simplification: there are several conventions for indexing percentiles in small samples — nearest-rank versus interpolation. At monitoring volumes the difference vanishes. What matters at small n is that high percentiles are jumpy, which is why dashboards compute them over rolling windows of many requests.

How far up the tail should you care?

Amazon's answer: internal services specify response-time requirements at the 99.9th percentile, even though it affects only 1 request in 1,000 — because the slowest requests land on the customers with the most account data, i.e., the most valuable ones [i]. But Amazon also concluded that optimizing the 99.99th percentile was too expensive for the benefit: each extra "nine" chases an event ten times rarer, increasingly caused by random factors you cannot control — GC pauses, packet loss, page faults — so the cost curve bends up while the payoff shrinks [i].

🌪️ Tail-latency amplification: fan-out multiplies your tail

Here is where tail latency stops being a statistical footnote. A modern page load is rarely one backend call — a single user request commonly fans out to many internal services, and even when the calls run in parallel, the user waits for the slowest one [i].

Run the numbers.

• Suppose every backend is individually healthy: p99 of 1 second, so only 1% of its calls run long.
• Fan out to 100 of them and wait for all. If latencies were independent, the chance that every call is fast is 0.99¹⁰⁰ ≈ 37% — meaning about 63% of user requests are slower than 1 second.
• A 1-in-100 event per backend has become a 2-in-3 event per user. At a fan-out of 30, it's still 1 − 0.99³⁰ ≈ 26%.
• Tail-latency amplification: This is tail-latency amplification: the more backend calls an end-user request touches, the greater the fraction of end-user requests that end up slow [i].

Name the simplification: independence is an idealization.

Real fleets have correlated slowness — a shared overloaded node, a GC storm, a hot shard — which can concentrate the pain into fewer, worse requests or spread it further. The qualitative law survives the caveat, and it drives a hard design consequence: your internal services must hold far stricter latency targets than the user-facing promise, and the deeper and wider your call graph grows, the more your architecture is governed by its tails. That is precisely why Amazon manages internal services at p999 rather than at the median [i].

📜 From percentiles to SLIs, SLOs, and SLAs

Percentiles are how performance promises get written down:

• SLI (service level indicator) — the thing you measure, e.g., client-observed response time or the fraction of non-error responses. (The term is Google SRE vocabulary rather than DDIA's; [web: Google SRE Book, "Service Level Objectives"].)
• SLO (service level objective) — the target: e.g., median response time < 200 ms and p99 < 1 s, with ≥ 99.9% of valid requests returning non-error responses [i].
• SLA (service level agreement) — a contract that attaches consequences, such as refunds, if the SLO is missed [i]. Even defining what counts as "available" for an SLA is not straightforward [i] — which requests count as "valid" is itself a negotiation, and the performance envelope you promise is part of your API's contract.

Operationally, percentiles are computed over a rolling window of recent response times.

• The naive implementation — keep every measurement, sort the list each minute — works but is inefficient; production systems use compact approximation structures such as HdrHistogram, t-digest, OpenHistogram, and DDSketch [i].
• One rule matters more than the machinery: to combine response-time data across machines or across time windows, you must add the histograms — never average the percentiles [i].
• The Pitfalls section shows how that mistake manufactures fiction.

⚖️ Trade-offs

Choosing which number to optimize is a genuine trade-off decision, not a formality — each step up the percentile ladder buys protection for rarer events at steeply rising cost.

Measurement point is a second axis with the same shape: server-side metrics are cheap, precise about service time, and blind to queueing in front of the process and to the network; client-side measurement sees the truth users experience but is noisier and harder to collect [i]. Mature setups do both — and alert on the client-side numbers.

🔢 Numbers that matter

Anchor figures to carry into any design discussion — the estimation method that uses them lives in Estimation & the Numbers, which carries the full 2025 hardware table.

And the famous "latency costs revenue" figures deserve honest handling, because DDIA2's own sidebar shows the much-cited data is shaky [i]:

a 2006 Google claim found that a 400→900 ms slowdown cut traffic and revenue by roughly 20%. Google's more careful 2009 experiment contradicts that figure: it found that 400 ms of added latency cost only about 0.6% fewer searches per day. Bing's 2009 experiment told a different story again — a 2-second slowdown cut ad revenue by 4.3%. An Akamai study claiming 100 ms of latency costs up to 7% of conversions undermines itself, because its fastest pages were often near-empty error pages, which also convert poorly. A Yahoo study that did control for result quality found that ≥ 1.25 s of speed difference produced 20–30% more clicks on the fast side.

Latency clearly matters; no single magic number survives scrutiny. Quote the spread, not a slogan — and measure your own funnel.

🏭 In production

The measurement pipeline is where this lesson becomes an engineering practice.

• Real monitoring stacks don't store every response time forever; they maintain per-service, per-minute histograms using sketch structures — HdrHistogram, t-digest, OpenHistogram, DDSketch — and compute rolling p50/p95/p99 curves from them [i].
• The architectural rule embedded in that choice: store and transmit histograms, not precomputed percentiles, because histograms can be added across machines and time windows while percentiles cannot be meaningfully combined afterwards [i].
• Fleet-wide dashboards, region rollups, and month-end SLO reports all depend on that one decision made early.

Where you put the probe changes what you can see.

• Server-side timing misses queueing in front of the process and everything the network does — the exact places where head-of-line blocking hides — so serious shops track client-observed (real-user) response time as the SLI of record and keep server-side service time as a diagnostic [i].

Percentile SLOs then become the operating contract.

• Internally, Amazon-style organizations hold services to p999 targets so that fan-out amplification doesn't destroy the end-user experience [i]; externally, SLAs attach financial consequences — refunds — to missed SLOs [i].
• Beware the definitional fine print: deciding which requests are "valid," what counts as an error, and over what window availability is computed is genuinely hard [i], and contracts have been argued over less.

Finally, capacity and latency meet in production the ugly way.

• A service running hot doesn't degrade linearly: queueing pushes the tail out first (your p99 alarms fire while p50 looks fine), then timeouts trigger retries, and the retry storm can hold the system in a metastable overloaded state even after the original spike has passed [i].
• The mitigations named earlier — backoff with jitter, circuit breakers, token buckets, load shedding, backpressure [i] — are the production answer, and your percentile dashboard is the early-warning system that tells you they're about to be needed.

🪤 Pitfalls & interview traps

The other traps interviewers reliably probe:

• Quoting service time as response time. "The database lookup takes 10 ms, so the API responds in 10 ms" ignores queueing delay and two network traversals [i]. Follow-up you should expect: "Measured where?" The right answer is client-side, because head-of-line blocking makes server-side numbers optimistic [i].
• Treating p99 as the worst case. p99 says nothing about the worst 1%. At 10 million requests/day, 100,000 daily requests live beyond your p99 — and per Amazon's finding, they may skew toward your most valuable users [i].
• Forgetting that sessions amplify like fan-outs do. A user who performs 20 actions meets your p99 tail with probability 1 − 0.99²⁰ ≈ 18% — the same arithmetic as backend fan-out, with the same independence caveat. "1%" events are everyday events at session scale.
• Promising heroic percentiles. An SLO of p9999 sounds rigorous and is mostly a commitment to chase GC pauses and packet loss forever [i]. Say what each nine costs.
• Overestimating baseline latencies, then over-building. A common estimation error: candidates assume a simple indexed database lookup is slow, when it's on the order of 10 ms, then bolt on a cache the design didn't need — or add a message queue to "buffer" 5k writes/second that a tuned Postgres handles natively. Justify infrastructure with numbers, not vibes — it's one of the calibration signals covered in The Interview at 10,000 Feet.

✅ Check yourself

🔬 PoC — Proof of concepts

Measuring latency honestly is harder than it looks — these are the tools and talks that show why:

• HdrHistogram — Gil Tene's high-dynamic-range histogram: the standard way to record p99/p999 across many orders of magnitude without the averaging that hides the tail. Ports exist for most languages.
• How NOT to Measure Latency — Gil Tene's talk on coordinated omission, the single most common way a percentile number quietly lies about the system it claims to describe. The mechanism: a closed-loop load generator waits for each response before sending the next request, so a stall doesn't get recorded as one slow measurement — it gets recorded as no measurement at all for the requests that would have arrived during the stall. The load generator quietly deletes the worst data points from the sample, and the reported percentiles look far better than what real, open-loop traffic would have experienced [web: Gil Tene — How NOT to Measure Latency, InfoQ].
• Latency Numbers Every Programmer Should Know, interactive — the same constants this lesson uses, adjustable by year so you can watch which ones actually improved.

📚 Sources

• DDIA2 ch. 2 pp. 37–42 (describing performance; response time vs. latency vs. service time; queueing and head-of-line blocking; percentiles and tail latency; Amazon p999/p9999; tail-latency amplification; SLOs/SLAs; percentile estimation and histogram aggregation) — via digest ch02-nonfunctional-requirements.
• DDIA2 ch. 2 p. 38 (retry storms, metastable failure, overload mitigations); pp. 49–52 (describing load; two lenses on rising load).
• DDIA2 ch. 2 p. 41 (latency-vs-revenue studies: Google 2006/2009, Bing 2009, Akamai, Yahoo — reliability caveats as given in the source).
• [web: Google SRE Book, "Service Level Objectives"] — the term SLI only; all SLO/SLA substance from DDIA2.
• [web: J.D.C. Little — A Proof for the Queuing Formula L = λW, Operations Research, 1961] — Little's Law statement and the concurrency worked example.
• [web: Colin Scott — Latency Numbers Every Programmer Should Know] — same-region/cross-region round trip and in-memory cache-read reference figures. The DB-read and message-queue rows in "Numbers that matter" are not from this or any other cited source — marked Rule of thumb, not from source per house style rather than left as an unsupported "Reference figure."
• [web: Gil Tene — How NOT to Measure Latency, InfoQ] — the coordinated-omission mechanism.
• [web: Eli Bendersky — Concurrent Servers, Part 2: Threads] — CPU-bound vs. I/O-bound concurrency scope note.