🚢 Deployment Strategies

Prerequisites: Encoding & Schema Evolution, Nonfunctional Requirements | You'll be able to: explain why old and new code always run simultaneously during a rollout and what that forces on your data formats; choose between rolling, blue-green, and canary for a given risk profile; and sequence an expand/contract schema migration under live traffic with no downtime.

🧨 The problem (why this exists)

You have version 1 of a service running on forty machines behind a load balancer, serving live traffic right now. You have version 2 built, tested, and ready. How do you get from one to the other?

The naive answer — stop all forty machines, swap the binary, start them again — has two fatal problems. The first is obvious: for the seconds or minutes it takes, the service is down, and every request fails. The second is subtler and far more dangerous: version 2 is new code. If it has a bug that only shows up under real production load — a memory leak, a bad query, a mishandled edge case — you have just deployed that bug to 100% of users at once, and rolling back means another full stop-the-world swap while the incident burns.

So you can't stop the world. You have to replace the running system while it keeps running. And the moment you accept that, you inherit a hard truth that this entire lesson is built around: during any rollout, old code and new code run at the same time. For some window — seconds for a fast rolling deploy, hours or days for a careful canary or a feature-flagged release — some of your machines answer requests with v1 while others answer with v2. They share the same database. They consume from the same queues. A v2 machine writes a record that a v1 machine reads a millisecond later; a v1 client calls a v2 server; a v2 producer publishes an event that a v1 consumer must handle.

This is exactly the coexistence problem from Encoding & Schema Evolution. DDIA states it plainly: because code changes cannot happen instantaneously, server-side systems use rolling upgrades — deploying to a few nodes at a time — so old and new code, and old and new data formats, coexist in the running system [DDIA2 p.161]. To keep working, the system must preserve compatibility in both directions at once: backward compatibility (new code reads data written by old code) and forward compatibility (old code reads data written by new code) [DDIA2 p.162]. That theory was abstract when you learned it. This is the lesson where it collects the bill. A deployment strategy is only as safe as the compatibility of the two versions it runs side by side.

💡 Intuition first

Imagine you have to replace every lane of a busy highway with fresh pavement — but you are not allowed to close the highway. You cannot dynamite all four lanes and pour new concrete; cars are on them. So you do one lane at a time: close lane 1, repave it, reopen it, move to lane 2. While you work, cars flow on the other lanes. Crucially, for the whole duration, old lanes and new lanes carry traffic simultaneously, and a car must be able to drive from an old lane straight onto a new one without falling into a pothole at the seam. The new pavement has to be compatible with the old at every boundary.

That is the entire mental model of live deployment. You never stop-and-swap; you ease the new version in, a slice of traffic or a slice of the fleet at a time, and you make sure the seams hold. The strategies in this lesson are just different answers to two questions:

1. How do you shift from old to new — replace machines gradually (rolling), stand up a whole second copy and flip (blue-green), or trickle a tiny fraction of traffic to the new version and watch (canary)?
2. How do you decouple "the code is deployed" from "the feature is live" — so you can ship dark and turn things on later (feature flags)?

And underneath all of them sits the hardest case, the one where the seam is not code but data: changing the database schema while both versions are reading and writing it. You cannot atomically change the schema and the code across a whole fleet in one instant, so you sequence the change into small, individually-safe steps. Hold onto the beginner takeaway: you can't just stop-and-swap a live system; you ease the new version in, and every version has to tolerate the ones running beside it.

⚙️ How it works

Think of the deployment strategies as a ladder, climbing from "simple and cheap but blunt" to "safe and precise but complex." You climb as high as the change's risk demands.

🔄 Rolling deploy — replace instances gradually

The default. Instead of swapping all forty machines at once, you take them out of the load balancer rotation a few at a time, upgrade them to v2, health-check them, put them back, and move on to the next batch. At any instant only a small slice is down for its swap, so the service as a whole never stops. This is DDIA's rolling upgrade exactly: a staged rollout, a few nodes at a time, monitored, with no downtime [DDIA2 pp.161–162]. Kubernetes ships this as the default Deployment strategy, governed by maxUnavailable (how many pods can be down at once) and maxSurge (how many extra pods it may spin up above the desired count) [web: Kubernetes docs, "Deployments — Rolling Update"].

Rolling is cheap — you need only a little spare capacity for the batch in flight, not a second fleet. But it has two costs. During the roll, v1 and v2 are both live and mixed, so they must be mutually compatible (back to the seam problem). And rollback is slow: to undo v2 you have to roll v1 back out the same gradual way, batch by batch, which is exactly what you do not want when v2 is actively causing an incident.

🔵 Blue-green — two full environments, flip the traffic

Stand up a complete second copy of production. "Blue" is the current live environment running v1; "green" is an identical environment running v2, fully deployed and warmed up but receiving no user traffic. You smoke-test green in isolation. When you're satisfied, you flip the router — usually the load balancer or DNS — so all traffic goes to green in one cut. Blue sits idle but intact [web: Martin Fowler, "BlueGreenDeployment"].

The prize is instant rollback: if green misbehaves, you flip the router straight back to blue, which is still running v1 exactly as before. Recovery is a routing change, not a redeploy. The price is double the resources — for the duration of the cutover you are paying for two full production environments — and the fact that the flip is all-or-nothing: the moment you cut, 100% of users are on v2, so a bug that passed your smoke tests still hits everyone. Blue-green also does not spare you the data seam: blue and green share the same database (you rarely duplicate production data), so v1 and v2 are still reading and writing the same store and still owe each other compatibility.

🐤 Canary — route a small percentage, watch, then ramp or roll back

Named for the canary in a coal mine: you send a small, sacrificial fraction of live traffic to the new version and watch its metrics before committing. Deploy v2 alongside v1, route perhaps 5% of real users to it, and compare the canary's error rate, latency percentiles, and business metrics against the v1 baseline. If the canary stays healthy through a bake time, you ramp — 5% → 25% → 50% → 100% — re-checking at each step. If it degrades, you route that traffic straight back to v1 and only the small canary cohort was ever exposed [web: Martin Fowler, "CanaryRelease"; web: Google SRE Book, "Canarying Releases"].

Canary is where deployment meets Observability: the decision to ramp or roll back is only as good as the SLO metrics you compare against, and mature setups automate the comparison (a statistical check on the canary-vs-baseline signals) rather than eyeballing a dashboard. The flow:

The benefit is that you cap the blast radius: a bug reaches 5% of users for a few minutes, not everyone for as long as it takes to notice. The cost is that canary is the slowest and most operationally involved of the three — you need traffic-splitting infrastructure, a solid baseline, good metrics, and the patience to bake at each step.

🚩 Feature flags — decouple deploy from release

The three strategies above all conflate two events: deploying the code and releasing the behavior. Feature flags (feature toggles) split them apart. You wrap the new behavior in a runtime conditional — if flag("new_checkout") { ... } else { ... } — and ship the code with the flag off. The new code is now deployed to production but dark: it runs on every machine yet does nothing user-visible. Later, independent of any deploy, you flip the flag on — for internal users first, then 1% of customers, then a region, then everyone — from a control plane, with no new rollout [web: Martin Fowler & Pete Hodgson, "Feature Toggles (Feature Flags)"].

This buys three things. Ship dark, release later: merge and deploy incrementally all week, turn the feature on when it's ready. Per-cohort rollout: enable by user segment, which is how you run A/B tests and gradual releases at the feature level rather than the fleet level. And a kill switch: if the feature misbehaves, flip it off in seconds — faster than any rollback, because nothing redeploys. The cost is that every flag is a branch in your code and your testing matrix; flags that never get cleaned up rot into permanent, tangled conditionals ("flag debt").

🧱 The hard one: schema and database migrations under live traffic

Every strategy so far shifts code. Data is harder, and it's where the coexistence problem bites hardest. Here's why an atomic swap is impossible: you cannot change the database schema and every machine's code in the same instant. There is always a window where the schema is in one state and part of the fleet expects another. If you ALTER TABLE to rename or drop a column that the still-running old code depends on, that old code starts throwing the moment the ALTER lands — instant, self-inflicted downtime. And it cuts the other way too: DDIA's Figure 5-1 hazard is that old code still writing the old format can silently drop or clobber a field the new code added, losing data [DDIA2 pp.162–163].

The way out is the expand/contract pattern (also called parallel change): never make a breaking change in one step. Instead, evolve through a sequence of individually-safe, backward-and-forward-compatible steps, so that at every moment both the currently-deployed code and the next version can operate against the schema as it stands [web: Martin Fowler, "ParallelChange"]. The canonical case — splitting a full_name column into first_name / last_name — runs like this:

Read the stages as a slideshow. Expand first (stage 2): add the new columns additively — nullable, with no code depending on them yet. Adding a nullable column is the one migration DDIA notes most databases can do cheaply, without rewriting existing rows, filling nulls on read [DDIA2 p.179]. Because the change is additive, old code neither knows nor cares — forward compatibility for free. Then deploy v2 that dual-writes (stage 3): every write goes to both the old full_name and the new columns, so no matter which version wrote a row, both representations are present. A backfill job walks the existing rows and populates the new columns for data written before the migration — this is "data outlives code" made concrete: the old rows are still in their original encoding until you explicitly rewrite them [DDIA2 p.179]. Once backfill completes and every row has both, switch reads to the new columns (stage 4) while still dual-writing, so a rollback to stage-3 code is safe. Finally, when you're confident, contract (stage 5): stop writing the old column and drop it. Each arrow between stages is a small, reversible deploy; at no point does a running version see a schema it can't handle.

↩️ Rollback vs roll-forward

When a deploy goes wrong you have two moves. Rollback — revert to the previous known-good version — is the default reflex and the reason blue-green and canary are prized. But rollback is only safe if the old code can still operate on the current state of the data. This is the deep reason the migration order above matters: because each step is backward-compatible, you can roll the code back a step without the schema betraying you. Once you've contracted — dropped full_name — you can no longer roll back to code that reads it. That's why the destructive step comes last and only after a long soak.

Roll-forward — ship a new fix on top — is the alternative when rollback is impossible (you've already dropped the old column) or when the bug is trivial to patch. The judgment call: roll back for anything you don't fully understand under incident pressure (fast, low-risk, buys time); roll forward when reverting would itself be destructive or when the fix is obviously correct. Tie this into Resilience & Incidents: a deploy is a leading cause of incidents, and a fast, rehearsed rollback path is a core part of limiting their blast radius.

⚖️ Trade-offs

Feature flags are orthogonal — they layer on top of any of these to separate deploy from release, giving a per-cohort rollout and a sub-second kill switch that no code-shifting strategy can match.

🔢 Numbers that matter

These are operational rules of thumb, not source-derived constants — treat them as starting points and tune with real data (see Estimation & Numbers).

• Canary traffic share: commonly start at 1–5% of live traffic, then ramp through roughly 5% → 25% → 50% → 100%. The share must be large enough that a real regression shows up above the noise in your metrics, and small enough that a bug is contained. Rule of thumb, not from source.
• Bake time per step: long enough to accumulate statistically meaningful data and to catch slow-burn failures (memory leaks, cache fill, cron-triggered paths) — often minutes for a small service, up to an hour or more per step for a critical one. Rule of thumb, not from source.
• Blue-green resource cost: ~2× production capacity for the duration of the cutover (two full environments live at once). Real cost.
• Rolling maxSurge / maxUnavailable: a typical Kubernetes default is 25% each — up to 25% extra pods spun up and 25% unavailable during the roll — trading rollout speed against capacity headroom [web: Kubernetes docs].
• Backfill throughput: the backfill in an expand/contract migration must be rate-limited so it doesn't saturate the database and starve live traffic — chunk the rows and throttle. A migration that backfills tens of millions of rows at full speed is a self-inflicted outage.

🏭 In production

Real systems make these strategies concrete. Kubernetes ships rolling updates as the default Deployment strategy (RollingUpdate with maxSurge/maxUnavailable), and the ecosystem layers progressive delivery on top: Argo Rollouts and Flagger drive automated canary and blue-green rollouts by talking to a service mesh or ingress to shift traffic in weighted steps and auto-analyze metrics [web: Argo Rollouts docs; web: Flagger docs]. Netflix built Spinnaker with Kayenta for automated canary analysis — a statistical judge that compares canary and baseline metrics and gates the promotion, so humans don't eyeball dashboards [web: Netflix Tech Blog, "Automated Canary Analysis at Netflix"]. This is Service Discovery & Mesh paying off: the mesh's weighted traffic-splitting is exactly what makes fine-grained canaries possible.

Feature flags are a mature product category — LaunchDarkly and similar platforms, plus in-house systems like Facebook's Gatekeeper, which is how large web companies "dark launch": ship code that's live in production but gated off, then ramp exposure by cohort. The classic example is launching a heavy new backend path to internal employees, then a small percent of users, watching load, and widening — deploy and release fully decoupled.

For schema migrations at scale, the online-schema-change tooling exists precisely because a naive ALTER TABLE locks the table: GitHub's gh-ost and Percona's pt-online-schema-change perform large MySQL schema changes without long locks by building a shadow table, backfilling, and cutting over [web: GitHub Engineering, "gh-ost"]. On the framework side, Rails' strong_migrations and similar linters block dangerous, non-additive migrations in code review before they ever reach production. The through-line is the same everywhere: additive expand, backfill under throttle, cut over, contract last.

🪤 Pitfalls & interview traps

Other traps interviewers probe:

• Forgetting that old code still writes the old format. Even after you deploy v2, the rolling window means v1 machines are still writing old-format rows. If v2 assumes every row is new-format, it breaks on the ones v1 just wrote. This is DDIA's Figure 5-1 hazard, and it's why dual-write and defensive reads exist [DDIA2 pp.162–163].
• Long-lived connections and in-flight work. WebSocket sessions, streaming RPCs, and long-running jobs pinned to a v1 instance don't just vanish when you drain it. Rolling and blue-green both need a connection-draining grace period, or you cut users off mid-request.
• Unthrottled backfill. Running the backfill at full speed saturates the database and takes down the live traffic you were trying to protect. Chunk and rate-limit.
• Canary sample too small or wrong metrics. 0.1% of traffic to the canary may never surface a regression that only appears under real load mix; and if you only watch CPU while the bug is a rise in p99 latency or a drop in checkout success, the canary passes and you promote a broken build. Canary is only as good as the SLOs it's judged against.
• Irreversible migrations blocking rollback. Once you've dropped a column or done a one-way data transform, "just roll back" is off the table — you're forced to roll forward under pressure. Keep the destructive step last and reversible for as long as you can.

✅ Check yourself

🔬 PoC — Proof of concepts

Blue-green, canary and progressive rollout, as the controllers that actually perform them:

• Argo Rollouts — a Kubernetes controller for blue-green and canary with automated analysis and rollback; the strategies in this lesson as runnable CRDs.
• Kubernetes — the built-in rolling update and readiness gates every strategy builds on; the baseline to understand first.
• Istio — traffic-weight shifting at the mesh, which is how a canary sends "5% of requests" to the new version independently of pod counts.

📚 Sources

• DDIA2 ch. 5 pp. 161–162 (rolling upgrades; old and new code coexist; backward + forward compatibility required in both directions)
• DDIA2 ch. 5 pp. 162–163 (Figure 5-1 forward-compatibility data-loss hazard: old code dropping fields written by new code)
• DDIA2 ch. 5 p. 179 (adding a nullable column without rewriting rows; "data outlives code"; migrating large datasets is expensive and deferred)
• [web: Martin Fowler, "BlueGreenDeployment"] · [web: Martin Fowler, "CanaryRelease"] · [web: Martin Fowler, "ParallelChange" (expand/contract)] · [web: Martin Fowler & Pete Hodgson, "Feature Toggles (Feature Flags)"]
• [web: Google SRE Book, "Canarying Releases"]
• [web: Kubernetes docs, "Deployments — Rolling Update" (maxSurge/maxUnavailable)]
• [web: Netflix Tech Blog, "Automated Canary Analysis at Netflix" (Spinnaker/Kayenta)] · [web: Argo Rollouts docs] · [web: Flagger docs]
• [web: GitHub Engineering, "gh-ost: online schema migrations for MySQL"] · Percona pt-online-schema-change