Containers

The content pipeline

How a git push becomes a live lesson: one path at one speed, many repositories merged into one library, a symlink that makes the deploy atomic, and a commit hash you can verify from both ends.

Suggest an edit

The content pipeline

You'll be able to: design a publish path with no authoring backend; explain why a symlink swap is the atomic primitive that makes it safe; and reason about the real staleness window between a push and what a reader sees.

There is no content backend

The platform has no CMS and no content database. Publishing is:

git commit -m "new lesson" && git push

That is the whole authoring interface. Everything that would normally sit between an author and a published page — draft state, permissions, revisions, preview, rollback — is delegated to git, which already does all of it better.

The consequence worth stating: nothing the application serves can write to the content it serves from. A lesson becomes live by being a commit on main, and by no other route.

One qualification, added later. There is now an endpoint that lets a reader propose an edit from inside the app — but it opens a pull request, and the change still becomes live only by the path described in this chapter. The sentence that had to be retired is "there is no authoring endpoint"; the one that matters, above, still holds. See Content contribution, without git.

How the repository is organised

synapse-content/
  <book-slug>/
    book.json                     title, description, tags, order, slug, minutes
    01-<part-slug>/
      01-<lesson-slug>.md         the lesson
      01-<lesson-slug>.editorial.md   worked solutions (revealed separately)
      01-<lesson-slug>.tests.json     the suite — only its samples reach the browser
      _assets/_simulators/        a lesson's interactive widgets, served at /content-assets/…
      _assets/_diagrams/          the D2 sources of its figures — kept, never served
  _media/<book-slug>/<lesson-slug>/…    images, video and rendered frames, served at /media/…
  local-only/                     never published

A repository can also be a single book: if its root holds a book.json, the whole repository is that book and its chapters sit at the root. That is the shape of the satellite repositories below.

Four conventions carry most of the weight:

  • Numeric prefixes order things; slugs identify them. 01-the-system/ sorts first and appears in the URL as the-system. Reordering is a rename, not a database update — and the URL only changes if the slug changes.
  • Underscore-prefixed directories are structural, not content. The content walker skips anything beginning with _ or ., which is what lets _d2-blocks/ sit inside a book without becoming a phantom lesson. Every other .md under a book is a lesson, including one you forgot about.
  • Media lives at the repository root, not beside the lesson — one _media/ tree, addressed by book and lesson slug. It is served path-addressed with a shared cache hour rather than content-hashed, because authors replace files in place. What a lesson keeps beside it goes in _assets/ (ADR RS012): _assets/_simulators/ is served, gated exactly like the book, and _assets/_diagrams/ holds diagram sources that are never served.
  • Sidecars sit beside their lesson. Editorial content and test suites share the lesson's stem and are unlocked by kind: problem in the frontmatter. Only the suite's sample cases are ever serialised into a response; the rest stays server-side with the judge.

local-only/ is excluded from what ships. Drafts stay local until they are a commit on main.

One path, one speed

Everything in the repository reaches production the same way, which is the answer to "why hasn't my change appeared?": if it has been a minute, something is wrong — there is no slower path it might be taking instead.

git push to synapse-contentgit-sync polls (every 60 s)fetch the commit into .worktrees/<sha>atomically repoint /content/currentorigin re-reads the commit hash per requestnew content served — under a minute

Nothing is built, because nothing in the repository is a build input. Markdown, media, sidecars and diagrams are all data, read at request time: a d2 or mermaid figure is a fenced source inside the lesson, drawn on demand by a renderer that already knows nothing about this repository.

That was not always true. Architecture diagrams used to be a second path — a model file compiled into a container image, built in CI, promoted by a commit to the infrastructure repository, and rolled out by the deployment controller. Editing one character took minutes and a pod rollout, and the two speeds were the main source of the question above. Folding diagrams into the same request-time path removed the second path, an image, a workflow and a deployment.

Many repositories, one library

The library is no longer one repository. Since ADR RS005 it is the merge of several content sources: the primary ani2fun/synapse-content, plus any number of satellite repositories — guides such as the Java, Python and SQL books each live in their own. Which sources exist is a row in the content_source table, managed from /admin, not a redeploy.

Primary Satellite
How it arrives git-sync clones it into a volume the origin fetches a tarball from the GitHub API into its cache
How often every 60 s every 60 s (SYNAPSE_CONTENT_SYNC_SECONDS)
Atomic switch symlink flip <cache>/<id>/<sha>/ written in full, then a current symlink flipped
Owns its placement — the admin row: repository, branch, grouping, order
Owns its identity book.json book.json, including the slug

The slug lives with the content on purpose. It is the URL, so deriving it from the repository name would let a rename move every lesson, and an admin field would let a typo do the same. When two sources claim the same book, the first source wins and the conflict is a warning on the admin screen, not an outage — which is also what made each guide's migration out of the primary safe: the old copy kept serving until the new repository was verified.

A source can also be private (ADR RS011). Its books are served only to the readers on its list in content_source_reader: an anonymous caller gets 401, a signed-in caller who is not listed gets 403, and a listed reader gets the page with private, no-store so no cache keeps it. The same gate covers the book's /media files and its lesson widgets.

The sidecar does not update files in place. It fetches each commit into its own directory and then repoints a symlink:

/content/current -> .worktrees/ddbf2ae4a3922558c724176254466181c4aef228

That matters because content is read per request. Updating in place would mean requests landing mid-write see a half-updated tree — one lesson from the new commit, another from the old. A symlink swap is a single atomic operation: every request sees exactly one commit, either the old one or the new one, never a blend.

It also makes rollback trivial. The previous worktree is still on disk, so reverting is repointing the symlink, and it is why a bad content push is not an incident.

The version is the commit hash

The application derives its content version from the checkout's git hash, re-read per request. That single decision does several jobs at once:

  • The cache gets a correct key. A lesson response is derived from a known commit, so a cached copy is the right answer for that version rather than a guess about freshness.
  • Invalidation is implicit. New commit, new version, new cache key. Nothing to purge.
  • No redeploy is needed for content. The application re-reads the hash, notices it changed, and re-indexes.

Verifying it end to end

The claim "what is on GitHub is what is being served" is checkable from both ends, and here it is, checked just now:

$ gh api repos/ani2fun/synapse-content/commits/main --jq .sha
ddbf2ae4a3922558c724176254466181c4aef228
 
$ kubectl exec <pod> -c git-sync -- ls -l /content/current
/content/current -> .worktrees/ddbf2ae4a3922558c724176254466181c4aef228

Same commit. That is the pipeline's correctness property reduced to one comparison anyone can run, and it is a much better answer than reading logs and hoping.

The staleness window

An honest accounting of how long a push takes to be universally visible:

Stage Delay
Sidecar notices the commit up to its poll interval
Symlink repointed, app re-reads the hash immediate
Edge caches still hold the previous response up to max-age=60
Stale-while-revalidate serving up to 600 s for the unlucky 
cache-control: public, max-age=60, stale-while-revalidate=600

So a reader may see the previous version for up to a minute, and — if their edge node has not revalidated — a stale-but-serving copy for longer while the fresh one is fetched behind them.

That is a deliberate trade. stale-while-revalidate means a reader never waits for revalidation: they get the cached copy instantly while the update happens in the background. For a learning platform, a one-minute delay on a typo fix is invisible, and never showing anyone a spinner is worth far more.

Hashed assets are the opposite case entirely:

cache-control: public, max-age=31536000, immutable

A year, immutable — safe because the filename contains a content hash, so a changed asset is a different URL. Content-addressed assets never need invalidation; that is the whole trick.

Using git as the content database means no drafts UI, no preview, no per-lesson permissions. When does that stop being enough?

It stops when the people change, not when the content volume does.

The model works because there is one author who is comfortable with git. Every capability a CMS would provide already exists in that workflow: branches are drafts, pull requests are review, the local dev server is preview, git revert is rollback, and blame is the audit log. Building a CMS would reimplement all of it, worse.

The first thing to break is non-technical authors. Asking a subject-matter expert to resolve a merge conflict is asking them to stop contributing. That is where a CMS earns its cost — not because git is inadequate, but because the interface is wrong for the person.

That is the one that broke, and the answer turned out not to be a CMS. It was an editor page that opens a pull request — a new interface onto the same pipeline, rather than a second pipeline. Content contribution, without git is that design, including the two shortcuts that would have destroyed the property this chapter is about.

Second is partial permissions. Git access is all-or-nothing per repository. "This person may edit one book" needs either repository splitting or a permissions layer git does not have. That one broke too, and the answer was repository splitting: each guide now lives in its own repository with its own maintainers, merged back into one library (see Many repositories, one library above).

Third is scheduled publishing, which needs something that acts at a time rather than on a push.

Notably, scale is not on that list. More lessons make the repository bigger and the sidecar's fetch slower, and both are a long way from mattering.

The upgrade path is pleasant, though, which is what makes deferring safe: because content is plain Markdown in a known layout, a CMS could be added later as a thing that writes to the repository — keeping git as the source of truth and the whole delivery pipeline unchanged. That is a much better position than having chosen a database up front.

Mark as readBook versionda22e04