The submission lifecycle

You'll be able to: explain why an accept-then-judge design needs a reconciler; distinguish a verdict about code from a verdict about machinery and say why conflating them is a bug; and use the type system to make "forgot to persist a state transition" a compile-time warning.

The problem with judging synchronously

Judging a submission means running the reader's code against every hidden case. That is seconds of CPU on a sandbox, and it is the one operation in this system whose duration is set by someone else's program — including programs that loop forever until a timeout kills them.

Doing that inside the request would mean an HTTP connection held open for the duration, a client timeout that loses the result while the work continues, and a slow submission occupying a connection that reads need. So submission is accepted, not answered:

Two orderings in that diagram are load-bearing:

The suite is fetched before the row is written. A lesson with no hidden suite is not a problem, and that is a 404 — so it must be discovered before anything is persisted. Reversing these would litter the table with rows for submissions that were never valid.

The task is spawned after the save. The judging task updates a row by id; if it started first, it could race the insert. Save-then-spawn means the row provably exists before anything tries to advance it.

Ownership across the request boundary

tokio::spawn returns a task that outlives the request handler, so it cannot borrow from it. The use case is therefore Clone, and cloning shares the same adapters:

let this = self.clone();
tokio::spawn(async move { this.judge_and_complete(submission, spec).await });

Each port is an Arc, so the clone is a handful of refcount bumps, not a duplicated connection pool. This is the Rust-shaped answer to "the background job needs the same dependencies as the request": share ownership explicitly, and let the type system prove the task cannot outlive what it uses.

Three states, and transitions you cannot forget

The transitions are functions returning a new value, and they are #[must_use]:

#[must_use]
pub fn judging(&self) -> Self { ... }
 
#[must_use]
pub fn completed(&self, outcome: SuiteOutcome, at: DateTime<Utc>) -> Self { ... }

That combination is doing real work. Because a transition produces a value instead of mutating in place, "advance the state" and "persist the state" cannot drift apart — and because the result is #[must_use], calling submission.judging(); and discarding it is a compiler warning rather than a submission that silently never moves.

The usual version of this bug — mutate the object, forget the save, wonder why the status is stale — is not available here. That is the difference between a rule you remember and a rule the compiler remembers for you.

Rejected and JudgeFailed are not the same failure

SuiteOutcome separates them deliberately:

Collapsing JudgeFailed into Rejected would be an outright lie: it would tell someone their correct solution is wrong because a sandbox was briefly unreachable. It also destroys the operational signal — rejections are the product working, judge failures are an incident, and a single "failed" status makes the second invisible inside the first.

Rejected carries first_failure rather than every failure, which is a teaching decision: the first case that broke is the one worth debugging, and a wall of failures is noise.

What happens when the process dies

Here is the cost of the design, stated plainly. The judging task lives inside the process that spawned it. When a pod is killed mid-judge — a deploy, an OOM, a node reboot — the task dies with it, and the in-task failure handler dies too. The row stays Judging forever.

Nothing in the request path can fix this, because there is no request left. So the fix runs at startup, before the server accepts traffic:

const JUDGE_GRACE: chrono::Duration = chrono::Duration::minutes(15);
...
if let Err(error) = submit.reconcile_unfinished(JUDGE_GRACE).await { ... }

The sweep finds rows still Pending or Judging and older than the grace window, and completes each with an honest verdict:

judging stopped when the server restarted — submit again

The grace window is the whole subtlety. Without it, a restarting replica would reconcile submissions another replica is actively judging — and the reconciler would overwrite a real verdict with a failure. Fifteen minutes is comfortably longer than any legitimate judging run, so anything older is genuinely abandoned rather than merely in flight.

One more detail worth the read. The sweep looks up each submission's suite only to report a believable 0/11 instead of a bare 0/0, and it deliberately ignores a lookup failure:

.ok().flatten().map_or(0, |spec| spec.cases.len())

That is presentation, not correctness, and presentation must never abort a repair. Likewise a failed update logs a warning and the loop continues — one unrecoverable row must not block the rest.

The layered failure handling

Read judge_and_complete as three tiers of "what if this fails":

let outcome = match self.repo.update(&submission.judging()).await {
    Ok(()) => self.judge(&spec, &submission.language, &submission.source).await,
    Err(error) => SuiteOutcome::JudgeFailed { passed: 0, total, detail: error.to_string() },
};
if let Err(error) = self.repo.update(&submission.completed(outcome, Utc::now())).await {
    tracing::warn!(id = %submission.id, %error, "could not record the outcome");
}

1. The mark-as-judging write fails → do not run the judge at all. Burning sandbox CPU on a verdict there is nowhere to put is waste; fail fast with JudgeFailed.
2. The judge itself fails → that is JudgeFailed too, produced inside judge.
3. The final write fails → warn and give up here. There is no client to tell and no useful retry in-process, so the row stays unfinished — and the startup reconciler is precisely the thing that will eventually close it.

Tier 3 is the interesting one: it is a deliberate hand-off from the request-time mechanism to the startup-time one, which is only safe because the sweeper exists.