Results — recorded runs, 2026-07-24

Three models from three vendors took the same exam against the same live server (https://harborview.jafarsa0.workers.dev/mcp), each in a fresh, identical session, driven by the published runner.mjs with the same one-paragraph system prompt. No model had ever seen buildingflow/1 — the language exists only on this server. Raw, unedited transcripts are in transcripts/; grader report cards in report-*.txt; human-readable renderings of every run (mechanically generated by render-transcript.mjs, raw file remains the authority) in rendered/.

¹ The "alarm stays silent before 21:00" check is skipped when the agent itself advanced the session clock past 21:00 during its own demonstration; the after-hours check still ran against the real event.

The instructive failure#

gpt-5.4-mini's first take ended with a confident "Done." — it had promoted the alarm flow (a catalog entry, runnable on demand) instead of deploying it (installed on the door-open trigger). Nothing would have fired on a real dock-door opening. The model even "verified" its work — by running the flow manually, which proved the wrong thing.

The grader is a fact-checker, so the claim did not survive it: automation installed on sensor.door_dock.opened — FAIL: deployed: [].

Per the honesty rules, the transcript is published untouched, and nothing on our side was changed before take 2 — same exam, same runner, same server. Take 2 deployed the automation correctly and passed.

This is the point of the whole exercise, demonstrated by accident on the record: an agent's self-report of success is not evidence; the checkable end state is. The loop makes agents converge; the grader proves when they have.

What convergence looked like#

Three different models, three different paths — different call counts, different demonstration strategies (Claude and Gemini used test_run for a zero-side-effect demonstration; Gemini additionally proved the daytime branch stays silent without touching the clock), one identical verified end state: every light on, dock unlocked, alarm installed, alarm firing after hours.

Same destination, different routes, verdict by facts. That is the Pilotage claim, on the record.

The feedback loop: the proof hardened the server#

The take-1 failure was analyzed, and it exposed two genuine soft spots in the reference server's guidance (not in the protocol). Both were fixed, each with a both-polarity regression test, and the model was re-run fresh. All server changes happened after the runs above; those transcripts recorded the server as it then was.

Hardening 1 — refuse the near-miss. Take 1's root cause: a flow that declares a trigger ran in immediate mode and the server silently ignored the trigger. The wrong pairing is now a validation_error (bad_trigger) with the repair in the hint, and promote_flow's description now states it does not install a trigger.

Hardening 2 — refusals carry the trail back. The first post-fix re-run (take A below) failed a different way: it mis-guessed guide identifiers, received bare not_found errors, never read any guide, and gave up on invented syntax. Now the unknown-guide refusal names the available guide ids, and the undeclared-topic refusal says what to call instead.

Take B is the controlled ending: at its 2nd tool call it made the exact same wrong first move as take A (topic on the guide tool) — and this time the refusal carried the recovery path, the model followed it, read the guides, and passed. Same model, same mistake; a bare refusal ended in surrender, a refusal-with-trail ended in a verified pass.

That is the loop working on both participants: the grader kept the agent honest, and the transcripts kept the server honest — each failure showed exactly which sentence of guidance to sharpen (now Part IV §9: "Refuse near-miss mechanisms loudly").

Reproduce it#

See README.md — one command per run, one command to grade. The world is deterministic and per-session, so your runs start exactly where these did. The live server includes both hardenings.