The proof: same destination, different routes

This page presents a recorded experiment, run twice against two versions of the wire, for developers evaluating Pilotage's central claim.

An agent that has never seen a system can complete real work in it using only what the server provides through Pilotage, and different agents, taking different paths, can converge on the same verified outcome.

The stronger lesson came from the failure:

An agent's self-report is not evidence. The checkable end state is.

One unfamiliar system, one neutral job#

Three models from three vendors received the same work order against the same live Harborview server.

The work order asked each agent to:

1. open the building for the day;
2. turn on the lobby lights;
3. unlock the loading-dock door;
4. install an after-hours dock-door alarm;
5. demonstrate that the alarm would notify Facilities.

The building accepts programs in buildingflow/1. That language was created for Harborview and exists nowhere else. None of the models had prior instructions for it.

Each attempt ran:

• in a fresh isolated session;
• through the same published runner of under 300 lines;
• under the same recorded system prompt;
• against the same deterministic building;
• with the same neutral work order.

After the agent stopped, a separate deterministic grader inspected the building's real state and fired the dock-door event. No model graded itself.

Result: convergence#

First recording, 2026-07-24 (pre-1.1 wire):

Second recording, 2026-07-26 (v1.1 wire, after the trace-door rename):

The successful agents did not follow one hidden script. Their tool-call counts and demonstration strategies differed. They converged on the same fact-checked destination, on both wires, and none of them was told the v1.1 names in advance: the handshake taught them.

That is the intended Pilotage property. The server standardizes the loop and evidence. It does not prescribe the agent's path.

The failure is the strongest result#

On its first take, gpt-5.4-mini produced a confident final report: “Done.”

The building disagreed.

The agent had used Harborview's promotion feature. Promotion stores a program so it can be called later through program_ref. It does not install the program's trigger.

The grader inspected the actual building state and found:

deployed: []

The after-hours alarm would not fire when the dock door opened. The report card was therefore:

FAIL

This is not an edge detail. It is the difference between storing an automation and deploying it.

The experiment did not hide the failure or reinterpret the agent's report. The factual grader overruled the fluent answer.

That is exactly why Pilotage carries plans, outcomes, traces, and live state through a checkable loop. “The tool call returned” and “the agent said it was done” are not mission verification.

It happened again, differently. In the 2026-07-26 recording, the same model produced another confident “Done.” against a server whose teaching surface had already been hardened. This time it did not promote the program: it authored the alarm logic correctly and then ran it once in immediate mode, describing that single run as an installation. The grader again found deployed: [], and again the unchanged second take passed. Two recordings, two different mistakes, one constant: the agent's self-report is not evidence, and a deterministic judge that reads the building catches what fluent prose hides.

The full five-step timeline#

The sequence matters. Compressing it into “failed, then fixed” would erase what the experiment showed.

1. Take 1: FAIL#

gpt-5.4-mini promoted the alarm instead of deploying it. It reported success. The deterministic grader found no installed automation.

This exposed a real ambiguity between two Harborview operations:

• promotion: store a program for on-demand calls;
• deployment: install a trigger for future events.

2. Take 2, unchanged: PASS#

The same model ran again against the unchanged server and passed.

This matters because it prevents a false diagnosis. The first failure did not prove that the server made the task impossible. A different model trajectory could already reach the correct result.

It also did not excuse the first failure. The grader's FAIL remained valid.

3. The server-hardening cycle began#

The failure was treated as a test of the server's teaching surface.

The relevant Harborview guidance was made sharper around the boundary between immediate execution, promotion, and deployment. The hardened server, as recorded in these transcripts, states that promote_flow stores an on-demand program and does not install a trigger. A triggered flow sent in mode: "immediate" is refused with the closed diagnostic code bad_trigger and a hint that names mode: "deploy" as the repair.

The hardening cycle ultimately required two changes, separated by another recorded run.

4. Post-fix take A: FAIL, differently#

A fresh run failed before it reached the guides.

This was not the original promote-versus-deploy error. It exposed a second weakness: correct guidance is useless when a cold-start agent does not reach it.

That run motivated the second hardening. Harborview's entry points and refusal messages were sharpened so that the trail back to the manifest, guides, and correct operation was explicit.

The failure remained in the record because it identified a different missing link in the teaching path.

5. Post-fix take B: PASS#

A final fresh run made the same initial mistake seen earlier.

This time, the server's sharpened refusal and guidance led the agent back into the loop. It repaired the operation, deployed the alarm, and passed the deterministic grader.

The improvement was not that the model became infallible. The improvement was that the server became better at turning a plausible mistake into a recoverable, structured path.

What the grader established#

The grader judged facts after the agent finished. It did not score writing quality or confidence.

The checks included the real end state required by the work order, including whether an installed automation fired when the dock-door event was driven after hours.

This distinction is the core of the proof:

Agent report  ≠  verified outcome

The passing report cards show that different agents can converge.

The failing report card shows that the judge is capable of rejecting a polished answer.

The recovery runs show that an agent failure can improve the server's guides, descriptions, and refusals. In that sense, the proof run acts as continuous integration for the teaching surface.

What this does, and does not, prove#

The experiment supports a bounded claim:

• an agent can learn an unseen, finite Language from one cooperative server;
• different agents can take different paths to the same checked result;
• structured refusals can help an agent recover;
• a deterministic, non-AI grader can catch a false completion report.

It does not show that:

• every agent will succeed;
• every task is expressible;
• Pilotage guarantees the user's goal;
• a malicious server becomes safe;
• a server's trace is independent proof of its own honesty.

Pilotage makes convergence checkable. The path remains the agent's own.

Inspect the evidence#

The proof is intended to be auditable, not taken on trust.

• Results and failure analysis
• Rendered transcripts
• Neutral work order
• Published runner
• Deterministic grader
• Results and failure analysis, second recording (v1.1)
• Reproduction and fairness notes
• Live Harborview server
• Harborview repository

The transcripts contain every tool call, every server response, each final report, and each grader card.

A note on names. The first recording was made on 2026-07-24, when the wire still used the pre-1.1 names (loop.explain, explain_run); those transcripts are frozen evidence and keep the names they were recorded with. On 2026-07-26 the same experiment was repeated, unchanged, against the migrated v1.1 server (loop.trace, trace_run, execute store). Both recordings are published side by side.

The conclusion#

Three models approached one unfamiliar system differently and reached the same verified destination, twice: once on the original wire, and once, cold, on the renamed v1.1 surface that only the handshake taught them.

One model also produced a confident false completion report, in each recording, for a different reason each time. The grader rejected both. The first failure improved the server's teaching path; the second showed the judge still catching fluent prose against an already-hardened server. In both recordings, an unchanged fresh run recovered.

The passes show convergence.

The failure shows honesty.

The feedback loop shows why the server, not every individual agent integration, should own the knowledge required to use its complex tools.

Next: complete the same job step by step or read the Reference.