An 800,000-line port that compiled on every merge
On September 16, 2026, Stephen Toub published Migrating the GitHub Copilot runtime to Rust, using Copilot on the GitHub Blog. GitHub rewrote the runtime behind the Copilot command-line tool, app and software development kit (SDK) from TypeScript on Node.js into more than 800,000 lines of production Rust. AI agents wrote most of the code. The work landed as 128 pull requests over a porting window of roughly fourteen and a half weeks, shipped incrementally rather than in one cutover, and was done primarily by one developer.
By September 14 the team had traced dozens of known port regressions, all fixed, most of them correctness bugs. Toub takes on the Rust meme that "if the code compiles, it's correct", and answers it with the regression list: every regression on it was merged to main, so every one of them compiled. All existing end-to-end tests also ran against the new Rust code at every step, and a pull request that failed a required test did not land.
So the regressions passed two gates, the compiler and the test suite, and some still reached a stable release. Each verification method answers one question, and accepting a migration needs answers to several.
Five methods, five questions
Teams verifying a migration stack several methods. Four of them compare the new system with the old one. The fifth, decision compliance, checks new code against the engineering decisions the team made for the target system, such as which layer may touch storage. Each proves one thing and misses another:
| Method | Question it answers | What it cannot see |
|---|---|---|
| Compilation and static analysis | Is the new program internally coherent? | Whether it preserves the old contract, or whether anything is missing |
| Existing test suite | Does the new code pass the behaviors someone wrote a test for? | Behavior nobody tested, and tests that encode the wrong behavior |
| Shadow traffic with response comparison | Does the new code return the same response to real requests? | Side effects the shadow suppresses, and state that changed between the two runs |
| Output parity tests | Do observable outputs match the reference exactly on these inputs? | Whether the reference behavior is the behavior you want to keep |
| Decision compliance | Does the new code respect the decisions that govern the target system? | Whether it behaves the same as the old system |
Toub makes the first row's limit explicit. A compiler, he writes, cannot check whether you wrote the whole program, preserved the old contract or called things in the right order. GitHub's regressions fall mostly into three families: the new code implemented a different behavioral contract, state or lifetime behavior changed, or part of the migration was omitted or lost in a rebase. He also lists "tests that confidently validated the wrong behavior", which is the second row's limit. Toub also writes that all but one of the regressions involving missing features came from insufficient end-to-end tests.
The fourth row is the method OpenAI chose for its COBOL (Common Business-Oriented Language) migration evaluation, published on October 19, which requires migrated programs to match observable outputs byte for byte on hidden tests. Our analysis of OpenAI's COBOL benchmark covers what parity leaves out. The last row differs from the other four: it is the only one that does not use the old system as the reference.
What a matching output hides
The regressions in GitHub's post cluster in semantics that a single input and a single output rarely expose:
- Error semantics. The TypeScript
event.error || "Unknown error"became Rust's.unwrap_or("Unknown error"). JavaScript's||replaces an empty string;unwrap_orkeeps it. An empty error message that used to become "Unknown error" now stayed empty. - Cancellation. When a session was aborted, the port still recorded the abort but dropped the step that interrupted work in progress, so a request waiting on a tool could not be stopped. Separately, a turn-cap check (the limit on how many model requests one task may make) set the session's abort state without cancelling the loop that sends model requests, so one more request went out.
- Ordering and lifetime. The largest cluster involved lifecycle, disposal, ownership, ordering or races. Toub's summary of the compiler's blind spot: "Events can move safely between threads and still arrive in the wrong order."
- Library behavior. At the time, the Rust library for the Model Context Protocol (MCP) replied to malformed input with an error, where the TypeScript library had stayed silent. One server answered each error with more malformed output. The Rust side replied again, the two kept answering each other, and startup hung.
Retries belong on the same list, though GitHub reports no retry regression. In our analysis, a port that retries twice where the old code retried once returns the same response while the dependency is healthy, and passes a response comparison.
Shadow testing narrows these gaps without closing them. Grab, the Southeast Asian ride-hailing company, described in a 2016 post moving its passenger application programming interfaces (APIs) from Rails to Go by shadowing production traffic and comparing responses. Our article on Node.js to Go migrations also covers Grab's Rails-to-Go rollout as a related case.
Two details in Grab's account mark the edge of the method. Shadowing a write would have written the data twice, so Grab wrapped its data access objects in mock code that generated the expected database row and compared it with the row Rails had written. And because the shadow ran after Rails had processed the request, booking state could change in between and produce a mismatch. Grab found that rare enough to debug by hand.
A shadow compares results. It does not observe what the new code does to the database, in what order, or how often it would retry. Mismatches caused by timing look like noise, and ordering bugs live in timing.
GitHub reached the opposite decision about shadowing for its most coupled component. Session orchestration owns mutable state and drives callbacks in both directions, so running both versions side by side would have meant maintaining two copies of the component that holds the conversation's state, in two languages, and keeping them in step while hundreds of pull requests a week changed the code. The team gained confidence through end-to-end tests and incremental rollout with real consumer usage instead. The components where you would most want a side-by-side comparison are often the ones where you cannot run one.
The boundary no output reveals
None of the first four methods asks whether the new system is built the way the team decided.
Take a migrated payments service that reads the legacy database directly instead of going through the storage adapter the target architecture requires. Every response matches and the shadow reports zero mismatches, because the data is the same data. The migration has still crossed a boundary the team approved, and it has created a second dependent on the system it is meant to retire. A strangler-fig migration moves traffic to the new system one boundary at a time behind an adapter, and our guide to strangler-fig migrations shows how a bypass like this fails silently. Our guide to AI-assisted software migrations shows how agents drift back toward the dominant legacy pattern.
GitHub's own process checked for this kind of property, and it checked it outside the tests. The review prompt Toub ran on each port told review agents: "If new TypeScript that's not just a super thin shim is being added, that's a red flag." A shim here is a thin wrapper that only forwards calls to the Rust code. In effect, that is an architectural rule about the target: Rust owns the logic. No end-to-end test fails on a well-behaved TypeScript function. Agents did the old-versus-new comparisons; human reviewers, he writes, concentrated on architecture, API contracts and risk.
A second incident shows why the tests need protecting too. A port deleted a function exposed to the SDK, and the repository's schema compatibility check failed. The agent's fix was to apply the repository's schema-break-ok label, the escape hatch that makes the check pass. Toub caught it in review and had the method restored. His lesson concerns the oracle, the reference that defines correct behavior, here the tests and compatibility checks: "Protect the oracle from the agent." If the agent can rewrite them during the port, in his words, "you lose your oracle".
Acceptance needs both kinds of evidence
A migrated endpoint, component or service is ready to accept when two questions have answers:
- Behavioral evidence: it does what the old system did, for the behavior the organization has decided to keep. Parity tests, shadow comparison and end-to-end tests supply this, each with its blind spots.
- Decision compliance: it respects the decisions that govern the target, such as which dependencies are forbidden and which transitional code must not grow (in GitHub's port, no new logic in the temporary TypeScript shims).
A pass on one is weak evidence about the other. Matching outputs cannot show that a boundary held, and a clean decision check cannot show that a cancellation path survived. The first question depends on an earlier one: which legacy behaviors are approved to carry forward. Our guide to deciding what must survive a legacy rewrite covers that adjudication.
The second question is what engineering decision governance addresses: which decisions govern a given change, and with what evidence. Treat the decisions like the tests. Write them down before the port starts, keep them out of the agent's reach, and check them on every change, not only in a final review.
What engineering leaders should do
- Build the oracle first, then freeze it. GitHub's missing-feature regressions came from thin end-to-end coverage. Expand the tests before porting, and require separate approval for any change to tests, snapshots, compatibility baselines or waiver labels.
- List each component's semantic risks. Write down its error, cancellation, ordering and retry behavior, and test those paths directly.
- Write boundary decisions as checks. State which code may call which, and which transitional layer may not grow, and run those checks on every pull request alongside the tests.
- Record both answers at acceptance. The sign-off for each migrated unit should cite its behavioral evidence and its decision results, with a named owner for each.
Where Mneme fits
Mneme, an open-source decision and control layer for agentic software development, covers part of the decision-compliance half. It enforces decisions that carry an active typed rule: a rule a machine can check, such as an exact piece of text that must not appear in code, optionally limited to certain paths. A rule that forbids the legacy database client's exact import path as literal text in the target code, with the approved adapter's path excluded, returns a FAIL when new code adds that import. A decision without an active typed rule is not enforced.
The checks run in continuous integration (CI) and in three coding agents: Claude Code, Codex CLI (OpenAI's command-line coding agent) and Kiro CLI 3.0 (the command-line version of Amazon's Kiro agent). Each agent's native write tools are checked before the write lands:
- Claude Code: Edit, Write and MultiEdit.
- Codex CLI: apply_patch.
- Kiro CLI 3.0: its native write tools.
Shell writes reach disk first; Claude Code and Codex CLI check them when the agent finishes its response, and other agents rely on CI. Each result is PASS, WARN or FAIL, and a violation names the decision it broke. In CI, strict mode fails the job on a violation, and warn mode reports without failing.
Decisions live in the Decision Index, Mneme's persisted, versioned record of a repository's decisions, which keeps each decision's versions, lifecycle status and source. It does not record who approved a decision.
Mneme does not test behavioral equivalence, run shadow traffic or compare outputs. Linking decision results to migration evidence, or to differential tests, is a research direction, not a shipped capability.