What OpenAI announced
On October 19, 2026, OpenAI published Advancing coding in low-resource programming languages. Low-resource languages are those with far less public code and documentation for models to learn from than Python or JavaScript, such as COBOL (Common Business-Oriented Language), Fortran and Visual Basic. The note introduces two evaluations and reports results for several OpenAI models, including two different ones, GPT-6.1 Sol and GPT-6 Astra.
OpenAI reports each score at a stated reasoning effort: a setting that controls how much computation the model spends working through a task before it answers.
One evaluation is a COBOL migration test. A model migrates COBOL programs to C++, Java, Kotlin, Python or Rust, and hidden tests require byte-for-byte parity on observable outputs. A migration counts as a success only if every test passes. At low reasoning effort, GPT-6.1 Sol scores 86.6%, against 5.4% for the older GPT-5.6 Sol at the same low setting. At higher reasoning effort, GPT-6.1 Sol reaches 97.9%. OpenAI describes that as close to saturating the evaluation, meaning the test has little headroom left to show further gains.
The other is MultiLangBench, which asks a model to build a SQLite-compatible database engine in a given language. At maximum reasoning effort, GPT-6 Astra averages an 87.4% test pass rate across 20 languages, up from 26.8% for GPT-5.6 Sol. COBOL is still the hardest language, at 72.1%. OpenAI released the setup and test suite on GitHub under the Apache License 2.0.
These are the vendor's own benchmarks, run on the vendor's own models. The jump from 5.4% to 86.6% at the same low setting is still a real signal.
Why exact parity is the right test
OpenAI picked a strict bar, and it is the correct one for this kind of code. Its own example is COBOL's fixed-point arithmetic, which drops extra decimal digits unless rounding is explicitly requested. A Java rewrite that rounds instead will compile, run without errors and produce different balances. In a banking system, a one-cent difference repeated across millions of transactions is a reconciliation incident.
A test that compares every output byte catches that class of error. Behavioral parity is how a team proves that a rewrite still does what the old system did.
What the evaluation does not ask
A parity test asks whether the new program produces the same observable outputs as the old one on these inputs. Four different claims tend to get merged with it:
- Benchmark success: the migrated program passed the hidden tests. That is what 97.9% measures.
- Semantic understanding: the model knows why the program behaves as it does. Passing tests is evidence of this, not proof.
- Architectural authority: the behavior being preserved is behavior the organization still wants. A parity test cannot say this, because it uses the legacy system as the reference for correct output.
- Production acceptance: the system is fit to run. OpenAI is explicit here: production use "still requires expert review and testing against the system's business rules and dependencies."
The third claim is the easiest to assume without noticing. If the old code truncates where the business meant to round, a perfect migration preserves that bug exactly.
Observed is not authoritative
Take a legacy banking application being moved off the mainframe. Analysis tools and models that read the old code will surface many facts about it, and they are not the same kind of fact. (Mneme, named in the table, is the open-source tool this site describes; its scope is at the end.)
| Recovered information | What it is | Treatment |
|---|---|---|
| COBOL truncates a decimal value | Observed behavior | Test parity, and confirm it is the intended financial rule |
| A legacy service reads the database directly | Architectural implementation | Decide whether it was deliberate or incidental |
| The target architecture requires a storage adapter | Reviewed engineering decision | Enforce it on new code with a code-level rule run by the coding agent and in CI, for example a Mneme typed rule |
| Data must stay in an approved jurisdiction | Deployment requirement | Check with an authorized deployment control, not Mneme |
| A workaround exists for a retired interface | Historical implementation | Review whether to remove it |
Parity testing covers the first row well. It says nothing useful about the other four. A direct database read produces the same outputs whether or not the team meant it. The storage adapter has no legacy output to compare against, because it is a decision about the target system. Data residency is not visible in program output at all. And if a team derives its parity tests from the legacy system's behavior, the workaround passes every one of them, because that system still contains it.
Observed does not automatically mean authoritative. Someone has to decide which of these facts survive the rewrite, and that decision needs an owner. AI legacy modernization: what must survive the rewrite works through that process in full.
Extraction is not the scarce part
Vendor tools already recover these facts. AWS (Amazon Web Services) documents AWS Transform as orchestrating analysis of mainframe codebases, generating documentation, extracting business logic and decomposing monoliths. IBM describes watsonx Code Assistant for Z as mapping application relationships and dependencies, refactoring COBOL into modular business services, and generating unit tests that compare the semantic equivalence of new Java services with the original COBOL.
So extraction and equivalence testing are becoming standard parts of a modernization toolchain. The part that stays hard is adjudication: which recovered rules are approved to carry forward, which are bugs, and which belong to an architecture the organization is leaving. A small September 2026 preliminary study by Karan, Dhar, Soliman and Vaidhyanathan tested language models on extracting architectural decisions from commits; a manual review of one model's outputs found they often described the change but missed the rationale. See what the research shows about reverse-engineering decisions.
What to do before the migration starts
- Use parity tests as the behavior gate. Byte-for-byte output comparison is the right acceptance test for preserved behavior.
- Classify every recovered fact. Mark it observed, inferred or approved, and keep the evidence with it.
- Give approval an owner. A changed rounding rule is a business decision. A removed direct database read is an architecture decision. Name who signs off on each kind.
- Turn approved engineering decisions into constraints. Record target-state and transitional decisions as migration ADRs (architecture decision records), and enforce them on the code agents write, so the migration does not drift back toward the legacy design.
- Route deployment requirements to deployment controls. Residency and access rules belong in the platform that runs the system.
This is the question engineering decision governance addresses: which decisions govern a given change, and with what evidence. Behavioral parity supplies one kind of evidence. It does not supply the decisions.
Where Mneme fits
Mneme is an open-source decision and control layer for agentic software development. Today it covers one step above. Once a team has recorded an engineering decision, Mneme enforces it on code changes. The mechanism is a typed rule, a machine-checkable form of the decision such as an exact import statement that must not appear in code, optionally limited to certain paths. Mneme checks typed rules deterministically in supported coding agents, such as Claude Code, Codex CLI and Kiro, and in continuous integration (CI), and returns a PASS, WARN or FAIL verdict. A decision without an active typed rule is not enforced.
Its Decision Index, the persisted and versioned record of a repository's decisions, keeps each decision's version history, its source and its lifecycle status (active, superseded or deprecated).
Deciding what is approved stays with your team. Mneme does not record who approved a decision, and it does not extract business logic, parse COBOL, enforce financial rules or check data residency. Tagging each decision as observed, inferred or approved, with its provenance, is design direction, not a shipped feature. For migration-time patterns, see AI-assisted software migrations.