Summary
OpenAI reports higher scores for its newer models on a 20-language software-building benchmark and a COBOL code-migration evaluation. The results measure performance on the company’s tests, with production migrations still requiring expert review and testing.
OpenAI says its newer coding models have improved on two evaluations aimed at programming languages with less code and documentation available for models to learn from. One tests whether a model can build a SQLite-compatible database engine in 20 languages; the other tests whether it can migrate COBOL programs while preserving their behaviour. On the company’s evaluations, GPT-6 Astra achieved an average 87.4% test pass rate on the first, while GPT-6.1 Sol reached 86.6% accuracy on the second at low reasoning effort.
Building a database engine across 20 languages
MultiLangBench asks a model to build a SQLite-compatible engine from scratch in a specified language. The task involves organising a codebase, managing state, handling errors and adding features without breaking existing functionality.
To compare programs written in different languages, the evaluation gives each one an initial database and a SQL query. The program returns a modified database, which is checked against the expected contents using the reference SQLite implementation. OpenAI adapted tests from SQLite’s existing test suite and reports both test pass rate and API cost.
At maximum reasoning effort, GPT-6 Astra averaged an 87.4% pass rate across the 20 languages, compared with 26.8% for GPT-5.6 Sol. OpenAI reports improvement in every language tested. Astra scored 72.1% in COBOL, 85.8% in OCaml, 83.5% in Fortran and 89.4% in Visual Basic; it exceeded 83% in 19 languages. COBOL was the most difficult language in the evaluation.
GPT-6 Luna also improved on GPT-5.6 Sol in every language, with a 40.3% average pass rate at maximum reasoning effort. OpenAI estimates an average API cost of $0.77 per sample for Luna, compared with $3.48 for GPT-5.6 Sol—about 78% lower. The company says it is releasing the MultiLangBench setup, test suite and results by language on GitHub.
Measuring whether COBOL migrations preserve behaviour
The second evaluation tests migrations from COBOL into C++, Java, Kotlin, Python or Rust. Its hidden tests require byte-for-byte parity in observable outputs, and a migration counts as successful only when all tests pass.
That standard matters because code that runs successfully can still behave differently. OpenAI points to COBOL fixed-point arithmetic: excess decimal digits are discarded unless rounding is explicitly requested. A rewritten program that rounds those values could therefore produce different financial results.
On this evaluation, GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort, compared with 5.4% for GPT-5.6 Sol at the same setting. OpenAI says the newer model cost 39% less at that setting. At higher reasoning effort, GPT-6.1 Sol reached 97.9% accuracy, close to the evaluation’s maximum.
These figures describe performance on OpenAI’s two evaluations, rather than a general success rate for every enterprise codebase. The company says production migrations still need expert review and testing against the system’s business rules and dependencies. The results point to progress on particular tasks—building software in a broad range of languages and preserving tested behaviour during COBOL migration—while leaving those checks central to applying the work in practice.