OpenAI publishes benchmarks showing big gains on COBOL and Fortran
A new public evaluation suite shows GPT-6 Astra scoring 72% on COBOL and 83% on Fortran, up from under 27%. A separate COBOL-migration test hits 97.9% accuracy.
OpenAI published two new coding evaluations on 19 October aimed at languages it calls “low-resource”: COBOL, Fortran, Visual Basic and others with far less training data available than Python or JavaScript. The first, MultiLangBench, asks a model to build a SQLite-compatible database engine from scratch in each of 20 languages. GPT-6 Astra averages 87.4% across all 20, against 26.8% for the prior generation. Per language, COBOL went from 10.9% to 72.1%, Fortran from 21.4% to 83.5%, and Visual Basic from 27.5% to 89.4%.
The second evaluation tests migrating an existing COBOL program to another language, checked by comparing outputs byte-for-byte rather than judging code style. GPT-6.1 Sol scores 86.6% at low reasoning effort, up from 5.4% for GPT-5.6 Sol, at 39% lower cost, and reaches 97.9% at higher reasoning effort. OpenAI is releasing the MultiLangBench evaluation setup and test suite on GitHub so developers can rerun it against other models.
Why it matters: COBOL and Fortran still run large parts of banking and scientific computing, and modernisation projects have historically stalled because models trained mostly on web code handle them badly. A public, rerunnable benchmark gives anyone evaluating a migration tool a way to check vendor claims instead of taking them on faith.
The caveat: these are OpenAI’s own evaluations, scored by OpenAI, on OpenAI’s models. There’s no independent reproduction yet.