🔍 Read the full analysis: FrontierCode: Can An AI Pull Request Meet A Project’s Standards? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis evaluated 23 AI models on FrontierCode, which checks whether code changes satisfy repository-specific review standards as well as tests. Claude Opus 5.5 led with a 52.2% score and cleared all blocking checks in 57.7% of attempts; the results show substantial gaps remain, while model costs varied sharply.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that evaluates whether a proposed code change would meet project standards, not simply whether it passes tests. Claude Opus 5.5 ranked first with a 52.2% score, but cleared every blocking check in 57.7% of attempts, leaving substantial room for human review.
The evaluation, added to Artificial Analysis’s benchmarks on October 12, 2026, used the benchmark’s 100-task Main set. The top nine models scored between 43.2% and 52.2%. GPT-6.1 Sol ranked second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Artificial Analysis reported that nine of the 100 tasks were not solved by any model on any attempt.
FrontierCode’s grading combines hidden tests, command checks, reverse tests and adaptive tests with limits on files touched and patch size, plus an LLM code review against maintainers’ criteria. A failure on any blocking check makes an attempt score zero, even if other tests pass. Artificial Analysis ran models through the same open-source agent harness, Stirrup, allowing up to 250 turns and three attempts per task. GPT-6 Luna (medium) served as the review judge.
Scores and costs did not move in lockstep. Artificial Analysis listed GPT-6 Luna (max) at 41.9% for $0.09 per task, MiMo-V2.6-Pro at 45.4% for $0.18, GPT-6.1 Sol at 49.1% for $0.81, and Opus 5.5 at 52.2% for $10.15. These are reported per-task costs under the evaluation, not a guarantee of what every user would pay in another workflow.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
FrontierCode addresses a practical weakness in coding benchmarks: a green test run does not by itself make a patch safe or useful to merge. A change can pass tests while adding unnecessary files, expanding its scope, or failing criteria a maintainer would apply in review. By making some of those checks blocking, the benchmark measures more than basic functional correctness.
The results do not establish how often AI-generated pull requests succeed across all software projects. They do show that, under this benchmark’s rules, the leading model failed to clear every blocking check in more than four in ten attempts. For teams considering autonomous code changes, that is a reason to retain review and safeguards rather than treating a strong leaderboard score as evidence that merge decisions can be automated reliably.
The cost results also matter for deployment choices. Moving from GPT-6.1 Sol’s 49.1% score to Opus 5.5’s 52.2% added 3.1 percentage points while the reported cost rose from $0.81 to $10.15 per task. The benchmark makes that trade-off visible, but does not say which model is the best choice for a particular codebase or workload.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Patches
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules intended to prevent agents from finding task solutions online. More than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost; according to the source material, they spent more than 40 hours on each task.
The full benchmark contains 150 private tasks, while Artificial Analysis evaluated the 100-task Main set. Keeping tasks private is intended to reduce the chance that a model has already encountered their solutions during training. The benchmark also penalizes oversized diffs, edits outside relevant files and unhelpful tests. A trajectory scanner flags attempts that look up solutions; those attempts receive zero under the benchmark rules.
In Cognition’s announcement, Celery maintainer Tomer Nosrati described the approach: “Where others grade like a CI, FrontierCode grades like a tech lead.” That is a characterization of the benchmark’s design, rather than an independent finding about how well it predicts real-world maintainer decisions.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, Celery maintainer, in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results are specific to FrontierCode’s tasks, scoring rules and evaluation setup. They do not establish how frequently models produce mergeable changes in everyday repositories, or how scores would change with different prompts, tools, reviewers or project standards. The available material does not provide enough detail to calculate uncertainty ranges for the scores.
Several models had no result as of October 13, 2026, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Comparisons with those systems are not possible from this leaderboard. It also remains unclear when the benchmark will next be updated or whether later results will use the same tasks and setup.
As an affiliate, we earn on qualifying purchases.
Pending Models and Future Runs
Readers can expect interest in results for the models missing from the October 13 leaderboard, but the source material gives no announced date for those evaluations. Any later comparisons will need to be checked against the benchmark version, task set and testing method used.
For now, the published results offer a snapshot rather than a final measure of AI coding performance. The next useful development would be additional model evaluations and evidence about how FrontierCode scores relate to maintainer decisions on live projects. Until then, its clearest finding is limited to the test: even the top-scoring model did not clear all blocking checks in every attempt.
software development code review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates whether AI-generated code changes satisfy a range of repository-specific checks, including tests, scope limits and an LLM review based on maintainers’ criteria.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led the Artificial Analysis results with a 52.2% FrontierCode score. Its reported blocking-check pass rate was 57.7%.
Does a 52.2% score mean that 52.2% of pull requests would be accepted?
Not necessarily. The score reflects FrontierCode’s task set and scoring rules. The published results do not establish an acceptance rate for pull requests across real-world projects.
Why can a patch score zero if its tests pass?
FrontierCode assigns zero when an attempt fails any blocking check. Those checks can include criteria beyond ordinary test results, such as command checks, reverse tests and restrictions on scope.
Which models were missing from the results?
As of October 13, 2026, the source report listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
