GPT-5.6 Sol vs Claude Fable 5 vs Gemini 3.1 Pro on SWE-Bench Pro

The Number That Changes the Shortlist

Claude Fable 5 is the model to beat on the reported SWE-Bench Pro row: 80.0%. GPT-5.6 Sol is at 64.6%. Gemini 3.1 Pro Preview is at 54.2%. Those three numbers come from OpenAI’s July 9, 2026 GPT-5.6 launch table, under the Coding section for SWE-Bench Pro (OpenAI).

That is a 15.4-point gap between Claude Fable 5 and GPT-5.6 Sol, and a 25.8-point gap between Claude Fable 5 and Gemini 3.1 Pro Preview. For a coding-agent builder, that is too large to hand-wave away.

But the right read is narrower than the headline. This is repository-level agentic coding. It is not Terminal-Bench. It is not Aider’s leaderboard. It is not a vibe-coded UI benchmark. If your product patches real repos, opens files, runs tests, edits across multiple modules, and needs to survive messy dependency graphs, SWE-Bench Pro is closer to your workload than many generic coding rows. If your product is mostly chat completions, code snippets, or CLI puzzle solving, this row is only one signal.

The Comparison Table

Here is the clean version, keeping the benchmark row separate from other coding-agent numbers.

Model	Vendor family	SWE-Bench Pro reported score	Source for score	API price checked
Claude Fable 5	Claude	80.0%	OpenAI GPT-5.6 table (OpenAI)	$10 input / $50 output per 1M tokens (Anthropic)
GPT-5.6 Sol	GPT	64.6%	OpenAI GPT-5.6 table (OpenAI)	$5 input / $30 output per 1M tokens (OpenAI)
Gemini 3.1 Pro Preview	Gemini	54.2%	OpenAI GPT-5.6 table (OpenAI); cross-checked against Google’s Gemini 3.5 Flash model card for Gemini 3.1 Pro on SWE-Bench Pro Public (Google DeepMind)	$2 input / $12 output per 1M tokens for prompts up to 200k tokens, standard tier (Google AI)

The Gemini number has the best cross-check. Google’s Gemini 3.5 Flash model card lists results “as of May, 2026” and reports Gemini 3.1 Pro at 54.2% on “SWE-Bench Pro (Public)” with a “Single attempt” setup (Google DeepMind). OpenAI’s July table uses “Gemini 3.1 Pro Preview” and the same 54.2% number. That does not prove identical harness settings in every detail, but it does make the Gemini score less suspicious than a single vendor-only citation.

Claude Fable 5’s 80.0% score, in this comparison, is sourced from OpenAI’s table rather than an Anthropic benchmark table. Anthropic’s own Fable page is still useful for release context and pricing: Fable 5 is positioned for “hardest knowledge and coding work,” uses the claude-fable-5 API name, and is priced at $10 per million input tokens and $50 per million output tokens (Anthropic). Anthropic also says Fable 5 and Mythos 5 were released on June 9, suspended on June 12 after a U.S. government directive, and restored globally on July 1 after export controls were lifted (Anthropic).

What SWE-Bench Pro Actually Measures

SWE-Bench Pro was built to push past the older SWE-Bench Verified ceiling. Scale describes it as a benchmark for long-horizon software engineering tasks in more realistic repositories, with 1,865 total tasks across 41 repositories: 731 public, 858 held-out, and 276 commercial (Scale). The public set uses open-source repositories with strong copyleft licenses; the private subset uses proprietary startup codebases.

The primary metric is resolve rate. A task counts as resolved when the submitted patch passes the new fail-to-pass tests and does not break pass-to-pass tests (Scale). That detail matters. A model does not get credit for a plausible patch, a pretty diff, or a helpful explanation. It has to change the repo in a way the benchmark environment accepts.

This is why SWE-Bench Pro belongs in the repository-agent bucket. A realistic agent run looks less like “write a function” and more like:

1. inspect issue and requirements
2. search repository conventions
3. edit one or more files
4. run targeted tests
5. debug failures
6. rerun regression tests
7. return a patch

That flow is exactly where context handling, tool use, error recovery, and test discipline start to matter. A model that shines in isolated code generation can still fall apart when the dependency tree is hostile or the failing test hides in a framework-specific edge case.

The Catch: The Benchmark Is Under Stress

The uncomfortable part: OpenAI published the GPT-5.6 table on July 9, 2026, one day after publishing a detailed SWE-Bench Pro audit on July 8. In that audit, OpenAI estimated that about 30% of SWE-Bench Pro tasks are broken and said it was retracting its earlier recommendation to adopt the benchmark (OpenAI).

The audit found two different estimates: an automated-plus-review pipeline flagged 200 broken tasks, or 27.4%, while a human annotation campaign identified 249, or 34.1% (OpenAI). The listed failure categories are exactly the ones benchmark users fear: overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts.

That does not make the table useless. It does mean you should not treat the second decimal place, or even a few percentage points, as engineering truth. A 15.4-point gap between Claude Fable 5 and GPT-5.6 Sol is still meaningful enough to investigate. A 1- or 2-point gap would be weak evidence on this benchmark in July 2026.

This is the practical stance: use SWE-Bench Pro as a reported repository-level signal, then rerun your own harness before buying architecture around it.

Don’t Mix It With Terminal-Bench or Aider

OpenAI’s same GPT-5.6 table also reports Terminal-Bench 2.1, where GPT-5.6 Sol is at 88.8%, GPT-5.6 Sol Ultra is at 91.9%, Claude Fable 5 is at 83.1%, and Gemini 3.1 Pro Preview is at 70.7% (OpenAI). That tells a different story because it is a different task shape. Terminal-Bench emphasizes terminal workflows. SWE-Bench Pro emphasizes repo patch resolution.

Mixing those scores into one “best coding model” number is how teams fool themselves. Aider-style benchmarks add another harness layer: prompt format, edit mode, retry policy, and model-specific adapters can move results. They are useful, but they are not interchangeable with a repository benchmark using its own task construction and grading rules.

For coding-agent builders, separate your eval sheet into lanes:

repo_patch_resolution:
  benchmark: SWE-Bench Pro
  metric: resolve_rate
terminal_workflows:
  benchmark: Terminal-Bench
  metric: task_success
interactive_editing:
  benchmark: Aider-style runs
  metric: accepted_patch_or_tests_passed
internal_repos:
  benchmark: private eval set
  metric: merged_patch_after_review

Then compare models inside each lane. Do not average the lanes unless you are explicit about weights. A migration agent, a test-writing bot, and a terminal ops agent have different failure costs.

Bottom Line

If SWE-Bench Pro is the lane, Claude Fable 5 is the reported leader among these three models: 80.0% versus 64.6% for GPT-5.6 Sol and 54.2% for Gemini 3.1 Pro Preview. GPT-5.6 Sol is cheaper than Fable 5 at list API pricing, $5/$30 per million tokens versus $10/$50, so it may still win on cost-adjusted throughput if your agent can tolerate the lower reported resolve rate. Gemini 3.1 Pro Preview is cheaper again at $2/$12 up to 200k-token prompts, but the reported SWE-Bench Pro gap is large enough that you should validate it on your own repositories before calling it the economical choice.

The sharpest takeaway is not “Claude wins everything.” It is this: for repo-level agents, use repo-level benchmarks, cite the harness, and keep benchmark health in view. SWE-Bench Pro currently says Fable 5 leads this comparison, but the benchmark itself has enough documented quality problems that serious teams should treat it as a shortlist filter, not a deployment decision.

Readers who want to try these models hands-on can call Claude and other models through onehop with an OpenAI-compatible API: change one base_url, compare runs, and keep your harness stable (call Claude and other models on onehop). New accounts get $10 free credit with no card required, and pricing is cheaper than first-party access (sign up for $10 free credit).