Search gap bench · 2026-10-03
Search gap bench
In ten seconds
With any search setup, Claude Code and Codex passed 71–95% of briefs about recent events; with no search, 0%.
Claude Code claude-opus-5-5
21 runs per setup
Parallel passed most often: 20 of 21 runs. Every other setup's range overlaps the leader's, so the test cannot separate them.
- 1Parallelleader64k tokens per pass95%77–99%20 of 21
- 2Firecrawl116k tokens per pass90%71–97%19 of 21
- 3Brave93k tokens per pass81%60–92%17 of 21
- 4Built-in61k tokens per pass76%55–89%16 of 21
Rerun
In the first runs Claude Code's built-in search could not open pages. It was fixed and rerun on 2026-10-06; this row is the rerun. The original row stays in the report. - 5Perplexity74k tokens per pass76%55–89%16 of 21
- 6Exa89k tokens per pass71%50–86%15 of 21
- 7Tavily83k tokens per pass71%50–86%15 of 21
Codex gpt-6.1-sol
18 runs per setup
Brave passed most often: 17 of 18 runs. Every other setup's range overlaps the leader's, so the test cannot separate them.
- 1Braveleader111k tokens per pass94%74–99%17 of 18
- 2Firecrawl166k tokens per pass89%67–97%16 of 18
- 3Tavily129k tokens per pass89%67–97%16 of 18
- 4Built-in117k tokens per pass78%55–91%14 of 18
- 5Exa146k tokens per pass78%55–91%14 of 18
- 6Parallel157k tokens per pass72%49–88%13 of 18
- 7Perplexity117k tokens per pass72%49–88%13 of 18
What the numbers mean
7 research briefs about events after the model's training data, each run 3 times: 21 runs per setup. A brief passes if it reaches the right decision, gets at least 70% of the key facts (important ones count more), and no more than 25% of its claims lack support in the sources it cites.
For scale: with no search the agent passed 0%; handed an excerpt containing the answer, it passed 95%. The full report gives each setup's gap closed: how far it got from 0% toward 95%. Tokens per pass: all tokens used, failed runs included, divided by passing runs.
Source: Search gap bench; ranges are 95% Wilson intervals, as published.
The full report
GAP battery results, 2026-10-03: does search change the outcome of the job?
Update (2026-10-06): the Claude Code native setup was rerun with page fetching working. In the original battery the setup exposed WebSearch and WebFetch but pre-approved only WebSearch, and headless Claude Code refused all 50 WebFetch calls across its 21 runs, so that row measured search without page fetching. The configuration is fixed and pinned by a test. The setup's 21 runs were rerun on 2026-10-06 on the same 7 tasks, repetitions, model and calibration; no call was refused. The Claude Code native row and every figure derived from it below use the rerun: 16/21 passed (76%), gap closure 0.80 (0.53–1.00), 61k tokens per success. The original row recorded 13/21 (62%), gap closure 0.65 (0.37–0.91) and 66k. The other setups are unchanged. The rerun's verdicts come from the claude-code primary judge, which decides every verdict in this battery. On 2026-10-07 the codex judge scored the same 21 stored payloads: it agreed on every verdict and on 94% of labels (kappa 0.70); see judge agreement. The rerun's agents used about 1.0M tokens. The codex native setup was unaffected. Details: agent search behavior.
The first full Search Gap Bench (GAP) battery ran on 2026-10-03, operator-approved: brief tasks on two agents, nine setups each, three repetitions.
- GAP grades the job, not the retrieval.
- A task counts only after calibration shows its knowledge gap: the no-search setup (floor) passes at most 1 of 5, and the setup given the answer excerpt (ceiling) passes at least 4 of 5.
- Each search setup is scored by gap closure: how much of the floor-to-ceiling distance it covers.
- Tokens are reported beside every outcome.
Headline. On these tasks, search decides the outcome.
- The floor passed 0% on both agents, and every search setup passed 71–95%.
- Which provider is best depends on the agent: Parallel led on claude-code and Brave on codex.
- Pass rate did not predict cost: providers with similar pass rates differed almost twofold in tokens per success.
These are the recorded results after the Codex reruns. Claude-code provider tool availability remains unverified in the published evidence: no transcript audit for those runs is documented, and they lack availability records. Its provider rankings are provisional pending that audit. In the first codex pass, 31 of 108 provider runs ran without their search tool. They were re-run on 2026-10-04 under the GAPMCP-01 fix, every re-run run loaded its tool, and the codex tables below include them (see "Execution changes").
Setup
| Agents | claude-code on claude-opus-5-5, codex on gpt-6.1-sol |
| Setups | floor (no search), ceiling (answer excerpt in the prompt), native, Brave, Tavily, Exa, Parallel, Firecrawl, Perplexity |
| Tasks | 10 brief tasks from GAP-09: 6 decision briefs and 4 research briefs. The code corpus (GAP-07) merged during the run and gets its own battery. |
| Repetitions | 5 per reference setup in calibration; 3 per setup in the battery |
| Grading | Decision correctness, weighted key-fact recall (≥ 0.7) and unsupported-claim rate (≤ 0.25), judged by two blinded judges (claude-code primary, codex secondary) against captured primary sources |
| Spend | about 50M tokens in total: battery agents 21.1M; battery judges 15.3M; calibration agents 4.2M; calibration judges 9.2M |
Calibration
Final admissions under GAPRULE-01. Each run shows floor passes and ceiling passes out of 5.
| Task | Family | claude-code | codex |
|---|---|---|---|
| billing-reporting | research | Floor: 0, Ceiling: 5 — admitted | Floor: 0, Ceiling: 5 — admitted |
| classroom-term | decision | Floor: 0, Ceiling: 5 — admitted | Floor: 0, Ceiling: 5 — admitted |
| models-production | decision | Floor: 0, Ceiling: 5 — admitted | Floor: 0, Ceiling: 5 — admitted |
| npm-token-scope | research | Floor: 0, Ceiling: 5 — admitted | Floor: 0, Ceiling: 5 — admitted |
| org-migration | decision | Floor: 0, Ceiling: 5 — admitted | Floor: 0, Ceiling: 4 — admitted |
| python-metadata | research | Floor: 0, Ceiling: 5 — admitted | Floor: 0, Ceiling: 5 — admitted |
| tarfile-upload | decision | Floor: 0, Ceiling: 5 — admitted | Floor: 0, Ceiling: 0 — rejected |
| cli-trust | decision | Floor: 0, Ceiling: 2 — rejected | Floor: 0, Ceiling: 2 — rejected |
| spark-editing | decision | Floor: 0, Ceiling: 1 — rejected | Floor: 0, Ceiling: 0 — rejected |
| python-security-march | research | Floor: 0, Ceiling: 0 — rejected | Floor: 0, Ceiling: 1 — rejected |
Every floor scored 0, so every task has a real knowledge gap. All rejections are ceiling failures, which point at task or rubric defects:
- spark-editing: the heaviest key fact is not what the prompt asks about.
- python-security-march: the excerpt omitted the date that its top fact requires. The excerpt fix is pending for the next battery; GAPRULE-01 changed the unsupported-claim limit, not the excerpt.
- cli-trust: with three claims, a single unsupported claim fails the brief.
- tarfile-upload on codex: the codex briefs miss key facts.
Gap closure
claude-code (7 tasks, 21 runs per setup)
| Setup | Gap closure (95% CI) | Pass rate (Wilson 95% CI) | Tokens per success | vs floor (exact McNemar) |
|---|---|---|---|---|
| Parallel | 1.00 (0.86–1.17) | 95% (77–99%) | 64k | p < 0.0001 |
| Firecrawl | 0.95 (0.83–1.00) | 90% (71–97%) | 116k | p < 0.0001 |
| Brave | 0.85 (0.62–1.00) | 81% (60–92%) | 93k | p < 0.0001 |
| Perplexity | 0.80 (0.50–1.00) | 76% (55–89%) | 74k | p < 0.0001 |
| native | 0.80 (0.53–1.00) | 76% (55–89%) | 61k | p < 0.0001 |
| Exa | 0.75 (0.47–0.95) | 71% (50–86%) | 89k | p < 0.0001 |
| Tavily | 0.75 (0.43–1.00) | 71% (50–86%) | 83k | p < 0.0001 |
| ceiling (reference) | 1.00 | 95% | 4k | |
| floor (reference) | 0.00 | 0% | — |
By family:
- Research briefs: every search setup passed 100%, native included.
- Decision briefs: they separate the providers. Parallel 92% and Firecrawl 83% lead; Brave scored 67%, Perplexity and native 58% each, and Exa and Tavily 50% each.
codex (6 tasks, 18 runs per setup)
| Setup | Gap closure (95% CI) | Pass rate (Wilson 95% CI) | Tokens per success | vs floor (exact McNemar) |
|---|---|---|---|---|
| Brave | 1.06 (0.88–1.29) | 94% (74–99%) | 111k | p < 0.0001 |
| Firecrawl | 1.00 (1.00–1.00) | 89% (67–97%) | 166k | p < 0.0001 |
| Tavily | 1.00 (0.71–1.29) | 89% (67–97%) | 129k | p < 0.0001 |
| Exa | 0.88 (0.47–1.29) | 78% (55–91%) | 146k | p = 0.0001 |
| native | 0.88 (0.47–1.20) | 78% (55–91%) | 117k | p = 0.0001 |
| Parallel | 0.81 (0.41–1.14) | 72% (49–88%) | 157k | p = 0.0002 |
| Perplexity | 0.81 (0.47–1.12) | 72% (49–88%) | 117k | p = 0.0002 |
| ceiling (reference) | 1.00 | 89% (67–97%) | 12k | |
| floor (reference) | 0.00 | 0% (0–18%) | — |
No search setup beat native significantly; the largest gain was Brave at +16.7 points (p = 0.25).
By family:
- Research briefs: every setup passed 100%, except Exa at 89%.
- Decision briefs: they separate the providers. Brave scored 89%; Firecrawl and Tavily 78% each; Exa 67%; native 56%; Parallel and Perplexity 44% each.
The first pass's estimates that excluded the 31 tool-less runs were close to the re-run results:
| Setup | Estimate | After re-run |
|---|---|---|
| Brave | 93% | 94% |
| Firecrawl | 85% | 89% |
| Tavily | 85% | 89% |
| Exa | 76% | 78% |
| Parallel | 67% | 72% |
| Perplexity | 62% | 72% |
What this shows
1. On these tasks, search changes the outcome. Without search, both agents failed every brief. With a good provider they reached the answer-excerpt setup's level. The difference against the floor is significant for every search setup on both agents (exact McNemar, p ≤ 0.0002). 2. The best provider depends on the agent. On claude-code, Parallel (95%) and Firecrawl (90%) led. On codex, Brave led (94%), ahead of Firecrawl and Tavily (89% each).
- Parallel tied for last on codex (72%), even with its tool loaded in every run. On codex decision briefs it passed 44%, against 92% on claude-code.
- So a provider's ranking on one agent does not carry over to another.
3. Pass rate does not predict cost between providers. Tokens per success counts every agent token in the setup, failed runs included, divided by passing runs.
- The hypothesis that better search shows up as both lower token spend and higher success holds against no search, which spent tokens and passed nothing. On claude-code, Parallel paired the top pass rate (95%) with 64k tokens per success; only native search, at 76%, spent less per success (61k).
- It does not hold as a ranking. Among the seven search setups, the rank correlation between pass rate and tokens per success was +0.05 on claude-code and −0.20 on codex; negative means higher-passing setups cost less. With seven setups, neither value is distinguishable from zero.
- Per-run spend varied widely: 46k (native) to 105k (Firecrawl) on claude-code, and 84k (Perplexity) to 148k (Firecrawl) on codex. Firecrawl passed 90% at 116k tokens per success, against Parallel's 95% at 64k.
- Choosing a provider means weighing pass rate and cost separately.
4. Decision briefs separate the providers; research briefs mostly don't. Research briefs passed at or near 100% on both agents. Decision briefs hinge on one recent fact, and they spread the search setups from 50% to 92% on claude-code and from 44% to 89% on codex. 5. Native search was mid-pack on both agents. On claude-code it passed 76%, level with Perplexity and behind Parallel, Firecrawl and Brave. On codex it passed 78%, level with Exa and behind Brave, Firecrawl and Tavily; under the codex judge it would pass 94% (see judge agreement). No search setup beat it significantly on either agent; on claude-code the largest gain was Parallel's, +19.0 points (p = 0.22).
Judge agreement
Two blinded judges scored every brief on the same evidence: the claude-code judge (the Claude judge below) decides the verdict and the codex judge measures agreement. Both judged at grading time, except for the 21 rerun native runs on claude-code, which codex scored on 2026-10-07 from the payloads the primary had judged.
| Agent | Briefs judged | Claude-only passes | Codex-only passes | Label agreement | Kappa |
|---|---|---|---|---|---|
| claude-code | 187 | 4 | 1 | 96% | 0.79 |
| codex | 161 | 0 | 7 | 96% | 0.78 |
A Claude-only pass is a disputed verdict the Claude judge passed and codex failed; a codex-only pass is the reverse. Label agreement pools the fact and claim labels both judges scored, and kappa is Cohen's kappa on them. Claims whose cited source could not be captured are left out, as no judge scored them. Briefs that failed schema validation were never judged (2 claude-code and 1 codex floor runs).
The disputes lean towards each judge's own model family, most clearly on codex's briefs: codex passed all 7 disputed ones. On claude-code's briefs the Claude judge passed 4 of the 5 here, but only 2 of the 4 in the replication, 6 of 9 together. The direction differs by agent (two-sided Fisher exact on dispute direction, both batteries' claude-code disputes pooled: p = 0.011). That is consistent with a judge favouring its own family's writing; with two judges the bench cannot say which judge carries the preference, or whether both do. Verdicts stay with the primary judge, as the design fixes in advance. Had codex decided, the pass counts would be:
| Setup | claude-code, Claude judge | claude-code, codex judge | codex, Claude judge | codex, codex judge |
|---|---|---|---|---|
| Parallel | 20/21 | 20/21 | 13/18 | 13/18 |
| Firecrawl | 19/21 | 19/21 | 16/18 | 17/18 |
| Brave | 17/21 | 15/21 | 17/18 | 17/18 |
| Perplexity | 16/21 | 16/21 | 13/18 | 15/18 |
| native | 16/21 | 16/21 | 14/18 | 17/18 |
| Exa | 15/21 | 14/21 | 14/18 | 14/18 |
| Tavily | 15/21 | 15/21 | 16/18 | 16/18 |
| ceiling (reference) | 20/21 | 20/21 | 16/18 | 17/18 |
| floor (reference) | 0/21 | 0/21 | 0/18 | 0/18 |
Only one count moves by more than two runs: codex's native setup, from 14/18 (78%) to 17/18 (94%), which would put it level with Brave and Firecrawl at the top of the codex table. Search against no search is unaffected: no floor brief passes under either judge.
Changes made during the run
The first live runs surfaced several bench defects. Two changed how runs execute, and two changed how runs are graded. The timeline was recorded from the battery change log and run start events.
Execution changes
Every run in the tables above ran under both of these fixes:
- GAPBUDGET-01: brief budgets became runaway caps (1M tokens, 60 calls). The old 30k cap was below the median codex search run.
- It was part of the pinned export before any calibration or battery run ran.
- No run ran under the old caps.
- GAPARM-01: brief search setups resolve the GAP catalog. Before it, no brief search setup could run, so no search-setup run predates it.
- It was applied at 18:30:04Z on 2026-10-03.
- The earliest search-setup run started at 18:30:29Z (codex round 1). claude-code started at 19:12Z.
The tool-isolation remediation, which forbids the GAP catalog to every GAP run, was applied at 18:50:40Z and the codex battery was restarted to load it.
- 18 codex round-1 runs started before it: classroom-term repetitions 1 and 2, all setups.
- No catalog access in them. Their transcripts contain only search calls and agent messages, with no file reads or commands, so the catalog was not read.
- They were kept, not re-run.
- Every other run ran after the remediation.
The codex provider re-run (2026-10-04). In the first pass, 31 of codex's 108 provider runs made no provider call:
| Setup | Runs without a provider call |
|---|---|
| Parallel | 12 |
| Firecrawl | 5 |
| Tavily | 5 |
| Perplexity | 5 |
| Brave | 3 |
| Exa | 1 |
- Cause: GAPMCP-01 reproduced codex's 30-second default MCP startup timeout. A 35-second server fails under it and starts with a 120-second allowance.
- How the runs were selected: the 31 runs were reclassified from an audit of their transcripts. All of them had zero provider calls and a
list_mcp_resourcesprobe; 29 of the 31 also said in their answer that no retrieval was available. The private audit is not distributed. - How they were re-run: with
--rerun-unavailableunder GAPMCP-01 (a 120-second startup allowance,required=trueand pre-warmed pinned packages), using the same matrix, calibration and catalog. - Every re-run run loaded its tool. Each one recorded a completed
initializeand tool listing. - A second fix was needed. The first re-run attempt misclassified working runs, because the MCP meter could not load its core (SEV2 METERAVAIL, below). Making the metering library available in each provider server's environment fixed it, and those runs were re-run again.
These 31 runs ran under the startup allowance; the other 77 provider runs ran without it but had their tool. No agent-side setting changed.
Grading changes
These changed only how stored answers were judged. Every affected run was regraded from the agents' stored answers, with no agent re-runs:
- GAPJUDGE-01: judges read readable text instead of raw HTML, and each source once. This cut judge tokens per run from about 250k to about 15k. It also covered:
- safe redirects;
- gzip decoding (python.org served gzip unconditionally, which garbled every python.org citation);
- over-budget sources withheld instead of the whole run going ungraded;
- the blinding leak check reading decoded text, and generic setup labels no longer counted as leaks.
- GAPRULE-01, operator-approved: the unsupported-claim limit was raised from 0.10 to 0.25. Ceiling agents see a short excerpt, and their hedges about it ("the notice does not specify X") were contradicted by the full pages the judges read. All stored grades were recomputed from their saved judge labels; 49 outcomes flipped, all from fail to pass.
Open issues
- SEV2 GAPMCP (fix GAPMCP-01, the bench fix): resolved. The 31 affected codex runs were re-run and the codex table updated.
- SEV2 METERAVAIL (fix METERAVAIL-01): provider-call metering was silently off for this battery on both agents. The meter runs inside the agent's restricted MCP child environment, and from the exported workbench tree it could not load its core.
- Vendor spend per run was therefore not metered.
- Only the 31 re-run runs have availability records; the report's "availability coverage" column counts those. Claude-code provider availability has not been verified in the published evidence.
- SEV2 ARGVKEY (fix ARGVKEY-01): the Parallel API key was passed in
mcp-remote's process arguments. The battery configuration now passes it through a 0600 header file. - The claude-code floor setup sometimes writes pretend searches as text when it has no tools. It is a legitimate failure, but slow and expensive (up to about 130k output tokens in 13 minutes).
- Next battery:
- the code corpus (GAP-07: 16 code tasks and 4 controls);
- the python-security-march excerpt fix;
- rubric fixes for spark-editing and cli-trust.
How this was run
GAP methodology
GAP grades completed jobs against captured primary sources. The initial corpus contained ten briefs: six decision and four research. Calibration is per agent and model: five repetitions of floor (no search) and ceiling (answer excerpt). Admission requires floor at most one pass and ceiling at least four passes. Seven Claude tasks and six Codex tasks were admitted. Code tasks were not run in this battery. The committed calibration file is an admission summary, not a runtime calibration file with captured oracle sources.
Each admitted task ran three times on nine setups. Brief grading requires decision correctness, weighted key-fact recall at least 0.7 and unsupported-claim rate at most 0.25. Two blinded judges used Claude Code as primary and Codex as secondary; the primary decides every verdict and the secondary measures agreement. The 21 Claude Code native runs rerun on 2026-10-06 were graded by the primary alone, and Codex scored their stored payloads on 2026-10-07 (sew gap agree), which rebuilds each payload from the stored source snapshots and checks its SHA-256 first. Gap closure is (setup pass rate − floor pass rate) / (ceiling pass rate − floor pass rate); it can exceed one. Wilson 95% intervals accompany pass rates. Gap-closure intervals use a seeded paired task bootstrap with 10,000 resamples. Exact McNemar tests pair task and repetition, against floor and native. The findings publish floor comparisons and the largest Codex native comparison, not every native test.
Tokens per success sum all agent tokens in an setup, including failures, divided by passes. Judge and calibration spend is separate. A zero-success floor has undefined tokens per success. Vendor spend was unmetered; tokens are not dollar cost.
GAPJUDGE-01 changed readable source extraction, redirects, gzip, source budget handling and blinding. GAPRULE-01 raised the unsupported-claim limit from 0.10 to 0.25; recomputing saved judge labels flipped 49 failures to passes. These grading changes did not rerun agents. Execution changes and the later rerun of 31 Codex runs with unavailable provider tools are preserved separately in REPORT.md.
See recorded results, summary, admission counts and reproduction.