Searchlight

Search gap bench · 2026-10-06

Search gap bench replication

In ten seconds

With any search setup, Claude Code passed 62–83% of briefs about recent events; with no search, 0%.

Claude Code claude-opus-5-5

24 runs per setup

Firecrawl passed most often: 20 of 24 runs. Every other setup's range overlaps the leader's, so the test cannot separate them.

  1. 1Firecrawlleader141k tokens per pass83%64–93%20 of 24
  2. 2Exa89k tokens per pass79%60–91%19 of 24
  3. 3Built-in55k tokens per pass75%55–88%18 of 24
  4. 4Parallel90k tokens per pass75%55–88%18 of 24
  5. 5Perplexity72k tokens per pass75%55–88%18 of 24
  6. 6Brave105k tokens per pass71%51–85%17 of 24
  7. 7Tavily126k tokens per pass62%43–79%15 of 24
What the numbers mean

8 research briefs about events after the model's training data, each run 3 times: 24 runs per setup. A brief passes if it reaches the right decision, gets at least 70% of the key facts (important ones count more), and no more than 25% of its claims lack support in the sources it cites.

For scale: with no search the agent passed 0%; handed an excerpt containing the answer, it passed 92%. The full report gives each setup's gap closed: how far it got from 0% toward 92%. Tokens per pass: all tokens used, failed runs included, divided by passing runs.

Source: Search gap bench replication; ranges are 95% Wilson intervals, as published.

The full report

GAP replication on Claude Code, 2026-10-06: the search gap bench, run again

Grading note. These verdicts come from the claude-code primary judge, which decides every verdict in GAP. The codex judge, which only measures agreement, was out of quota during the battery and scored the same stored payloads on 2026-10-07. It disputed 4 of the 211 judged verdicts and agreed on 96% of labels (kappa 0.82); no verdict changed. See judge agreement.

A second, independent Search Gap Bench battery on Claude Code (claude-opus-5-5), run on 2026-10-06 and 2026-10-07 with the apparatus as fixed after the 2026-10-03 battery: the built-in search setup can open pages, and every provider setup loaded its tool in every run. It repeats that battery's method on the current task catalog: calibrate each brief, keep the tasks with a real knowledge gap, and run every admitted task on nine setups, three times each.

Headline.

  • Search still decides the outcome. Without search the agent passed 0 of 24 runs; every search setup passed 62–83%, and each differs from no search significantly (exact McNemar, p < 0.0001).
  • The leader changed. Firecrawl led at 83%. Parallel, which led the 2026-10-03 battery at 95%, passed 75%. No search setup differs significantly from the agent's built-in search (largest gap: Firecrawl, +8.3 points, p = 0.50).
  • Built-in search was competitive and cheapest. It passed 75%, level with Parallel and Perplexity, at 55k tokens per success, the lowest of any search setup.
  • Pass rate again did not predict cost. The rank correlation between pass rate and tokens per success across the seven search setups was −0.04. Firecrawl's top pass rate came at 141k tokens per success.

Setup

Agentclaude-code on claude-opus-5-5
Setupsfloor (no search), ceiling (answer excerpt in the prompt), native (built-in WebSearch and WebFetch), Brave, Tavily, Exa, Parallel, Firecrawl, Perplexity
Tasksthe 10 brief tasks of the current GAP catalog; 8 admitted by calibration
Repetitions5 per reference setup in calibration; 3 per setup in the battery
Gradingdecision correctness, weighted key-fact recall (≥ 0.7) and unsupported-claim rate (≤ 0.25), judged blind by the claude-code primary judge against captured primary sources; the codex judge measured agreement afterwards on the same payloads
Spendabout 13.5M agent tokens in the battery; calibration and judge usage are not totalled here

Calibration

TaskFamilyclaude-code
billing-reportingresearchFloor: 0, Ceiling: 5 — admitted
classroom-termdecisionFloor: 0, Ceiling: 5 — admitted
models-productiondecisionFloor: 0, Ceiling: 5 — admitted
npm-token-scoperesearchFloor: 0, Ceiling: 5 — admitted
org-migrationdecisionFloor: 0, Ceiling: 4 — admitted
python-metadataresearchFloor: 0, Ceiling: 4 — admitted
python-security-marchresearchFloor: 0, Ceiling: 5 — admitted
tarfile-uploaddecisionFloor: 0, Ceiling: 5 — admitted
cli-trustdecisionFloor: 0, Ceiling: 3 — rejected
spark-editingdecisionFloor: 0, Ceiling: 1 — rejected

Every floor scored 0. Two briefs were rejected for failing the ceiling, as in the 2026-10-03 calibration: cli-trust (ceiling 3/5) and spark-editing (ceiling 1/5). python-security-march is admitted this time (ceiling 5/5); in the 2026-10-03 calibration its answer excerpt omitted the date its top fact requires, and the catalog has since fixed the excerpt.

Gap closure

claude-code (8 tasks, 24 runs per setup)
SetupGap closure (95% CI)Pass rate (Wilson 95% CI)Tokens per successvs floor (exact McNemar)
Firecrawl0.91 (0.80–1.00)83% (64–93%)141kp < 0.0001
Exa0.86 (0.58–1.11)79% (60–91%)89kp < 0.0001
Built-in (native)0.82 (0.60–0.96)75% (55–88%)55kp < 0.0001
Parallel0.82 (0.54–1.17)75% (55–88%)90kp < 0.0001
Perplexity0.82 (0.64–0.96)75% (55–88%)72kp < 0.0001
Brave0.77 (0.50–1.06)71% (51–85%)105kp < 0.0001
Tavily0.68 (0.33–0.96)62% (43–79%)126kp < 0.0001
ceiling (reference)1.0092% (74–98%)5k
floor (reference)0.000% (0–14%)—

By family:

  • Research briefs: Firecrawl, Perplexity and Exa 92%; Parallel and Built-in 83%; Brave and Tavily 75%.
  • Decision briefs: Firecrawl 75%; Parallel, Brave, Exa and Built-in 67%; Perplexity 58%; Tavily 50%.

Against the 2026-10-03 battery

The task set differs by one brief (python-security-march), so the fair comparison is on the seven briefs both batteries admitted. The 2026-10-03 built-in row is its 2026-10-06 rerun with page fetching working.

Setup2026-10-03 battery (7 tasks)Replication, same 7 tasksFisher exactReplication, all 8 tasks
Parallel20/21 (95%)16/21 (76%)p = 0.1818/24 (75%)
Firecrawl19/21 (90%)18/21 (86%)p = 1.0020/24 (83%)
Brave17/21 (81%)16/21 (76%)p = 1.0017/24 (71%)
Built-in (native)16/21 (76%)17/21 (81%)p = 1.0018/24 (75%)
Perplexity16/21 (76%)16/21 (76%)p = 1.0018/24 (75%)
Exa15/21 (71%)17/21 (81%)p = 0.7219/24 (79%)
Tavily15/21 (71%)15/21 (71%)p = 1.0015/24 (62%)
ceiling (reference)20/21 (95%)19/21 (90%)p = 1.0022/24 (92%)

On the shared briefs no setup's change between batteries is statistically distinguishable (two-sided Fisher exact, smallest p = 0.18); Parallel's drop from 20/21 to 16/21 is the largest. With 21 runs per setup, neighbouring providers cannot be ranked against each other, and a single battery's ordering should not be quoted as a ranking. On the new brief, python-security-march, the setups passed: Exa, Parallel, Firecrawl and Perplexity 2/3 each; Brave and built-in 1/3 each; Tavily 0/3 (ceiling 3/3).

Judge agreement

On 2026-10-07 the codex judge scored every judged brief from the payload the primary judge had scored. Each payload was rebuilt from the run's stored source snapshots, so nothing was captured again, and codex was called only after the rebuilt payload's SHA-256 matched the primary's and the stored labels reproduced the stored verdict. The primary was not rerun. 5 briefs failed schema validation and were never judged; they fail under either judge.

SetupBriefs judgedClaude-only passesCodex-only passesLabel agreementKappaPass, Claude judgePass, codex judge
Firecrawl240198%0.8220/2421/24
Exa240195%0.7219/2420/24
Built-in (native)240097%0.8518/2418/24
Parallel231095%0.7018/2417/24
Perplexity240097%0.8118/2418/24
Brave240097%0.7417/2417/24
Tavily240094%0.7115/2415/24
ceiling (reference)241098%0.5922/2421/24
floor (reference)200093%0.860/240/24
All setups2112296%0.82147/216147/216

A Claude-only pass is a disputed verdict the Claude judge passed and codex failed; a codex-only pass is the reverse. Label agreement pools the fact and claim labels both judges scored, and kappa is Cohen's kappa on them; claims whose cited source could not be captured are left out, as no judge scored them.

The four disputed verdicts split evenly: the Claude judge alone passed one ceiling and one Parallel brief, and codex alone passed one Exa and one Firecrawl brief. Had codex decided, the battery's pass count would be the same, 147 of 216, and no setup would move by more than one run. Every headline above holds under either judge: the floor passes nothing, every search setup differs from it (exact McNemar, p ≤ 0.0001), Firecrawl still leads at 21/24, and no search setup differs significantly from built-in search (largest gap: Firecrawl, p = 0.25). Agreement is in line with the 2026-10-03 battery on Claude Code's briefs (96% of labels, kappa 0.79). There, 4 of the 5 disputes on Claude Code's briefs went the Claude judge's way; here that lean did not repeat.

What this shows

1. Search changes the outcome, again. Without search the agent failed every brief; with any provider it passed most of them. This finding replicated exactly. 2. Provider rankings did not replicate. The top setup changed and the spread between search setups is within sampling noise at this size. Choose a provider on the agent you use, on cost and on integration, and measure. 3. Built-in search is a serious baseline once it can read pages. Level with the middle of the field at the lowest token cost per success. 4. Pass rate and cost remain separate questions. The cheapest setup per success and the highest-passing setup were different setups in both batteries.

Limits

  • One agent and model; the codex half of the 2026-10-03 design was not repeated.
  • One judge decides the verdicts. The second judge scored the stored payloads after the battery rather than alongside it; the evidence it saw is identical, checked by payload hash.
  • 24 runs per setup: 95% intervals span roughly 30 points.
  • Provider indexes, sources and models drift between batteries; a replication tests the method, not a fixed truth.
How this was run

GAP replication methodology

This battery repeats the 2026-10-03 GAP methodology on one agent: calibration per task (five floor and five ceiling repetitions; admission requires the floor to pass at most once and the ceiling at least four times), then every admitted task three times on nine setups.

Differences from the 2026-10-03 battery:

  • Catalog. The current GAP catalog (SHA-256 29f8fd1731cd0155f2a993688d8f1dfac7ad6ef1f6effc549185af026ad24443), in which python-security-march's answer excerpt carries the date its top fact needs. 8 of the 10 briefs were admitted.
  • Agent. Claude Code on claude-opus-5-5 only.
  • Built-in search. The native setup pre-approves WebFetch as well as WebSearch, so the agent can open pages.
  • Provider tools. Every provider setup made provider calls in every run; Parallel's credential is passed as a header file the workbench writes per run.
  • Judging. The claude-code primary judge decides every verdict, as before. The codex judge, which measures agreement, was out of quota during the battery. It scored the same stored payloads on 2026-10-07 with sew gap agree, which rebuilds each payload from the stored source snapshots and calls codex only when the payload's SHA-256 matches the primary's and the stored labels reproduce the verdict.

Statistics are unchanged: Wilson 95% intervals on pass rates, a seeded paired task bootstrap (10,000 resamples) for gap closure, exact McNemar tests against the floor and the built-in setup, and tokens per success counted over all agent tokens in the setup, failures included. The comparison with the 2026-10-03 battery uses two-sided Fisher exact tests on the seven shared briefs, because the two batteries' runs are not paired.

See recorded results, summary, admission counts and reproduction.

Files: reports/2026-10-06-search-gap-replication/