WSB live battery — initial results (2026-09-29)

Correction (2026-10-06): the native arm's page fetches were mostly refused. The arm exposed WebSearch and WebFetch but pre-approved only WebSearch, and headless Claude Code refuses tools that are not pre-approved. WebFetch was refused in 40 of 54 native cells (110 calls). Provider arms were unaffected. The native rows therefore understate Claude Code's own web tools. The configuration is fixed and pinned by a test; the recorded numbers below are unchanged. Details: agent search behavior.

Pack: web-search-bakeoff (WSB) on the Search Evaluation Workbench (SEW). Harness / model: claude-code, Claude Opus 5.5 (claude-opus-5-5[1m]), authenticated execution on the original host. Status: historical initial grading; headline and Results tables are superseded by the 2026-09-30 regrade, where Firecrawl leads at 90%. The numbers below are graded with SEWBENCH-01 applied to the stored deliverables. The same deliverables graded without those fixes are summarised in Failure dive and were materially wrong.

Headline

Historical figures under the initial grader: rankings and cost lower bounds here apply only to that grading, not the later operator-ruled catalog.

On the 14 competitive production tasks (42 cells per arm), every search arm except Brave scores above answering without search:

• Firecrawl and Perplexity lead at 83%. Exa is at 81%, Tavily 79%, native web search and Parallel 76%, no-search 71% and Brave 69%.
• The 95% intervals overlap. At n=42 per arm the ordering is directional, not conclusive: no arm's difference from no-search or from native has an interval that excludes zero.
• Search matters most on tasks that need live data. On the cloud egress pricing table only Firecrawl and Perplexity went 3/3 (Brave 2/3, the rest 0/3). The AWS pricing pages need JavaScript rendering, and most arms reported the intra-AZ cell as unavailable, honestly.
• Among fully priced arms, native search costs least per success ($0.167), followed by Tavily ($0.206). Perplexity (at least $0.148; 37/42 cells priced) and Firecrawl (at least $0.159; 39/42 priced) are lower bounds, excluded from the cost ranking; Perplexity's ask spend is unmetered. Brave and no-search are also partially priced (at least $0.28 each); no-search had a few long runaway answers.
• Brave is last, failing unanswerable-nonexistent-postgres-guc and upstream-diagnosis-node-openssl3-md4 in all three repetitions.

What ran

• Tasks: all 18 production tasks (production task catalog), × 3 repetitions.
  • 14 are competitive.
  • 4 are expected_fail by design: no retrieval surface can answer them, and the honest deliverable is a partial answer or a decline.
• Arms:
  • Provider arms expose one vendor MCP server each, and nothing else.
  • The native arm uses the harness's own web search.
  • The no-search arm has no network tools.
• Pinned MCP servers:
  • @brave/brave-search-mcp-server version 2.1.4
  • tavily-mcp@0.2.22
  • exa-mcp-server@3.4.1
  • firecrawl-mcp@3.26.0
  • @perplexity-ai/mcp-server version 1.3.0
  • Parallel's hosted Search MCP (https://search.parallel.ai/mcp-oauth, authenticated) via mcp-remote@0.14.3
• Grading:
  • Deterministic validators where a task has one.
  • Otherwise a blinded claude-code judge scoring the task's rubric.
  • bakeoff grade --regrade re-scores the stored deliverables. Nothing was re-run.
• Cost: model tokens are priced at the published Opus 5.5 rates. Vendor spend is metered per MCP call where a tariff exists. It is unknown for Exa, Parallel and Perplexity's ask tool, which report no spend over MCP. In the tables, partial is the priced cells' spend divided by all of the arm's successes. Unpriced cells can only add spend, so each partial figure is a lower bound on the full-arm $/success, and these arms are excluded from the cost ranking. bakeoff report divides by the successes among priced cells instead, which gives $0.172 for Perplexity, $0.163 for Firecrawl and $0.29 for Brave on the competitive set, and no figure for Exa or Parallel on the expected-fail set, where every success is in an unpriced cell.

Results

All Results tables below use the historical initial grading. The ungraded column counts delivered answers that could not be graded. Cells that exhausted their budget before delivering are counted under budget-exhausted and as failures, rather than under ungraded.

Competitive tasks (14 tasks × 3 repetitions)

Competitive deltas against controls

Each arm against each control over the same 14 tasks × 3 repetitions. Intervals are 95% Newcombe (hybrid Wilson score) intervals for a difference of two independent proportions. They ignore the task pairing, so they are conservative. Token ratios compare mean tokens per cell. Provider arms and their controls come from two runs about ten hours apart (see Caveats).

Expected-fail tasks (4 tasks × 3 repetitions)

Passing here means an honest partial answer or decline, which the validator can confirm.

Per task

Findings

1. Search earns its cost on live or long-tail data, not on stable knowledge.

• Tasks whose answers are stable and well known were 3/3 in every arm, including no-search: copyleft obligations, PEP 594 removals, Python feature PEPs and the urllib3 diagnosis.
• The separation comes from the tasks that need current data:
  • the egress pricing table;
  • the SCA tool shortlist (no-search 0/3; Exa, Parallel and Perplexity 3/3);
  • the dockershim change-detection task.

2. Rendering matters for vendor pricing pages. Firecrawl (which renders JavaScript) and Perplexity were the only arms to read AWS's intra-AZ transfer pricing. The others honestly reported it unavailable, and the validator fails an unavailable cell. 3. Declining is not the default for search arms. unanswerable-nonexistent-postgres-guc separates the arms: Brave 0/3 and Perplexity 1/3 against 3/3 for native, Tavily and Firecrawl. 4. Exa handles impossible tasks best (5/12 honest partials on the expected-fail set). It is the only arm to go 3/3 on multi-hop-npm-semver-dependents, where the honest move is to state the method and its limits rather than invent a ranking. 5. Provider arms use more reported tokens per cell. The MCP provider arms report 42k–65k tokens per cell against 15k for no-search. Native search reports 28k. These totals do not separate input from output tokens, which have different prices, and some vendor spend is unknown; the totals alone cannot establish what dominates cost per success.

Failure dive

Two tasks were initially at 0/24 across all arms; a third mostly failed, and the judged SCA task was mostly ungraded. All four exposed bench bugs. Each was confirmed by reading the deliverables, and each is fixed in the bench fix:

With the fixes, the competitive pass rate rose between 14 and 29 points per arm (no-search +14, Exa +29). It rose most for search arms, which had been penalised for citing and explaining evidence. The first reading of this battery, that no-search was about as good as any provider, was an artifact of these bugs.

Genuine failures that remain:

• Egress pricing: unavailable AWS intra-AZ cells, as above.
• change-detection-openapi-30-to-31: most misses omit example→examples from the breaking changes. OAS 3.1 deprecates example rather than removing it. The tables above still require it; the operator has since ruled it is not a break (see the update below).
• Budget exhaustion: 8 cells. Six are Firecrawl or Parallel on long list-building and enumeration tasks; Brave and Tavily have one each.
• By design: the two remaining expected_fail tasks (unversioned-runtime-drift, unindexed-ordering-regression) are 0/3 everywhere.

Codex (partial)

The codex harness (gpt-6-sol) ran one competitive task on 2026-09-28 before its weekly quota ran out:

• Egress pricing: native 2/3, no-search 1/3. Brave and Tavily went 0/3 on the token budget, reading 393k–448k tokens per cell.
• Expected-fail tasks: codex's search arms burned the full provider-call budget on multi-hop-npm-semver-dependents instead of declining.

A full codex battery waits for the quota reset (2026-10-04 12:52Z).

Caveats

• Sample size: n=42 per arm on the competitive set. Adjacent arms are within each other's intervals.
• Timing: the two runs were about ten hours apart. Same harness, model, tasks and budgets.
• Pricing: vendor spend is unpriced for Exa and Parallel, and for Perplexity's ask tool. Their $/success counts model tokens plus metered calls only.
• Grading: the judged tasks use a single claude judge, so agreement is not measured.

Update (2026-09-30): operator decisions

• OpenAPI: example→examples is a deprecation, not a break, so the task no longer requires it. The task also stops requiring a construct labelled as the JSON Schema dialect: the dialect has its own field, and complete answers list its consequences instead. Both changes are in the bench fix. On replay, the task goes from 7/24 to 22/24. The two remaining misses cite hosts outside the allowlist.
• Codex native search is priced at Brave's per-request rate as a proxy for its backing vendor. Claude native search keeps Anthropic's published rate.
• Tokens and outcome rate are reported together. Tokens per success come from the bench fix.

Competitive pass rates regraded with the corrected catalog at the recorded grader revision:

The historical tokens/cell column averages measured total-billable tokens over attempted cells with measured usage; unknown or estimated usage is excluded, never treated as zero. The update's tokens/success sums measured tokens over graded cells and divides by measured successes, as defined by the bench fix; it excludes ungraded and unmeasured cells and is not a full-arm lower bound. The per-arm measured coverage counts were not retained here. Multiplying a measured-cell mean by all 42 attempts need not reproduce it. The previously quoted fresh/cache/output means are withdrawn: their cell coverage and bucket accounting were not recorded here and they do not reconcile for Firecrawl, Parallel or no-search. They cannot support a token-mix comparison from this document.

The output-reduction claim is also withdrawn. The quoted no-search mean included long runaway answers; without a median or an outlier-excluded comparison, this battery does not establish a typical reduction or a causal effect of search on output length. Among the six MCP providers, a higher pass rate goes with fewer tokens per success: Perplexity is best on tokens, and Brave is worst on both. Spearman ρ ≈ −0.83 (pass rate versus tokens/success, n=6) is suggestive, not established, and is not significant at α=0.05. No arm's pass-rate difference from no-search has an interval that excludes zero. Firecrawl comes closest at +14 points (−2 to +30).

WSB methodology

The experiment used Claude Code with claude-opus-5-5[1m], all 18 production tasks, eight tool arms and three repetitions. The two arm groups ran about ten hours apart. Arms differ in exposed search tools: no-search, native, or exactly one provider MCP server. Budgets and contamination checks apply to each cell.

The 14 competitive tasks and four expected-fail tasks have separate denominators. Expected-fail success means an honest partial answer or decline. Deterministic validators grade structured tasks; a blinded Claude judge grades rubric tasks. Single-judge agreement was not measured. Pass-rate intervals are Wilson 95%; control deltas use independent-proportion Newcombe intervals, ignoring pairing. Adjacent rankings are not statistically established.

The original bench corrections fixed overly strict date/dollar decline checks, a prose-versus-URL field mismatch and judge JSON parsing/blinding. The later regrade relaxed incorrect OpenAPI breaking-change requirements. Both grade the same stored answers; neither reran agents. REPORT.md retains initial and updated tables with their original precision.

Historical cost lower bounds sum priced spend and divide by all successes. The report command instead uses priced-cell successes, so its output can differ. Unknown vendor spend is not zero; partially priced arms are excluded from cost ranking. Updated tokens per success use measured tokens from graded cells divided by measured successes; coverage counts were not retained. Do not multiply the historical tokens/cell means by attempted-cell counts to derive that update. The output-reduction and fresh/cache/output comparisons were withdrawn.

See recorded results, summary and reproduction. Calibration was not part of this experiment: calibration record.

GAP battery results, 2026-10-03: does search change the outcome of the job?

Correction (2026-10-06): the Claude Code native arm could not fetch pages. The arm exposed WebSearch and WebFetch but pre-approved only WebSearch, and headless Claude Code refuses tools that are not pre-approved. Every WebFetch call in the native arm was refused (50 calls across all 21 native cells). Each provider arm could use its vendor's fetch tool, so the Claude Code native row measures search without page fetching and is not comparable to the provider rows. The codex native arm was unaffected. The configuration is fixed and pinned by a test; the recorded numbers below are unchanged. Details: agent search behavior.

The first full Search Gap Bench (GAP) battery ran on 2026-10-03, operator-approved: brief tasks on two harnesses, nine arms each, three repetitions.

• GAP grades the job, not the retrieval.
• A task counts only after calibration shows its knowledge gap: the no-search arm (floor) passes at most 1 of 5, and the arm given the answer excerpt (ceiling) passes at least 4 of 5.
• Each search arm is scored by gap closure: how much of the floor-to-ceiling distance it covers.
• Tokens are reported beside every outcome.

Headline. On these tasks, search decides the outcome.

• The floor passed 0% on both harnesses, and every search arm passed 62–95%.
• Which provider is best depends on the harness: Parallel led on claude-code and Brave on codex.
• Pass rate did not predict cost: providers with similar pass rates differed almost twofold in tokens per success.

These are the recorded results after the Codex reruns. Claude-code provider tool availability remains unverified in the published evidence: no transcript audit for those cells is documented, and they lack availability records. Its provider rankings are provisional pending that audit. In the first codex pass, 31 of 108 provider cells ran without their search tool. They were re-run on 2026-10-04 under the GAPMCP-01 fix, every re-run cell loaded its tool, and the codex tables below include them (see "Execution changes").

Setup

Calibration

Final admissions under GAPRULE-01. Each cell shows floor passes and ceiling passes out of 5.

Every floor scored 0, so every task has a real knowledge gap. All rejections are ceiling failures, which point at task or rubric defects:

• spark-editing: the heaviest key fact is not what the prompt asks about.
• python-security-march: the excerpt omitted the date that its top fact requires. The excerpt fix is pending for the next battery; GAPRULE-01 changed the unsupported-claim limit, not the excerpt.
• cli-trust: with three claims, a single unsupported claim fails the brief.
• tarfile-upload on codex: the codex briefs miss key facts.

Gap closure

claude-code (7 tasks, 21 cells per arm)

By family:

• Research briefs: every provider arm passed 100% (native 89%).
• Decision briefs: they separate the providers. Parallel 92% and Firecrawl 83% lead; Brave scored 67%, Perplexity 58%, Exa and Tavily 50% each, and native 42%.

codex (6 tasks, 18 cells per arm)

No search arm beat native significantly; the largest gain was Brave at +16.7 points (p = 0.25).

By family:

• Research briefs: every arm passed 100%, except Exa at 89%.
• Decision briefs: they separate the providers. Brave scored 89%; Firecrawl and Tavily 78% each; Exa 67%; native 56%; Parallel and Perplexity 44% each.

The first pass's estimates that excluded the 31 tool-less cells were close to the re-run results:

What this shows

1. On these tasks, search changes the outcome. Without search, both harnesses failed every brief. With a good provider they reached the answer-excerpt arm's level. The difference against the floor is significant for every search arm on both harnesses (exact McNemar, p ≤ 0.0002). 2. The best provider depends on the harness. On claude-code, Parallel (95%) and Firecrawl (90%) led. On codex, Brave led (94%), ahead of Firecrawl and Tavily (89% each).

• Parallel tied for last on codex (72%), even with its tool loaded in every cell. On codex decision briefs it passed 44%, against 92% on claude-code.
• So a provider's ranking on one harness does not carry over to another.

3. Pass rate does not predict cost between providers. Tokens per success counts every agent token in the arm, failed cells included, divided by passing cells.

• The hypothesis that better search shows up as both lower token spend and higher success holds against no search, which spent tokens and passed nothing. It also holds for Parallel on claude-code, which led on both measures (64k tokens per success).
• It does not hold as a ranking. Among the seven search arms, the rank correlation between pass rate and tokens per success was +0.13 on claude-code and −0.20 on codex; negative means higher-passing arms cost less. With seven arms, neither value is distinguishable from zero.
• Per-cell spend varied widely: 41k (native) to 105k (Firecrawl) on claude-code, and 84k (Perplexity) to 148k (Firecrawl) on codex. Firecrawl passed 90% at 116k tokens per success, against Parallel's 95% at 64k.
• Choosing a provider means weighing pass rate and cost separately.

4. Decision briefs separate the providers; research briefs mostly don't. Research briefs passed at or near 100% on both harnesses. Decision briefs hinge on one recent fact, and they spread the search arms from 42% to 92% on claude-code and from 44% to 89% on codex. 5. Native search was not the best choice on either harness. It came last among search arms on claude-code (62%). On codex it was mid-pack (78%, level with Exa), behind Brave, Firecrawl and Tavily, though no arm beat it significantly there.

Changes made during the run

The first live runs surfaced several bench defects. Two changed how cells execute, and two changed how cells are graded. The timeline was recorded from the battery change log and cell start events.

Execution changes

Every cell in the tables above ran under both of these fixes:

• GAPBUDGET-01: brief budgets became runaway caps (1M tokens, 60 calls). The old 30k cap was below the median codex search cell.
  • It was part of the pinned export before any calibration or battery cell ran.
  • No cell ran under the old caps.
• GAPARM-01: brief search arms resolve the GAP catalog. Before it, no brief search arm could run, so no search-arm cell predates it.
  • It was applied at 18:30:04Z on 2026-10-03.
  • The earliest search-arm cell started at 18:30:29Z (codex round 1). claude-code started at 19:12Z.

The tool-isolation remediation, which forbids the GAP catalog to every GAP cell, was applied at 18:50:40Z and the codex battery was restarted to load it.

• 18 codex round-1 cells started before it: classroom-term repetitions 1 and 2, all arms.
• No catalog access in them. Their transcripts contain only search calls and agent messages, with no file reads or commands, so the catalog was not read.
• They were kept, not re-run.
• Every other cell ran after the remediation.

The codex provider re-run (2026-10-04). In the first pass, 31 of codex's 108 provider cells made no provider call:

• Cause: GAPMCP-01 reproduced codex's 30-second default MCP startup timeout. A 35-second server fails under it and starts with a 120-second allowance.
• How the cells were selected: the 31 cells were reclassified from an audit of their transcripts. All of them had zero provider calls and a list_mcp_resources probe; 29 of the 31 also said in their answer that no retrieval was available. The private audit is not distributed.
• How they were re-run: with --rerun-unavailable under GAPMCP-01 (a 120-second startup allowance, required=true and pre-warmed pinned packages), using the same matrix, calibration and catalog.
• Every re-run cell loaded its tool. Each one recorded a completed initialize and tool listing.
• A second fix was needed. The first re-run attempt misclassified working cells, because the MCP meter could not load its core (SEV2 METERAVAIL, below). Making the metering library available in each provider server's environment fixed it, and those cells were re-run again.

These 31 cells ran under the startup allowance; the other 77 provider cells ran without it but had their tool. No agent-side setting changed.

Grading changes

These changed only how stored answers were judged. Every affected cell was regraded from the agents' stored answers, with no agent re-runs:

• GAPJUDGE-01: judges read readable text instead of raw HTML, and each source once. This cut judge tokens per cell from about 250k to about 15k. It also covered:
  • safe redirects;
  • gzip decoding (python.org served gzip unconditionally, which garbled every python.org citation);
  • over-budget sources withheld instead of the whole cell going ungraded;
  • the blinding leak check reading decoded text, and generic arm labels no longer counted as leaks.
• GAPRULE-01, operator-approved: the unsupported-claim limit was raised from 0.10 to 0.25. Ceiling agents see a short excerpt, and their hedges about it ("the notice does not specify X") were contradicted by the full pages the judges read. All stored grades were recomputed from their saved judge labels; 49 outcomes flipped, all from fail to pass.

Open issues

• SEV2 GAPMCP (fix GAPMCP-01, the bench fix): resolved. The 31 affected codex cells were re-run and the codex table updated.
• SEV2 METERAVAIL (fix METERAVAIL-01): provider-call metering was silently off for this battery on both harnesses. The meter runs inside the harness's restricted MCP child environment, and from the exported workbench tree it could not load its core.
  • Vendor spend per cell was therefore not metered.
  • Only the 31 re-run cells have availability records; the report's "availability coverage" column counts those. Claude-code provider availability has not been verified in the published evidence.
• SEV2 ARGVKEY (fix ARGVKEY-01): the Parallel API key was passed in mcp-remote's process arguments. The battery configuration now passes it through a 0600 header file.
• The claude-code floor arm sometimes writes pretend searches as text when it has no tools. It is a legitimate failure, but slow and expensive (up to about 130k output tokens in 13 minutes).
• Next battery:
  • the code corpus (GAP-07: 16 code tasks and 4 controls);
  • the python-security-march excerpt fix;
  • rubric fixes for spark-editing and cli-trust.

GAP methodology

GAP grades completed jobs against captured primary sources. The initial corpus contained ten briefs: six decision and four research. Calibration is per harness and model: five repetitions of floor (no search) and ceiling (answer excerpt). Admission requires floor at most one pass and ceiling at least four passes. Seven Claude tasks and six Codex tasks were admitted. Code tasks were not run in this battery. The committed calibration file is an admission summary, not a runtime calibration file with captured oracle sources.

Each admitted task ran three times on nine arms. Brief grading requires decision correctness, weighted key-fact recall at least 0.7 and unsupported-claim rate at most 0.25. Two blinded judges used Claude Code as primary and Codex as secondary. Gap closure is (arm pass rate − floor pass rate) / (ceiling pass rate − floor pass rate); it can exceed one. Wilson 95% intervals accompany pass rates. Gap-closure intervals use a seeded paired task bootstrap with 10,000 resamples. Exact McNemar tests pair task and repetition, against floor and native. The findings publish floor comparisons and the largest Codex native comparison, not every native test.

Tokens per success sum all agent tokens in an arm, including failures, divided by passes. Judge and calibration spend is separate. A zero-success floor has undefined tokens per success. Vendor spend was unmetered; tokens are not dollar cost.

GAPJUDGE-01 changed readable source extraction, redirects, gzip, source budget handling and blinding. GAPRULE-01 raised the unsupported-claim limit from 0.10 to 0.25; recomputing saved judge labels flipped 49 failures to passes. These grading changes did not rerun agents. Execution changes and the later rerun of 31 Codex cells with unavailable provider tools are preserved separately in REPORT.md.

See recorded results, summary, admission counts and reproduction.

How agents used search: query style, fetch-from-memory and primary-source retrieval (2026-10-05)

This report analyses the stored transcripts of the two published batteries, the web search bakeoff and the search gap bench. No agent was rerun. It asks how agents used the search tools they were given, rather than which provider scored best. Every number below comes from scripts/analyze_search_behavior.py, whose classification rules are listed in methodology. The analysis covers 974 search queries from the GAP battery (both harnesses) and 655 from the bakeoff (Claude Code).

Headline

• Agents write keyword queries whatever the engine asks for. Of 974 GAP search queries, 7.3% read as natural language. In the bakeoff the figure is 6.0%. Exa's tool asks for a description of the ideal page rather than keywords; its queries were natural language in 3% (Claude Code) and 2% (codex) of GAP calls. Because the automatic rule can miss descriptive phrasing, all 111 Exa queries from both batteries were also read by hand (appendix): none is phrased as a description of a page. The closest are a headline-like phrase used three times ("Python 4.0 release date announced by Python Steering Council") and a page title.
• The two harnesses express constraints differently. On the six provider arms, codex put site: into 31% to 67% of its queries, even where a structured domain filter existed. Claude Code wrote site: only on Firecrawl (6%) and otherwise used structured domain filters where a tool offered one (Perplexity search_domain_filter, Tavily include_domains, WebSearch allowed_domains).
• Agents often fetch remembered URLs instead of searching. In the bakeoff, Claude Code fetched pages without any search in 36 of 55 Exa cells, 35 of 55 Firecrawl cells, 34 of 54 Tavily cells and 30 of 55 Parallel cells. Those provider arms therefore measured page fetching as much as search. On post-cutoff GAP tasks the same habit failed: Claude Code's Exa cells that only fetched passed 1/3, against 14/18 for its Exa cells that searched.
• Retrieving the primary source mattered most on Claude Code. Pooled over provider and native arms, Claude Code passed 94/111 cells (85%) when the task's primary source appeared in search results and 20/33 (61%) when it did not. Codex surfaced the primary source in most searched cells, so the comparison is thin there (95/116 against 8/10). Two arms passed without the primary page: Parallel (6/7) and Firecrawl (9/9) on Claude Code, which indicates their results carried the facts from other pages.

These are observations about agent behavior with each vendor's MCP server at its pinned version and default settings. They are not measurements of the vendors' APIs.

What each search tool asks for

Descriptions are quoted from the pinned MCP server packages and, for Parallel's hosted server, from its published documentation.

Query style (GAP battery)

Parallel's search_queries are keyword queries by design; its natural-language part is the separate objective field, which is counted as a structured parameter and not as a query. Perplexity's user messages count as queries alongside its query inputs; system and assistant messages are excluded. Codex native search reports its query lists in the transcript; page views are counted as fetches.

Query style (web search bakeoff, Claude Code)

Searching versus fetching from memory (web search bakeoff, Claude Code)

A cell is "fetched without searching" when it made at least one fetch call and no search call. Brave's and Perplexity's servers expose no page-fetch tool, so their agents had to search or answer from memory. Every arm has a handful of cells with no provider call: answers given from memory and cells that ended before any tool call.

On the bakeoff's stable-knowledge tasks, fetching a remembered URL is efficient: those tasks passed in every arm, including no-search. It means the bakeoff's provider arms with a fetch tool mostly measured that fetch path. The GAP battery, whose answers post-date the models, shows the cost of the same habit on new information.

Harness refusals: the Claude Code native arm

Claude Code's native arm exposed WebSearch and WebFetch through --tools but pre-approved only WebSearch through --allowedTools. Headless Claude Code refuses a tool that is exposed but not pre-approved ("Claude requested permissions to use WebFetch, but you haven't granted it yet"). In the GAP battery every native WebFetch call was refused (50 calls in 21 of 21 cells); in the bakeoff, 110 calls in 40 of 54 cells. No provider arm and no codex arm had a refused call. The Claude Code native results in both batteries therefore understate that harness's web tools. The arm configuration is fixed and covered by a regression test. The fetch counts in this report include refused attempts.

Primary-source retrieval (GAP battery)

Each GAP brief has one primary-source URL (the catalog oracle). A cell surfaced it when that URL appeared among the URLs of returned search results, excluding search inputs. Unknown coverage is excluded from both pass-rate groups. Recomputing against returned evidence leaves the historical source counts unchanged; no searched cell has unknown coverage. Facts can also come from secondary pages, so surfacing is a proxy for retrieval, not a requirement for passing.

Limits

• Cell counts per arm are small (18 to 21 in GAP, 54 to 55 in the bakeoff). Per-arm percentages move by about five points per cell.
• The natural-language rule is a heuristic, stated in methodology; readers can rerun the script with their own rule.
• Each vendor was used through its MCP server at the pinned version and default settings. Other interfaces of the same vendor (direct API, other tools, other modes) were not tested.
• The two harnesses ran different task subsets in GAP (7 tasks for Claude Code, 6 for codex), so cross-harness differences mix agent behavior with task mix.

Appendix: every Exa query

Every query sent to web_search_exa in both batteries, in transcript order, so readers can judge the phrasing directly. Other arms' queries are summarised above but not reproduced.

Agent search behavior: methodology

The analysis reads the stored transcripts of the web search bakeoff (Claude Code, eight arms, 18 tasks, three repetitions) and the search gap bench (Claude Code and codex, brief tasks, three repetitions per admitted task). It runs no agent and changes no grade. Pass and fail come from each cell's recorded evaluation.

scripts/analyze_search_behavior.py extracts every completed provider tool call from a transcript:

• codex MCP tool calls, and codex built-in web_search items;
• Claude Code tool_use and tool_result pairs, including vendor MCP tools, WebSearch and WebFetch.

Classification rules

• Search call: a call to a search tool (web_search_exa, brave_web_search, tavily_search, Parallel web_search, firecrawl_search, the Perplexity search tools, Claude Code WebSearch, and codex built-in search actions).
• Query: each entry of a multi-query call (Parallel search_queries, codex built-in query lists) counts as one query. Each user message in Perplexity's conversational inputs also counts as one query; system and assistant messages are excluded. Parallel's objective is counted as a structured parameter when search_queries is present, not as another query.
• Fetch call: a page-retrieval tool call (web_fetch_exa, tavily_extract, firecrawl_scrape, Parallel web_fetch, Claude Code WebFetch, and codex built-in open_page, find_in_page and other page views).
• Natural-language query: ends with "?", starts with an interrogative or auxiliary verb, or contains at least two function words from a fixed list (the, a, an, is, are, was, were, does, do, did, how, what, which, why, when, where, who, that, this, of, to, for, with, will, should, can, about, on, from, by, as, be, has, have).
• Contains a year: a token from 2000 to 2099. Quoted phrase: contains a double quote. site:: contains site:.
• Fetched without searching: the cell made at least one fetch call and no search call.
• Primary source surfaced (GAP only): the task's catalog oracle source_url, normalised (scheme, query, fragment and trailing slash removed), appears among the URLs in any returned search result of the cell, never in the search input. It is measured only over cells that searched. A cell is unknown when any search lacks exposed results and no observed result contains the primary source. Unknown cells are excluded from both surfaced and not-surfaced pass-rate denominators. Recomputing against returned evidence leaves the historical source counts unchanged; no searched cell has unknown coverage.

The rules are deliberately simple so they can be audited. The natural-language rule undercounts descriptive keyword-free phrases and overcounts keyword strings that happen to contain function words. Readers can substitute their own rule in the script and rerun it.

Scope

Pooled figures cover the provider and native arms. Reference arms (no-search, floor and ceiling) make no search calls and are excluded from query statistics. GAP cell counts differ by harness because calibration admitted seven brief tasks for Claude Code and six for codex.

See the report, summary data and reproduction. Calibration does not apply: calibration record.