What the benchmark measured

WSB live battery — initial results (2026-09-29)

Update (2026-10-06): the native setup was rerun with page fetching working. In the original battery the setup exposed WebSearch and WebFetch but pre-approved only WebSearch, and headless Claude Code refused WebFetch in 40 of 54 native runs (110 calls). The configuration is fixed and pinned by a test. All 54 native runs (18 tasks × 3) were rerun on 2026-10-06 with the same seed, per-task budgets and model, and graded against the historical regrade catalog by a blinded Claude judge; no call was refused. In the regrade table the native row now comes from the rerun: 35/42 competitive runs (83%) at 39.7k tokens per success, against 33/42 (79%) and 36.3k recorded originally. The rerun's 12 expected-fail runs passed 0/12. The initial-grading tables below are the original record and still carry the original native runs. Provider setups were unaffected. Details: agent search behavior.

Pack: web-search-bakeoff (WSB) on the Search Evaluation Workbench (SEW). Agent / model: claude-code, Claude Opus 5.5 (claude-opus-5-5[1m]), authenticated execution on the original host. Status: historical initial grading; headline and Results tables are superseded by the 2026-09-30 regrade, where Firecrawl leads at 90%. The numbers below are graded with SEWBENCH-01 applied to the stored deliverables. The same deliverables graded without those fixes are summarised in Failure dive and were materially wrong.

Headline

Historical figures under the initial grader: rankings and cost lower bounds here apply only to that grading, not the later operator-ruled catalog.

On the 14 competitive production tasks (42 runs per setup), every search setup except Brave scores above answering without search:

• Firecrawl and Perplexity lead at 83%. Exa is at 81%, Tavily 79%, native web search and Parallel 76%, no-search 71% and Brave 69%.
• The 95% intervals overlap. At n=42 per setup the ordering is directional, not conclusive: no setup's difference from no-search or from native has an interval that excludes zero.
• Search matters most on tasks that need live data. On the cloud egress pricing table only Firecrawl and Perplexity went 3/3 (Brave 2/3, the rest 0/3). The AWS pricing pages need JavaScript rendering, and most setups reported the intra-AZ run as unavailable, honestly.
• Among fully priced setups, native search costs least per success ($0.167), followed by Tavily ($0.206). Perplexity (at least $0.148; 37/42 runs priced) and Firecrawl (at least $0.159; 39/42 priced) are lower bounds, excluded from the cost ranking; Perplexity's ask spend is unmetered. Brave and no-search are also partially priced (at least $0.28 each); no-search had a few long runaway answers.
• Brave is last, failing unanswerable-nonexistent-postgres-guc and upstream-diagnosis-node-openssl3-md4 in all three repetitions.

What ran

• Tasks: all 18 production tasks (production task catalog), × 3 repetitions.
  • 14 are competitive.
  • 4 are expected_fail by design: no retrieval surface can answer them, and the honest deliverable is a partial answer or a decline.
• Setups:
  • Provider setups expose one vendor MCP server each, and nothing else.
  • The native setup uses the agent's own web search.
  • The no-search setup has no network tools.
• Pinned MCP servers:
  • @brave/brave-search-mcp-server version 2.1.4
  • tavily-mcp@0.2.22
  • exa-mcp-server@3.4.1
  • firecrawl-mcp@3.26.0
  • @perplexity-ai/mcp-server version 1.3.0
  • Parallel's hosted Search MCP (https://search.parallel.ai/mcp-oauth, authenticated) via mcp-remote@0.14.3
• Grading:
  • Deterministic validators where a task has one.
  • Otherwise a blinded claude-code judge scoring the task's rubric.
  • bakeoff grade --regrade re-scores the stored deliverables. Nothing was re-run.
• Cost: model tokens are priced at the published Opus 5.5 rates. Vendor spend is metered per MCP call where a tariff exists. It is unknown for Exa, Parallel and Perplexity's ask tool, which report no spend over MCP. In the tables, partial is the priced runs' spend divided by all of the setup's successes. Unpriced runs can only add spend, so each partial figure is a lower bound on the full-setup $/success, and these setups are excluded from the cost ranking. bakeoff report divides by the successes among priced runs instead, which gives $0.172 for Perplexity, $0.163 for Firecrawl and $0.29 for Brave on the competitive set, and no figure for Exa or Parallel on the expected-fail set, where every success is in an unpriced run.

Results

All Results tables below use the historical initial grading. The ungraded column counts delivered answers that could not be graded. Runs that exhausted their budget before delivering are counted under budget-exhausted and as failures, rather than under ungraded.

Competitive tasks (14 tasks × 3 repetitions)

Competitive deltas against controls

Each setup against each control over the same 14 tasks × 3 repetitions. Intervals are 95% Newcombe (hybrid Wilson score) intervals for a difference of two independent proportions. They ignore the task pairing, so they are conservative. Token ratios compare mean tokens per run. Provider setups and their controls come from two runs about ten hours apart (see Caveats).

Expected-fail tasks (4 tasks × 3 repetitions)

Passing here means an honest partial answer or decline, which the validator can confirm.

Per task

Findings

1. Search earns its cost on live or long-tail data, not on stable knowledge.

• Tasks whose answers are stable and well known were 3/3 in every setup, including no-search: copyleft obligations, PEP 594 removals, Python feature PEPs and the urllib3 diagnosis.
• The separation comes from the tasks that need current data:
  • the egress pricing table;
  • the SCA tool shortlist (no-search 0/3; Exa, Parallel and Perplexity 3/3);
  • the dockershim change-detection task.

2. Rendering matters for vendor pricing pages. Firecrawl (which renders JavaScript) and Perplexity were the only setups to read AWS's intra-AZ transfer pricing. The others honestly reported it unavailable, and the validator fails an unavailable run. 3. Declining is not the default for search setups. unanswerable-nonexistent-postgres-guc separates the setups: Brave 0/3 and Perplexity 1/3 against 3/3 for native, Tavily and Firecrawl. 4. Exa handles impossible tasks best (5/12 honest partials on the expected-fail set). It is the only setup to go 3/3 on multi-hop-npm-semver-dependents, where the honest move is to state the method and its limits rather than invent a ranking. 5. Provider setups use more reported tokens per run. The MCP provider setups report 42k–65k tokens per run against 15k for no-search. Native search reports 28k. These totals do not separate input from output tokens, which have different prices, and some vendor spend is unknown; the totals alone cannot establish what dominates cost per success.

Failure dive

Two tasks were initially at 0/24 across all setups; a third mostly failed, and the judged SCA task was mostly ungraded. All four exposed bench bugs. Each was confirmed by reading the deliverables, and each is fixed in the bench fix:

With the fixes, the competitive pass rate rose between 14 and 29 points per setup (no-search +14, Exa +29). It rose most for search setups, which had been penalised for citing and explaining evidence. The first reading of this battery, that no-search was about as good as any provider, was an artifact of these bugs.

Genuine failures that remain:

• Egress pricing: unavailable AWS intra-AZ runs, as above.
• change-detection-openapi-30-to-31: most misses omit example→examples from the breaking changes. OAS 3.1 deprecates example rather than removing it. The tables above still require it; the operator has since ruled it is not a break (see the update below).
• Budget exhaustion: 8 runs. Six are Firecrawl or Parallel on long list-building and enumeration tasks; Brave and Tavily have one each.
• By design: the two remaining expected_fail tasks (unversioned-runtime-drift, unindexed-ordering-regression) are 0/3 everywhere.

Codex (partial)

The codex agent (gpt-6-sol) ran one competitive task on 2026-09-28 before its weekly quota ran out:

• Egress pricing: native 2/3, no-search 1/3. Brave and Tavily went 0/3 on the token budget, reading 393k–448k tokens per run.
• Expected-fail tasks: codex's search setups burned the full provider-call budget on multi-hop-npm-semver-dependents instead of declining.

A full codex battery waits for the quota reset (2026-10-04 12:52Z).

Caveats

• Sample size: n=42 per setup on the competitive set. Adjacent setups are within each other's intervals.
• Timing: the two runs were about ten hours apart. Same agent, model, tasks and budgets.
• Pricing: vendor spend is unpriced for Exa and Parallel, and for Perplexity's ask tool. Their $/success counts model tokens plus metered calls only.
• Grading: the judged tasks use a single claude judge, so agreement is not measured.

Update (2026-09-30): operator decisions

• OpenAPI: example→examples is a deprecation, not a break, so the task no longer requires it. The task also stops requiring a construct labelled as the JSON Schema dialect: the dialect has its own field, and complete answers list its consequences instead. Both changes are in the bench fix. On replay, the task goes from 7/24 to 22/24. The two remaining misses cite hosts outside the allowlist.
• Codex native search is priced at Brave's per-request rate as a proxy for its backing vendor. Claude native search keeps Anthropic's published rate.
• Tokens and outcome rate are reported together. Tokens per success come from the bench fix.

Competitive pass rates regraded with the corrected catalog at the recorded grader revision (native: the 2026-10-06 rerun; see the update at the top):

The historical tokens/run column averages measured total-billable tokens over attempted runs with measured usage; unknown or estimated usage is excluded, never treated as zero. The update's tokens/success sums measured tokens over graded runs and divides by measured successes, as defined by the bench fix; it excludes ungraded and unmeasured runs and is not a full-setup lower bound. The per-setup measured coverage counts were not retained here. Multiplying a measured-run mean by all 42 attempts need not reproduce it. The previously quoted fresh/cache/output means are withdrawn: their run coverage and bucket accounting were not recorded here and they do not reconcile for Firecrawl, Parallel or no-search. They cannot support a token-mix comparison from this document.

The output-reduction claim is also withdrawn. The quoted no-search mean included long runaway answers; without a median or an outlier-excluded comparison, this battery does not establish a typical reduction or a causal effect of search on output length. Among the six MCP providers, a higher pass rate goes with fewer tokens per success: Perplexity is best on tokens, and Brave is worst on both. Spearman ρ ≈ −0.83 (pass rate versus tokens/success, n=6) is suggestive, not established, and is not significant at α=0.05. No setup's pass-rate difference from no-search has an interval that excludes zero. Firecrawl comes closest at +14 points (−2 to +30).

WSB methodology

The experiment used Claude Code with claude-opus-5-5[1m], all 18 production tasks, eight tool setups and three repetitions. The two setup groups ran about ten hours apart. Setups differ in exposed search tools: no-search, native, or exactly one provider MCP server. Budgets and contamination checks apply to each run.

The 14 competitive tasks and four expected-fail tasks have separate denominators. Expected-fail success means an honest partial answer or decline. Deterministic validators grade structured tasks; a blinded Claude judge grades rubric tasks. Single-judge agreement was not measured. Pass-rate intervals are Wilson 95%; control deltas use independent-proportion Newcombe intervals, ignoring pairing. Adjacent rankings are not statistically established.

The original bench corrections fixed overly strict date/dollar decline checks, a prose-versus-URL field mismatch and judge JSON parsing/blinding. The later regrade relaxed incorrect OpenAPI breaking-change requirements. Both grade the same stored answers; neither reran agents. REPORT.md retains initial and updated tables with their original precision.

Historical cost lower bounds sum priced spend and divide by all successes. The report command instead uses priced-run successes, so its output can differ. Unknown vendor spend is not zero; partially priced setups are excluded from cost ranking. Updated tokens per success use measured tokens from graded runs divided by measured successes; coverage counts were not retained. Do not multiply the historical tokens/run means by attempted-run counts to derive that update. The output-reduction and fresh/cache/output comparisons were withdrawn.

See recorded results, summary and reproduction. Calibration was not part of this experiment: calibration record.

GAP battery results, 2026-10-03: does search change the outcome of the job?

Update (2026-10-06): the Claude Code native setup was rerun with page fetching working. In the original battery the setup exposed WebSearch and WebFetch but pre-approved only WebSearch, and headless Claude Code refused all 50 WebFetch calls across its 21 runs, so that row measured search without page fetching. The configuration is fixed and pinned by a test. The setup's 21 runs were rerun on 2026-10-06 on the same 7 tasks, repetitions, model and calibration; no call was refused. The Claude Code native row and every figure derived from it below use the rerun: 16/21 passed (76%), gap closure 0.80 (0.53–1.00), 61k tokens per success. The original row recorded 13/21 (62%), gap closure 0.65 (0.37–0.91) and 66k. The other setups are unchanged. The rerun's verdicts come from the claude-code primary judge, which decides every verdict in this battery. On 2026-10-07 the codex judge scored the same 21 stored payloads: it agreed on every verdict and on 94% of labels (kappa 0.70); see judge agreement. The rerun's agents used about 1.0M tokens. The codex native setup was unaffected. Details: agent search behavior.

The first full Search Gap Bench (GAP) battery ran on 2026-10-03, operator-approved: brief tasks on two agents, nine setups each, three repetitions.

• GAP grades the job, not the retrieval.
• A task counts only after calibration shows its knowledge gap: the no-search setup (floor) passes at most 1 of 5, and the setup given the answer excerpt (ceiling) passes at least 4 of 5.
• Each search setup is scored by gap closure: how much of the floor-to-ceiling distance it covers.
• Tokens are reported beside every outcome.

Headline. On these tasks, search decides the outcome.

• The floor passed 0% on both agents, and every search setup passed 71–95%.
• Which provider is best depends on the agent: Parallel led on claude-code and Brave on codex.
• Pass rate did not predict cost: providers with similar pass rates differed almost twofold in tokens per success.

These are the recorded results after the Codex reruns. Claude-code provider tool availability remains unverified in the published evidence: no transcript audit for those runs is documented, and they lack availability records. Its provider rankings are provisional pending that audit. In the first codex pass, 31 of 108 provider runs ran without their search tool. They were re-run on 2026-10-04 under the GAPMCP-01 fix, every re-run run loaded its tool, and the codex tables below include them (see "Execution changes").

Setup

Calibration

Final admissions under GAPRULE-01. Each run shows floor passes and ceiling passes out of 5.

Every floor scored 0, so every task has a real knowledge gap. All rejections are ceiling failures, which point at task or rubric defects:

• spark-editing: the heaviest key fact is not what the prompt asks about.
• python-security-march: the excerpt omitted the date that its top fact requires. The excerpt fix is pending for the next battery; GAPRULE-01 changed the unsupported-claim limit, not the excerpt.
• cli-trust: with three claims, a single unsupported claim fails the brief.
• tarfile-upload on codex: the codex briefs miss key facts.

Gap closure

claude-code (7 tasks, 21 runs per setup)

By family:

• Research briefs: every search setup passed 100%, native included.
• Decision briefs: they separate the providers. Parallel 92% and Firecrawl 83% lead; Brave scored 67%, Perplexity and native 58% each, and Exa and Tavily 50% each.

codex (6 tasks, 18 runs per setup)

No search setup beat native significantly; the largest gain was Brave at +16.7 points (p = 0.25).

By family:

• Research briefs: every setup passed 100%, except Exa at 89%.
• Decision briefs: they separate the providers. Brave scored 89%; Firecrawl and Tavily 78% each; Exa 67%; native 56%; Parallel and Perplexity 44% each.

The first pass's estimates that excluded the 31 tool-less runs were close to the re-run results:

What this shows

1. On these tasks, search changes the outcome. Without search, both agents failed every brief. With a good provider they reached the answer-excerpt setup's level. The difference against the floor is significant for every search setup on both agents (exact McNemar, p ≤ 0.0002). 2. The best provider depends on the agent. On claude-code, Parallel (95%) and Firecrawl (90%) led. On codex, Brave led (94%), ahead of Firecrawl and Tavily (89% each).

• Parallel tied for last on codex (72%), even with its tool loaded in every run. On codex decision briefs it passed 44%, against 92% on claude-code.
• So a provider's ranking on one agent does not carry over to another.

3. Pass rate does not predict cost between providers. Tokens per success counts every agent token in the setup, failed runs included, divided by passing runs.

• The hypothesis that better search shows up as both lower token spend and higher success holds against no search, which spent tokens and passed nothing. On claude-code, Parallel paired the top pass rate (95%) with 64k tokens per success; only native search, at 76%, spent less per success (61k).
• It does not hold as a ranking. Among the seven search setups, the rank correlation between pass rate and tokens per success was +0.05 on claude-code and −0.20 on codex; negative means higher-passing setups cost less. With seven setups, neither value is distinguishable from zero.
• Per-run spend varied widely: 46k (native) to 105k (Firecrawl) on claude-code, and 84k (Perplexity) to 148k (Firecrawl) on codex. Firecrawl passed 90% at 116k tokens per success, against Parallel's 95% at 64k.
• Choosing a provider means weighing pass rate and cost separately.

4. Decision briefs separate the providers; research briefs mostly don't. Research briefs passed at or near 100% on both agents. Decision briefs hinge on one recent fact, and they spread the search setups from 50% to 92% on claude-code and from 44% to 89% on codex. 5. Native search was mid-pack on both agents. On claude-code it passed 76%, level with Perplexity and behind Parallel, Firecrawl and Brave. On codex it passed 78%, level with Exa and behind Brave, Firecrawl and Tavily; under the codex judge it would pass 94% (see judge agreement). No search setup beat it significantly on either agent; on claude-code the largest gain was Parallel's, +19.0 points (p = 0.22).

Judge agreement

Two blinded judges scored every brief on the same evidence: the claude-code judge (the Claude judge below) decides the verdict and the codex judge measures agreement. Both judged at grading time, except for the 21 rerun native runs on claude-code, which codex scored on 2026-10-07 from the payloads the primary had judged.

A Claude-only pass is a disputed verdict the Claude judge passed and codex failed; a codex-only pass is the reverse. Label agreement pools the fact and claim labels both judges scored, and kappa is Cohen's kappa on them. Claims whose cited source could not be captured are left out, as no judge scored them. Briefs that failed schema validation were never judged (2 claude-code and 1 codex floor runs).

The disputes lean towards each judge's own model family, most clearly on codex's briefs: codex passed all 7 disputed ones. On claude-code's briefs the Claude judge passed 4 of the 5 here, but only 2 of the 4 in the replication, 6 of 9 together. The direction differs by agent (two-sided Fisher exact on dispute direction, both batteries' claude-code disputes pooled: p = 0.011). That is consistent with a judge favouring its own family's writing; with two judges the bench cannot say which judge carries the preference, or whether both do. Verdicts stay with the primary judge, as the design fixes in advance. Had codex decided, the pass counts would be:

Only one count moves by more than two runs: codex's native setup, from 14/18 (78%) to 17/18 (94%), which would put it level with Brave and Firecrawl at the top of the codex table. Search against no search is unaffected: no floor brief passes under either judge.

Changes made during the run

The first live runs surfaced several bench defects. Two changed how runs execute, and two changed how runs are graded. The timeline was recorded from the battery change log and run start events.

Execution changes

Every run in the tables above ran under both of these fixes:

• GAPBUDGET-01: brief budgets became runaway caps (1M tokens, 60 calls). The old 30k cap was below the median codex search run.
  • It was part of the pinned export before any calibration or battery run ran.
  • No run ran under the old caps.
• GAPARM-01: brief search setups resolve the GAP catalog. Before it, no brief search setup could run, so no search-setup run predates it.
  • It was applied at 18:30:04Z on 2026-10-03.
  • The earliest search-setup run started at 18:30:29Z (codex round 1). claude-code started at 19:12Z.

The tool-isolation remediation, which forbids the GAP catalog to every GAP run, was applied at 18:50:40Z and the codex battery was restarted to load it.

• 18 codex round-1 runs started before it: classroom-term repetitions 1 and 2, all setups.
• No catalog access in them. Their transcripts contain only search calls and agent messages, with no file reads or commands, so the catalog was not read.
• They were kept, not re-run.
• Every other run ran after the remediation.

The codex provider re-run (2026-10-04). In the first pass, 31 of codex's 108 provider runs made no provider call:

• Cause: GAPMCP-01 reproduced codex's 30-second default MCP startup timeout. A 35-second server fails under it and starts with a 120-second allowance.
• How the runs were selected: the 31 runs were reclassified from an audit of their transcripts. All of them had zero provider calls and a list_mcp_resources probe; 29 of the 31 also said in their answer that no retrieval was available. The private audit is not distributed.
• How they were re-run: with --rerun-unavailable under GAPMCP-01 (a 120-second startup allowance, required=true and pre-warmed pinned packages), using the same matrix, calibration and catalog.
• Every re-run run loaded its tool. Each one recorded a completed initialize and tool listing.
• A second fix was needed. The first re-run attempt misclassified working runs, because the MCP meter could not load its core (SEV2 METERAVAIL, below). Making the metering library available in each provider server's environment fixed it, and those runs were re-run again.

These 31 runs ran under the startup allowance; the other 77 provider runs ran without it but had their tool. No agent-side setting changed.

Grading changes

These changed only how stored answers were judged. Every affected run was regraded from the agents' stored answers, with no agent re-runs:

• GAPJUDGE-01: judges read readable text instead of raw HTML, and each source once. This cut judge tokens per run from about 250k to about 15k. It also covered:
  • safe redirects;
  • gzip decoding (python.org served gzip unconditionally, which garbled every python.org citation);
  • over-budget sources withheld instead of the whole run going ungraded;
  • the blinding leak check reading decoded text, and generic setup labels no longer counted as leaks.
• GAPRULE-01, operator-approved: the unsupported-claim limit was raised from 0.10 to 0.25. Ceiling agents see a short excerpt, and their hedges about it ("the notice does not specify X") were contradicted by the full pages the judges read. All stored grades were recomputed from their saved judge labels; 49 outcomes flipped, all from fail to pass.

Open issues

• SEV2 GAPMCP (fix GAPMCP-01, the bench fix): resolved. The 31 affected codex runs were re-run and the codex table updated.
• SEV2 METERAVAIL (fix METERAVAIL-01): provider-call metering was silently off for this battery on both agents. The meter runs inside the agent's restricted MCP child environment, and from the exported workbench tree it could not load its core.
  • Vendor spend per run was therefore not metered.
  • Only the 31 re-run runs have availability records; the report's "availability coverage" column counts those. Claude-code provider availability has not been verified in the published evidence.
• SEV2 ARGVKEY (fix ARGVKEY-01): the Parallel API key was passed in mcp-remote's process arguments. The battery configuration now passes it through a 0600 header file.
• The claude-code floor setup sometimes writes pretend searches as text when it has no tools. It is a legitimate failure, but slow and expensive (up to about 130k output tokens in 13 minutes).
• Next battery:
  • the code corpus (GAP-07: 16 code tasks and 4 controls);
  • the python-security-march excerpt fix;
  • rubric fixes for spark-editing and cli-trust.

GAP methodology

GAP grades completed jobs against captured primary sources. The initial corpus contained ten briefs: six decision and four research. Calibration is per agent and model: five repetitions of floor (no search) and ceiling (answer excerpt). Admission requires floor at most one pass and ceiling at least four passes. Seven Claude tasks and six Codex tasks were admitted. Code tasks were not run in this battery. The committed calibration file is an admission summary, not a runtime calibration file with captured oracle sources.

Each admitted task ran three times on nine setups. Brief grading requires decision correctness, weighted key-fact recall at least 0.7 and unsupported-claim rate at most 0.25. Two blinded judges used Claude Code as primary and Codex as secondary; the primary decides every verdict and the secondary measures agreement. The 21 Claude Code native runs rerun on 2026-10-06 were graded by the primary alone, and Codex scored their stored payloads on 2026-10-07 (sew gap agree), which rebuilds each payload from the stored source snapshots and checks its SHA-256 first. Gap closure is (setup pass rate − floor pass rate) / (ceiling pass rate − floor pass rate); it can exceed one. Wilson 95% intervals accompany pass rates. Gap-closure intervals use a seeded paired task bootstrap with 10,000 resamples. Exact McNemar tests pair task and repetition, against floor and native. The findings publish floor comparisons and the largest Codex native comparison, not every native test.

Tokens per success sum all agent tokens in an setup, including failures, divided by passes. Judge and calibration spend is separate. A zero-success floor has undefined tokens per success. Vendor spend was unmetered; tokens are not dollar cost.

GAPJUDGE-01 changed readable source extraction, redirects, gzip, source budget handling and blinding. GAPRULE-01 raised the unsupported-claim limit from 0.10 to 0.25; recomputing saved judge labels flipped 49 failures to passes. These grading changes did not rerun agents. Execution changes and the later rerun of 31 Codex runs with unavailable provider tools are preserved separately in REPORT.md.

See recorded results, summary, admission counts and reproduction.

How agents used search: query style, fetch-from-memory and primary-source retrieval (2026-10-05)

This report analyses the stored transcripts of the two published batteries, the web search bakeoff and the search gap bench. No agent was rerun. It asks how agents used the search tools they were given, rather than which provider scored best. Every number below comes from scripts/analyze_search_behavior.py, whose classification rules are listed in methodology. The analysis covers 974 search queries from the GAP battery (both agents) and 655 from the bakeoff (Claude Code).

Headline

• Agents write keyword queries whatever the engine asks for. Of 974 GAP search queries, 7.3% read as natural language. In the bakeoff the figure is 6.0%. Exa's tool asks for a description of the ideal page rather than keywords; its queries were natural language in 3% (Claude Code) and 2% (codex) of GAP calls. Because the automatic rule can miss descriptive phrasing, all 111 Exa queries from both batteries were also read by hand (appendix): none is phrased as a description of a page. The closest are a headline-like phrase used three times ("Python 4.0 release date announced by Python Steering Council") and a page title.
• The two agents express constraints differently. On the six provider setups, codex put site: into 31% to 67% of its queries, even where a structured domain filter existed. Claude Code wrote site: only on Firecrawl (6%) and otherwise used structured domain filters where a tool offered one (Perplexity search_domain_filter, Tavily include_domains, WebSearch allowed_domains).
• Agents often fetch remembered URLs instead of searching. In the bakeoff, Claude Code fetched pages without any search in 36 of 55 Exa runs, 35 of 55 Firecrawl runs, 34 of 54 Tavily runs and 30 of 55 Parallel runs. Those provider setups therefore measured page fetching as much as search. On post-cutoff GAP tasks the same habit failed: Claude Code's Exa runs that only fetched passed 1/3, against 14/18 for its Exa runs that searched.
• Retrieving the primary source mattered most on Claude Code. Pooled over provider and native setups, Claude Code passed 94/111 runs (85%) when the task's primary source appeared in search results and 20/33 (61%) when it did not. Codex surfaced the primary source in most searched runs, so the comparison is thin there (95/116 against 8/10). Two setups passed without the primary page: Parallel (6/7) and Firecrawl (9/9) on Claude Code, which indicates their results carried the facts from other pages.

These are observations about agent behavior with each vendor's MCP server at its pinned version and default settings. They are not measurements of the vendors' APIs.

What each search tool asks for

Descriptions are quoted from the pinned MCP server packages and, for Parallel's hosted server, from its published documentation.

Query style (GAP battery)

Parallel's search_queries are keyword queries by design; its natural-language part is the separate objective field, which is counted as a structured parameter and not as a query. Perplexity's user messages count as queries alongside its query inputs; system and assistant messages are excluded. Codex native search reports its query lists in the transcript; page views are counted as fetches.

Query style (web search bakeoff, Claude Code)

Searching versus fetching from memory (web search bakeoff, Claude Code)

A run is "fetched without searching" when it made at least one fetch call and no search call. Brave's and Perplexity's servers expose no page-fetch tool, so their agents had to search or answer from memory. Every setup has a handful of runs with no provider call: answers given from memory and runs that ended before any tool call.

On the bakeoff's stable-knowledge tasks, fetching a remembered URL is efficient: those tasks passed in every setup, including no-search. It means the bakeoff's provider setups with a fetch tool mostly measured that fetch path. The GAP battery, whose answers post-date the models, shows the cost of the same habit on new information.

Agent refusals: the Claude Code native setup

Claude Code's native setup exposed WebSearch and WebFetch through --tools but pre-approved only WebSearch through --allowedTools. Headless Claude Code refuses a tool that is exposed but not pre-approved ("Claude requested permissions to use WebFetch, but you haven't granted it yet"). In the GAP battery every native WebFetch call was refused (50 calls in 21 of 21 runs); in the bakeoff, 110 calls in 40 of 54 runs. No provider setup and no codex setup had a refused call. The Claude Code native results in both batteries therefore understate that agent's web tools. The setup configuration is fixed and covered by a regression test. The fetch counts in this report include refused attempts. The setup was rerun with fetching working; see Native rerun (2026-10-06).

Primary-source retrieval (GAP battery)

Each GAP brief has one primary-source URL (the catalog oracle). A run surfaced it when that URL appeared among the URLs of returned search results, excluding search inputs. Unknown coverage is excluded from both pass-rate groups. Recomputing against returned evidence leaves the historical source counts unchanged; no searched run has unknown coverage. Facts can also come from secondary pages, so surfacing is a proxy for retrieval, not a requirement for passing.

Native rerun (2026-10-06)

The Claude Code native setup was rerun on both batteries with WebFetch pre-approved: the 21 GAP runs and all 54 bakeoff runs, on the same tasks, repetitions and model. No call was refused. The tables above keep the original runs, because they document the refusal; the GAP and bakeoff reports use the rerun for their native rows.

With page fetching available, the agent searched far less and read pages instead. On GAP it issued 45 queries instead of 101, and it fetched 126 pages. On the bakeoff it issued 19 queries instead of 137, and 36 of its 54 runs fetched remembered pages without searching at all.

The rerun's GAP queries read as natural language in 2.2% of cases (original 7%), with a median of 8 words (original 11). The rerun's run bundles are private, like the originals.

Limits

• Run counts per setup are small (18 to 21 in GAP, 54 to 55 in the bakeoff). Per-setup percentages move by about five points per run.
• The natural-language rule is a heuristic, stated in methodology; readers can rerun the script with their own rule.
• Each vendor was used through its MCP server at the pinned version and default settings. Other interfaces of the same vendor (direct API, other tools, other modes) were not tested.
• The two agents ran different task subsets in GAP (7 tasks for Claude Code, 6 for codex), so cross-agent differences mix agent behavior with task mix.

Appendix: every Exa query

Every query sent to web_search_exa in both batteries, in transcript order, so readers can judge the phrasing directly. Other setups' queries are summarised above but not reproduced.

Agent search behavior: methodology

The analysis reads the stored transcripts of the web search bakeoff (Claude Code, eight setups, 18 tasks, three repetitions) and the search gap bench (Claude Code and codex, brief tasks, three repetitions per admitted task). It runs no agent and changes no grade. Pass and fail come from each run's recorded evaluation.

scripts/analyze_search_behavior.py extracts every completed provider tool call from a transcript:

• codex MCP tool calls, and codex built-in web_search items;
• Claude Code tool_use and tool_result pairs, including vendor MCP tools, WebSearch and WebFetch.

Classification rules

• Search call: a call to a search tool (web_search_exa, brave_web_search, tavily_search, Parallel web_search, firecrawl_search, the Perplexity search tools, Claude Code WebSearch, and codex built-in search actions).
• Query: each entry of a multi-query call (Parallel search_queries, codex built-in query lists) counts as one query. Each user message in Perplexity's conversational inputs also counts as one query; system and assistant messages are excluded. Parallel's objective is counted as a structured parameter when search_queries is present, not as another query.
• Fetch call: a page-retrieval tool call (web_fetch_exa, tavily_extract, firecrawl_scrape, Parallel web_fetch, Claude Code WebFetch, and codex built-in open_page, find_in_page and other page views).
• Natural-language query: ends with "?", starts with an interrogative or auxiliary verb, or contains at least two function words from a fixed list (the, a, an, is, are, was, were, does, do, did, how, what, which, why, when, where, who, that, this, of, to, for, with, will, should, can, about, on, from, by, as, be, has, have).
• Contains a year: a token from 2000 to 2099. Quoted phrase: contains a double quote. site:: contains site:.
• Fetched without searching: the run made at least one fetch call and no search call.
• Primary source surfaced (GAP only): the task's catalog oracle source_url, normalised (scheme, query, fragment and trailing slash removed), appears among the URLs in any returned search result of the run, never in the search input. It is measured only over runs that searched. A run is unknown when any search lacks exposed results and no observed result contains the primary source. Unknown runs are excluded from both surfaced and not-surfaced pass-rate denominators. Recomputing against returned evidence leaves the historical source counts unchanged; no searched run has unknown coverage.

The rules are deliberately simple so they can be audited. The natural-language rule undercounts descriptive keyword-free phrases and overcounts keyword strings that happen to contain function words. Readers can substitute their own rule in the script and rerun it.

Scope

Pooled figures cover the provider and native setups. Reference setups (no-search, floor and ceiling) make no search calls and are excluded from query statistics. GAP run counts differ by agent because calibration admitted seven brief tasks for Claude Code and six for codex.

See the report, summary data and reproduction. Calibration does not apply: calibration record.

GAP replication on Claude Code, 2026-10-06: the search gap bench, run again

Grading note. These verdicts come from the claude-code primary judge, which decides every verdict in GAP. The codex judge, which only measures agreement, was out of quota during the battery and scored the same stored payloads on 2026-10-07. It disputed 4 of the 211 judged verdicts and agreed on 96% of labels (kappa 0.82); no verdict changed. See judge agreement.

A second, independent Search Gap Bench battery on Claude Code (claude-opus-5-5), run on 2026-10-06 and 2026-10-07 with the apparatus as fixed after the 2026-10-03 battery: the built-in search setup can open pages, and every provider setup loaded its tool in every run. It repeats that battery's method on the current task catalog: calibrate each brief, keep the tasks with a real knowledge gap, and run every admitted task on nine setups, three times each.

Headline.

• Search still decides the outcome. Without search the agent passed 0 of 24 runs; every search setup passed 62–83%, and each differs from no search significantly (exact McNemar, p < 0.0001).
• The leader changed. Firecrawl led at 83%. Parallel, which led the 2026-10-03 battery at 95%, passed 75%. No search setup differs significantly from the agent's built-in search (largest gap: Firecrawl, +8.3 points, p = 0.50).
• Built-in search was competitive and cheapest. It passed 75%, level with Parallel and Perplexity, at 55k tokens per success, the lowest of any search setup.
• Pass rate again did not predict cost. The rank correlation between pass rate and tokens per success across the seven search setups was −0.04. Firecrawl's top pass rate came at 141k tokens per success.

Setup

Calibration

Every floor scored 0. Two briefs were rejected for failing the ceiling, as in the 2026-10-03 calibration: cli-trust (ceiling 3/5) and spark-editing (ceiling 1/5). python-security-march is admitted this time (ceiling 5/5); in the 2026-10-03 calibration its answer excerpt omitted the date its top fact requires, and the catalog has since fixed the excerpt.

Gap closure

claude-code (8 tasks, 24 runs per setup)

By family:

• Research briefs: Firecrawl, Perplexity and Exa 92%; Parallel and Built-in 83%; Brave and Tavily 75%.
• Decision briefs: Firecrawl 75%; Parallel, Brave, Exa and Built-in 67%; Perplexity 58%; Tavily 50%.

Against the 2026-10-03 battery

The task set differs by one brief (python-security-march), so the fair comparison is on the seven briefs both batteries admitted. The 2026-10-03 built-in row is its 2026-10-06 rerun with page fetching working.

On the shared briefs no setup's change between batteries is statistically distinguishable (two-sided Fisher exact, smallest p = 0.18); Parallel's drop from 20/21 to 16/21 is the largest. With 21 runs per setup, neighbouring providers cannot be ranked against each other, and a single battery's ordering should not be quoted as a ranking. On the new brief, python-security-march, the setups passed: Exa, Parallel, Firecrawl and Perplexity 2/3 each; Brave and built-in 1/3 each; Tavily 0/3 (ceiling 3/3).

Judge agreement

On 2026-10-07 the codex judge scored every judged brief from the payload the primary judge had scored. Each payload was rebuilt from the run's stored source snapshots, so nothing was captured again, and codex was called only after the rebuilt payload's SHA-256 matched the primary's and the stored labels reproduced the stored verdict. The primary was not rerun. 5 briefs failed schema validation and were never judged; they fail under either judge.

A Claude-only pass is a disputed verdict the Claude judge passed and codex failed; a codex-only pass is the reverse. Label agreement pools the fact and claim labels both judges scored, and kappa is Cohen's kappa on them; claims whose cited source could not be captured are left out, as no judge scored them.

The four disputed verdicts split evenly: the Claude judge alone passed one ceiling and one Parallel brief, and codex alone passed one Exa and one Firecrawl brief. Had codex decided, the battery's pass count would be the same, 147 of 216, and no setup would move by more than one run. Every headline above holds under either judge: the floor passes nothing, every search setup differs from it (exact McNemar, p ≤ 0.0001), Firecrawl still leads at 21/24, and no search setup differs significantly from built-in search (largest gap: Firecrawl, p = 0.25). Agreement is in line with the 2026-10-03 battery on Claude Code's briefs (96% of labels, kappa 0.79). There, 4 of the 5 disputes on Claude Code's briefs went the Claude judge's way; here that lean did not repeat.

What this shows

1. Search changes the outcome, again. Without search the agent failed every brief; with any provider it passed most of them. This finding replicated exactly. 2. Provider rankings did not replicate. The top setup changed and the spread between search setups is within sampling noise at this size. Choose a provider on the agent you use, on cost and on integration, and measure. 3. Built-in search is a serious baseline once it can read pages. Level with the middle of the field at the lowest token cost per success. 4. Pass rate and cost remain separate questions. The cheapest setup per success and the highest-passing setup were different setups in both batteries.

Limits

• One agent and model; the codex half of the 2026-10-03 design was not repeated.
• One judge decides the verdicts. The second judge scored the stored payloads after the battery rather than alongside it; the evidence it saw is identical, checked by payload hash.
• 24 runs per setup: 95% intervals span roughly 30 points.
• Provider indexes, sources and models drift between batteries; a replication tests the method, not a fixed truth.

GAP replication methodology

This battery repeats the 2026-10-03 GAP methodology on one agent: calibration per task (five floor and five ceiling repetitions; admission requires the floor to pass at most once and the ceiling at least four times), then every admitted task three times on nine setups.

Differences from the 2026-10-03 battery:

• Catalog. The current GAP catalog (SHA-256 29f8fd1731cd0155f2a993688d8f1dfac7ad6ef1f6effc549185af026ad24443), in which python-security-march's answer excerpt carries the date its top fact needs. 8 of the 10 briefs were admitted.
• Agent. Claude Code on claude-opus-5-5 only.
• Built-in search. The native setup pre-approves WebFetch as well as WebSearch, so the agent can open pages.
• Provider tools. Every provider setup made provider calls in every run; Parallel's credential is passed as a header file the workbench writes per run.
• Judging. The claude-code primary judge decides every verdict, as before. The codex judge, which measures agreement, was out of quota during the battery. It scored the same stored payloads on 2026-10-07 with sew gap agree, which rebuilds each payload from the stored source snapshots and calls codex only when the payload's SHA-256 matches the primary's and the stored labels reproduce the verdict.

Statistics are unchanged: Wilson 95% intervals on pass rates, a seeded paired task bootstrap (10,000 resamples) for gap closure, exact McNemar tests against the floor and the built-in setup, and tokens per success counted over all agent tokens in the setup, failures included. The comparison with the 2026-10-03 battery uses two-sided Fisher exact tests on the seven shared briefs, because the two batteries' runs are not paired.

See recorded results, summary, admission counts and reproduction.

Search API head-to-head: Exa, Tavily, Parallel and Firecrawl on a research workload (2026-09-26)

This study called four search APIs directly, not through a coding agent. It asked how much verified new information each adds to a research knowledge base that had already been built with an agent's built-in search tools until they stopped finding new facts, and what each provider surfaces that the others don't. It was pre-registered and cost-capped, results were pooled and graded blind, and every counted claim was verified on its source page. Stages A and B ran on 2026-09-26, Stage C on 2026-09-27, and an Exa-only Monitors probe ran daily from 2026-09-27 to 2026-10-06. The full record (design, pre-registration with twelve amendments, judge briefs and stage write-ups), the code and the sanitized data are published in this directory.

Context. The experiment was run while building a go-to-market knowledge base about Exa, one of the four providers tested. Its questions come from that knowledge base's open questions about Exa's market and competitors, so the workload reflects what that knowledge base needed. The design added workloads where Exa might be weak on purpose (freshness, and pages that need rendering). Weigh the results with that in mind; the record, code and data are here so readers can check them.

Headline

• On top of built-in search, Exa added the most verified new information. On 40 questions the knowledge base couldn't answer, verified new or correcting claims under strict credit: Exa 25, Parallel 19, Tavily 11, Firecrawl 10, with 11 of Exa's found by no other provider. On two weeks of company news: Exa 20 new dated facts, Tavily 12, Parallel 3, Firecrawl 2. Building three entity lists: Exa returned 28 entities new to the knowledge base (21 that no other provider returned), Firecrawl 19 and Parallel 7.
• The other providers led elsewhere. On pages the built-in tools couldn't read, Firecrawl recovered 14 of 15, Parallel 12, Tavily 10 and Exa 5. Filling a 20-company, eight-field schema, Parallel was right on 113 of 133 ground-truth cells, Exa on 105 and Firecrawl on 94; the Parallel–Exa gap is not significant, at the same price.
• Finding a source the knowledge base already cites was a tie: 16 to 19 of 20 for all four, on identical inputs.
• On evidence about four big grounding vendors (Stage C), no lead is significant. Verified new tier-1 claims across a search arm and a research arm, after the post-hoc corrections of Amendment 12: Exa 16, Parallel 14, Tavily 13, Firecrawl 9. Parallel's research agent was the most accurate: 0.6% of its checkable items were unsupported or false, against Exa's 4.6%, Firecrawl's 2.7% and Tavily's 18.1%.
• Exa's Monitors caught real competitor changes, with some noise. Over 49 daily runs on five competitors' pricing and changelog pages, they reported 20 changes: 14 confirmed, 5 false positives. 13 were real, dated pricing or product changes, 12 of them missing from the knowledge base or only partly recorded there.
• One provider for everything leaves gaps. For this workload the record recommends Exa for discovery, list building and news sweeps; Firecrawl or Parallel for pages that need rendering; and Parallel or Exa for filling a fixed schema.

What was tested

Each provider ran with parameters taken from its own best-practice documentation and frozen before the first call, on the same day, with the same top 10. The design is a capability matrix rather than four copies of every job: Stage A's cheap probes ran on all four providers, and Stage B's list and schema sets ran only on providers that sell those capabilities. Tavily sat out S5 and S6 because it sells no list-building or structured-extraction product. See methodology for the endpoints, grading and amendments.

Stage A: four providers, cheap probes

Strict credit counts a claim only when the provider's own result page states it (Amendment 4). S2 was re-run with identical inputs after a review found that hand-written queries leaked answer terms. The Stage A write-up has broad credit, the first grading round and the diagnostics.

Stage B: lists, schema fill and find-similar

S5 counts entities verified against the list's criteria and new to the knowledge base, under the fact-level rule; the stricter entity-level rule gives Exa 27, Firecrawl 18 and Parallel 3, in the same order. S6 compares cells with the knowledge base's verified ground truth. Parallel's spend is at list price; Firecrawl's credits are shown as a range across plans. The Stage B write-up has each list, each field and the review fixes (Amendments 6 and 7).

Find-similar (SP, Exa only) is a cheap sweep for long-tail competitors, but it needs triage: 50 of the 61 new domains were directories, profiles, clones or unrelated sites. It was graded, but not blind, since only Exa ran.

Stage C: evidence on the big grounding vendors

Twenty-four pre-registered questions asked for evidence on four supply lines a market sizing couldn't anchor: Google grounding, Anthropic's and OpenAI's web search tools, and Microsoft Grounding with Bing. Each provider ran a search arm and a research-agent arm, and eight blind judges verified every claim on its page. A tier-1 claim is a figure that enters a line's arithmetic: a volume, a revenue figure or share, a denominator, or a price. After a review, Amendment 12 (post hoc) added a claim-level correction layer for novelty, tiers, duplicates and Tavily's cap-refused runs; the Stage C write-up shows the as-judged figures beside the corrected ones.

By arm, corrected. The research error rate is the share of a research agent's checkable items whose cited page didn't state them or contradicted them.

Is the lead real? Exact two-sided sign tests over the 24 questions, chosen after the results were known, with Holm's adjustment for the six comparisons and question-level bootstraps:

Monitors probe (SM)

Five Exa Monitors watched competitors' pricing and changelog pages once a day. The pre-registered question: does a monitor surface a real, dated pricing or product change on the competitor's own page within the knowledge base's 30-day staleness window, and was the change already in the knowledge base?

• Deviation from the plan. The probe was planned to run to 2026-10-10 and then delete its monitors. It was harvested and the monitors paused on 2026-10-06, four days early, so it covers 49 daily runs rather than about 70. The monitors were paused, not deleted.
• What the monitors reported. 14 of 49 runs reported at least one change, 20 changes in all. Monitors report only what changed since their previous run.
• Graded result. Of the 20 reported changes, 14 were confirmed against the vendor's page and archived snapshots, 5 were false positives and 1 could not be settled either way. 13 were real, dated pricing or product changes, and 12 of those were missing from the knowledge base or only partly recorded there. 3 of 5 monitors answered the pre-registered question yes: Parallel, Firecrawl and Perplexity API.
• Where it worked and where it didn't. Perplexity's monitor was the strongest: its changelog carries exact timestamps, and every confirmed change fell inside its run window. Firecrawl's monitor reported a security feature added, removed and added again on its pricing page; the page was unchanged in every archived capture, so those reports are noise. Both of Brave's reports were false positives: prices were unchanged, and the watched pages had started serving a version without prices. Tavily's monitor reported nothing, and its changelog had nothing new.
• Misses were not measured. The grader noticed one: a Firecrawl changelog entry from 2026-09-29 was never reported. Per-change verdicts, evidence links and notes are in the grading record and its write-up.
• How it was graded. One Claude agent, not blind (only Exa ran), checked each reported change against the vendor's current page and Internet Archive snapshots from before and after the run, and searched the knowledge base for it.

Spend

Stages A and B together cost at most $23.29 of the $40 cap, valuing credits at the most they cost (Stage A $5.07, Stage B $18.22). Stage C's costs are in its provider table above; Exa's are billed, Parallel's are list-price estimates and Tavily's and Firecrawl's credits are ranges across plans. The Monitors probe made 49 runs, at $15 per 1,000 runs on Exa's price list.

Read this before comparing providers

• Small samples, one run. Each item ran once on one day. The web and the indexes change, so a re-run will differ.
• Model judges. Claude agents graded blind to the provider and verified every counted claim on its page; no human graded the sets.
• Post-hoc changes are logged, not hidden. Amendments 4, 6, 7 and 12 changed scoring after results were seen. The record keeps the earlier figures beside the corrected ones, and the pre-registration lists every amendment.
• New means new to one knowledge base. Novelty is measured against the knowledge base the questions came from.
• Direct APIs, tuned parameters. Unlike the agent benchmarks, where each provider ran through its MCP server at default settings, this study called each API directly with its best-practice parameters. Results do not transfer between the two setups.
• Vendor terms. The experiment was first run as an internal evaluation. The record's overview summarises each vendor's terms on benchmarking as read on 2026-09-26 (terms).
• What is and isn't published. For Stages A to C, page text, titles, snippets, vendor-written values and raw responses are not published, and social-media URLs are hashed. Those raw responses were deleted after grading (Amendment 7), so their grading cannot be replayed exactly. Every published table rebuilds from data/ except Stage A's S3 rows, which need page text; their per-page outcomes are in the scripted summary. The Monitors probe is the exception: by the owner's decision of 2026-10-06, its raw API responses, including Exa's change titles, summaries and run notes, are published verbatim in data/stage-b/runs/SM/raw/ (each monitor's webhook secret was never stored), and its sanitized record rebuilds from them byte for byte.

Record and data

• The experiment's overview, design and pre-registration with Amendments 1–12.
• Judge briefs: first round, re-grade, Stage B and Stage C.
• Stage write-ups: Stage A, Stage B and Stage C, each with every table and the verified findings.
• Code: the Stage A harness code/harness.py, Stage B code/harness_b.py, Stage C code/stage_c/harness_research.py, the exporter code/export_data.py and the Monitors harvest code/monitors_harvest.py.
• Scores: Stage A, Stage B, Stage C corrected and as judged; the Monitors harvest, raw responses and grading.

Search API head-to-head: methodology

This study calls search APIs directly; no coding agent is involved. Its design, query sets, parameters and metrics were frozen in the design and the pre-registration before the first paid call, with SHA-256 hashes of the frozen files. Twelve amendments follow, each dated and marked as made before or after results were seen. This page summarises the method; the record is authoritative.

Providers and endpoints

Every provider's parameters come from its own best-practice documentation, read on 2026-09-26, and are written into the harness configs (code/config.json, code/config_b.json, code/stage_c/config.json and config_c2.json). Every provider got the same top 10 and ran on the same day.

• Exa: /search (auto, highlights) for S1 and S2; /contents with a live crawl for S3; the news category with a date filter for S4; the Agent API for S5 (capped) and S6 (with an output schema); find-similar for SP; Monitors for SM; and the Agent API at medium effort for Stage C's research arm.
• Tavily: search (advanced, chunks); extract for S3; news with a time range for S4; the Research API (mini) for Stage C's research arm. Tavily sat out S5 and S6: it sells no list-building or structured-extraction product, and search plus our own model would measure our model, not Tavily.
• Parallel: search (advanced, with an objective); extract for S3; after_date for S4; FindAll for S5; Task with an output schema for S6; Task (pro) for Stage C's research arm.
• Firecrawl: search plus a scrape of the top 3; scrape with JavaScript rendering for S3; tbs for S4; /agent for S5 and S6 (with a schema); /agent capped at 100 credits for Stage C's research arm.

Sets

The query sets are drawn from the knowledge base's open questions and claims (see the design): S1 40 known unknowns, S2 20 known knowns, S3 15 blocked pages, S4 10 companies' last two weeks, S5 three entity lists, S6 20 companies by eight fields, SP 10 competitor homepages, SM 5 monitors and SV 24 questions on four grounding vendors. Their hashes are pinned in code/sets/SHA256SUMS and match the prefixes frozen in the pre-registration.

Grading

• Pooled and blind. Each question's results are pooled across providers, deduplicated and, from Amendment 4, shuffled with a recorded seed. Judges never see provider names; attribution is rejoined only after judging.
• Verified on the page. Judges are Claude agents working from written briefs (first round, re-grade, Stage B, Stage C). Every counted claim must be stated on its source page.
• Strict per-claim credit (Amendment 4): a provider is credited with a claim only when its own result page states it. Broad credit, where any pooled page may support the claim, is reported in the stage write-ups.
• Novelty is judged against the knowledge base at a recorded baseline commit. Claims are weighted for usefulness: 3 corrects a knowledge-base claim, 2 is a new fact for a page, 1 is a new corroborating source and 0 is trivia.
• Scripted sets. S2 counts whether the cited primary source is in the top 10. S3 counts a page as recovered when the fetch returns at least 500 characters and a target keyword, with one recorded correction (a redirect).
• S5 and S6. An S5 entity counts when it meets every criterion of its list and is new to the knowledge base (a fact-level and a stricter entity-level rule are both reported). S6 cells are compared with the knowledge base's verified ground truth where it has one; the false-claim rate is the share of filled cells that are wrong.
• Stage C. Eight blind judges. A tier-1 claim enters a sizing line's arithmetic. The research error rate is the share of a research agent's checkable items whose cited page didn't state or contradicted them. Amendment 12 (post hoc) adds a claim-level correction layer (data/stage-c/corrections.json): novelty against the knowledge base's working notes, tiers, duplicates across questions, and scoring Tavily's four cap-refused runs as returning nothing, as pre-registered. Sign tests, Holm's adjustment and bootstraps were chosen after the results were known.
• Monitors (SM). One Claude agent, not blind (only Exa ran), checked each reported change against the vendor's current page and Internet Archive snapshots from before and after the run, and searched the knowledge base for it. A monitor passes when at least one change is confirmed, is a pricing or product change, and is dated inside the 30 days before the run.

Gates, caps and stop rules

S1 ran on 20 questions first and would have stopped there had no provider resolved at least 3 that the built-in tools couldn't. Stage B ran only for providers and workloads that showed signal in Stage A. Each provider had a hard cap, and the owner set a $40 cap for Stages A and B together (Amendment 5); Stage C had its own caps (Amendment 8).

Amendments that changed scoring after results were seen

• Amendment 4: after a review, Stage A was re-graded with shuffled pools and strict per-claim credit, and S2 was re-run on identical inputs.
• Amendment 6: Stage B scoring choices, written after unblinding.
• Amendment 7: fixes from the Stage B review, and the per-file allowlist for exported data.
• Amendment 12: the Stage C correction layer described above.

The record keeps the earlier figures beside the corrected ones in each case.

Costs

Exa's costs are billed. Parallel's are list-price estimates from the harness config. Tavily's and Firecrawl's credits are shown as ranges across their plans. Stage A and B costs value credits at the most they cost, as the cap guard did.

Published data

code/export_data.py writes the data through an allowlist. It keeps request parameters, status, latency, cost, each result's rank, URL, domain, published date and text length, crawl status codes, the pools (URLs only), attribution maps, judge verdicts and scores. It never exports page titles, snippets, page text, extracted content, vendor-written values or raw responses, and replaces the paths of social-media profile and post URLs with a stable hash. Judge notes may quote short fragments of pages. The Monitors export (code/monitors_harvest.py) follows the same rule: change types and fingerprints are kept, change summaries and titles only as lengths. The Monitors probe is the one exception to the no-vendor-text rule: by the owner's decision of 2026-10-06, its raw API responses, including Exa's change titles, summaries and run notes, are published verbatim in data/stage-b/runs/SM/raw/.

See the report, summary data and reproduction. Calibration does not apply: calibration record.