Searchlight

Search behavior · 2026-10-05

How agents used search

In ten seconds

Only 7% of searches about recent events read as plain questions; Claude Code passed 85% of briefs when search found the official page (release notes, advisory or docs), 61% without.

7%

of recent-event searches read as plain questions.

36 of 55

Claude Code Exa runs that fetched remembered pages instead of searching.

85% vs 61%

Claude Code passes, official page found vs not.

For your agent: Find the official page (a pattern, not a cause).

The full report

How agents used search: query style, fetch-from-memory and primary-source retrieval (2026-10-05)

This report analyses the stored transcripts of the two published batteries, the web search bakeoff and the search gap bench. No agent was rerun. It asks how agents used the search tools they were given, rather than which provider scored best. Every number below comes from scripts/analyze_search_behavior.py, whose classification rules are listed in methodology. The analysis covers 974 search queries from the GAP battery (both agents) and 655 from the bakeoff (Claude Code).

Headline

  • Agents write keyword queries whatever the engine asks for. Of 974 GAP search queries, 7.3% read as natural language. In the bakeoff the figure is 6.0%. Exa's tool asks for a description of the ideal page rather than keywords; its queries were natural language in 3% (Claude Code) and 2% (codex) of GAP calls. Because the automatic rule can miss descriptive phrasing, all 111 Exa queries from both batteries were also read by hand (appendix): none is phrased as a description of a page. The closest are a headline-like phrase used three times ("Python 4.0 release date announced by Python Steering Council") and a page title.
  • The two agents express constraints differently. On the six provider setups, codex put site: into 31% to 67% of its queries, even where a structured domain filter existed. Claude Code wrote site: only on Firecrawl (6%) and otherwise used structured domain filters where a tool offered one (Perplexity search_domain_filter, Tavily include_domains, WebSearch allowed_domains).
  • Agents often fetch remembered URLs instead of searching. In the bakeoff, Claude Code fetched pages without any search in 36 of 55 Exa runs, 35 of 55 Firecrawl runs, 34 of 54 Tavily runs and 30 of 55 Parallel runs. Those provider setups therefore measured page fetching as much as search. On post-cutoff GAP tasks the same habit failed: Claude Code's Exa runs that only fetched passed 1/3, against 14/18 for its Exa runs that searched.
  • Retrieving the primary source mattered most on Claude Code. Pooled over provider and native setups, Claude Code passed 94/111 runs (85%) when the task's primary source appeared in search results and 20/33 (61%) when it did not. Codex surfaced the primary source in most searched runs, so the comparison is thin there (95/116 against 8/10). Two setups passed without the primary page: Parallel (6/7) and Firecrawl (9/9) on Claude Code, which indicates their results carried the facts from other pages.

These are observations about agent behavior with each vendor's MCP server at its pinned version and default settings. They are not measurements of the vendors' APIs.

What each search tool asks for

Descriptions are quoted from the pinned MCP server packages and, for Parallel's hosted server, from its published documentation.

SetupSearch interface the agent sawWhat the tool tells the agent about queries
Exaweb_search_exa(query, numResults)"describe the ideal page, not keywords"; query is a "semantically rich description of the ideal page, not just keywords"
Parallelweb_search(objective, search_queries, …)a natural-language objective plus "concise, related keyword queries of 3–6 words each"
Perplexityperplexity_search(query, …) plus ask, research, reason"Supports recency filters, domain restrictions, and a lower-latency fast search mode"
Tavilytavily_search(query, search_depth, …)query is described only as "Search query"
Firecrawlfirecrawl_search(query, …)"Query operators, domain filters, categories … are described on their parameters"
Bravebrave_web_search(query, …)lists when to use the tool; no guidance on query phrasing
native (Claude Code)WebSearch(query, allowed_domains, …)agent built-in
native (codex)built-in web searchagent built-in; no parameters exposed to the transcript

Query style (GAP battery)

Parallel's search_queries are keyword queries by design; its natural-language part is the separate objective field, which is counted as a structured parameter and not as a query. Perplexity's user messages count as queries alongside its query inputs; system and assistant messages are excluded. Codex native search reports its query lists in the transcript; page views are counted as fetches.

AgentSetupQueriesNatural languageContains a yearQuoted phrasesite:Median wordsStructured parameters passed (calls)
claude-codenative1017%43%39%0%11allowed_domains ×11
claude-codeBrave7217%44%18%0%8count ×23, extra_snippets ×21, maximum_number_of_tokens ×18, freshness ×1, maximum_number_of_urls ×1
claude-codeTavily3813%45%18%0%9max_results ×38, include_domains ×19, search_depth ×17, include_raw_content ×1, start_date ×1
claude-codeExa383%58%0%0%10numResults ×38
claude-codeParallel945%24%2%0%6model_name ×27, objective ×27, session_id ×27
claude-codeFirecrawl340%59%9%6%7limit ×34, sources ×34, includeDomains ×3
claude-codePerplexity7020%36%10%0%9max_results ×60, search_domain_filter ×42, max_tokens_per_page ×5, messages ×5, search_context_size ×3, search_recency_filter ×2, search_type ×1
codexnative480%50%33%10%8none
codexBrave1142%52%26%46%8maximum_number_of_tokens ×30, count ×15, maximum_number_of_urls ×13, extra_snippets ×7, url ×3
codexTavily1306%50%59%67%8max_results ×123, search_depth ×26, include_raw_content ×25, end_date ×3, include_domains ×3
codexExa522%60%35%58%10numResults ×10
codexParallel626%39%27%48%7objective ×28, session_id ×10, max_results ×3, max_chars_per_result ×1
codexFirecrawl422%74%26%31%8limit ×33, sources ×10
codexPerplexity7914%71%33%42%10max_results ×20, messages ×18, max_tokens_per_page ×16

Query style (web search bakeoff, Claude Code)

SetupRunsQueriesNatural languageContains a yearQuoted phrasesite:Median wordsStructured parameters passed (calls)
native541372%34%9%4%7allowed_domains ×36
Brave541994%7%11%6%6count ×85, maximum_number_of_tokens ×80, freshness ×17, goggles ×16, extra_snippets ×11, maximum_number_of_urls ×9, maximum_number_of_tokens_per_url ×3
Tavily54250%32%12%0%10max_results ×18, include_domains ×10, search_depth ×4, time_range ×2, start_date ×1
Exa55210%19%5%0%10numResults ×21
Parallel55770%20%0%1%5model_name ×22, objective ×22, session_id ×22
Firecrawl55200%25%15%5%7limit ×20, sources ×20
Perplexity5517616%13%10%0%8max_results ×154, search_domain_filter ×115, max_tokens_per_page ×21, search_type ×21, messages ×13, search_context_size ×13, search_recency_filter ×5

Searching versus fetching from memory (web search bakeoff, Claude Code)

A run is "fetched without searching" when it made at least one fetch call and no search call. Brave's and Perplexity's servers expose no page-fetch tool, so their agents had to search or answer from memory. Every setup has a handful of runs with no provider call: answers given from memory and runs that ended before any tool call.

SetupRunsFetch tool in the setupFetched without searchingNo provider callCalls refused by the agent
native54WebFetch (mostly refused; see below)126110
Brave54no070
Tavily54yes (tavily_extract)3460
Exa55yes (web_fetch_exa)3660
Parallel55yes (web_fetch)3060
Firecrawl55yes (firecrawl_scrape)3560
Perplexity55no070

On the bakeoff's stable-knowledge tasks, fetching a remembered URL is efficient: those tasks passed in every setup, including no-search. It means the bakeoff's provider setups with a fetch tool mostly measured that fetch path. The GAP battery, whose answers post-date the models, shows the cost of the same habit on new information.

Agent refusals: the Claude Code native setup

Claude Code's native setup exposed WebSearch and WebFetch through --tools but pre-approved only WebSearch through --allowedTools. Headless Claude Code refuses a tool that is exposed but not pre-approved ("Claude requested permissions to use WebFetch, but you haven't granted it yet"). In the GAP battery every native WebFetch call was refused (50 calls in 21 of 21 runs); in the bakeoff, 110 calls in 40 of 54 runs. No provider setup and no codex setup had a refused call. The Claude Code native results in both batteries therefore understate that agent's web tools. The setup configuration is fixed and covered by a regression test. The fetch counts in this report include refused attempts. The setup was rerun with fetching working; see Native rerun (2026-10-06).

Primary-source retrieval (GAP battery)

Each GAP brief has one primary-source URL (the catalog oracle). A run surfaced it when that URL appeared among the URLs of returned search results, excluding search inputs. Unknown coverage is excluded from both pass-rate groups. Recomputing against returned evidence leaves the historical source counts unchanged; no searched run has unknown coverage. Facts can also come from secondary pages, so surfacing is a proxy for retrieval, not a requirement for passing.

AgentSetupRuns that searchedUnknown coveragePrimary source surfacedPass when surfacedPass when not surfaced
claude-codenative21018/2113/180/3
claude-codeBrave21018/2115/182/3
claude-codeTavily21016/2113/162/5
claude-codeExa18014/1813/141/4
claude-codeParallel21014/2114/146/7
claude-codeFirecrawl21012/2110/129/9
claude-codePerplexity21019/2116/190/2
codexnative18017/1813/171/1
codexBrave18016/1815/162/2
codexTavily18016/1814/162/2
codexExa18017/1814/170/1
codexParallel18017/1812/171/1
codexFirecrawl18015/1814/152/3
codexPerplexity18018/1813/18—

Native rerun (2026-10-06)

The Claude Code native setup was rerun on both batteries with WebFetch pre-approved: the 21 GAP runs and all 54 bakeoff runs, on the same tasks, repetitions and model. No call was refused. The tables above keep the original runs, because they document the refusal; the GAP and bakeoff reports use the rerun for their native rows.

With page fetching available, the agent searched far less and read pages instead. On GAP it issued 45 queries instead of 101, and it fetched 126 pages. On the bakeoff it issued 19 queries instead of 137, and 36 of its 54 runs fetched remembered pages without searching at all.

BatteryRunsQueries (original)Fetch callsRefused calls (original)Fetched without searchingPrimary source surfacedPass when surfacedPass when not surfaced
GAP2145 (101)1260 (50)017/2115/171/4
Bakeoff5419 (137)4260 (110)36———

The rerun's GAP queries read as natural language in 2.2% of cases (original 7%), with a median of 8 words (original 11). The rerun's run bundles are private, like the originals.

Limits

  • Run counts per setup are small (18 to 21 in GAP, 54 to 55 in the bakeoff). Per-setup percentages move by about five points per run.
  • The natural-language rule is a heuristic, stated in methodology; readers can rerun the script with their own rule.
  • Each vendor was used through its MCP server at the pinned version and default settings. Other interfaces of the same vendor (direct API, other tools, other modes) were not tested.
  • The two agents ran different task subsets in GAP (7 tasks for Claude Code, 6 for codex), so cross-agent differences mix agent behavior with task mix.

Appendix: every Exa query

Every query sent to web_search_exa in both batteries, in transcript order, so readers can judge the phrasing directly. Other setups' queries are summarised above but not reproduced.

BatteryAgentQuery
GAPclaude-codeCopilot Billing Preview app deprecated usage report
GAPclaude-codeGitHub Copilot premium request budgets per user cap and usage report CSV export docs
GAPclaude-codeGitHub docs AI usage page billing settings group filter export AI credits Copilot limitations
GAPclaude-codeGitHub Copilot Billing Preview app deprecated usage-based billing budgets
GAPclaude-codeGitHub docs Copilot premium requests usage report export CSV billing 2026
GAPclaude-codeGitHub docs AI usage page billing settings group filter export AI credits usage report limitations
GAPclaude-codeGitHub Copilot Billing Preview app deprecated usage report
GAPclaude-codeGitHub docs set budget for individual Copilot premium requests user-level budget
GAPclaude-codeGitHub docs AI usage page billing settings group filter export AI credits Copilot usage report limitations
GAPclaude-codegithub.blog changelog 2026-08-04 Retiring the Copilot Billing Preview app
GAPclaude-codeGitHub Classroom deprecation sunset announcement 2026
GAPclaude-codeClassroom 50 CS50 foundation GitHub Classroom alternative import autograding LTI
GAPclaude-codeCodio GitHub Classroom migration partner
GAPclaude-codeGitHub Classroom sunset deprecation announcement 2026
GAPclaude-codeClassroom 50 CS50 open source GitHub Classroom replacement import migration
GAPclaude-codeGitHub Classroom deprecation sunset announcement 2026
GAPclaude-codeGitHub Models deprecation retirement changelog 2026 BYOK inference
GAPclaude-codeGitHub Models deprecation retirement changelog 2026 BYOK paid usage
GAPclaude-codeAzure AI Inference beta SDK deprecated retirement August 26 2026 migrate to OpenAI v1 API
GAPclaude-codeGitHub Models deprecation retirement changelog 2026 BYOK
GAPclaude-codeGitHub secret scanning npm token revocation credential types 2026 changelog
GAPclaude-codeGitHub changelog npm granular access tokens publish and manage package access 2026
GAPclaude-codeGitHub secret scanning npm token revocation credential types in scope 2026
GAPclaude-codeGitHub changelog npm granular access tokens bypass 2FA publish manage package access 2026
GAPclaude-codeGitHub changelog npm granular access tokens bypass 2FA publish package access management 2026
GAPclaude-codeGitHub secret scanning npm token revocation credential types 2026 changelog
GAPclaude-codeGitHub changelog npm staged publishing generally available npm stage publish 2026
GAPclaude-codePython 3.10 security release tarfile extraction filter bypass CVE-2025-4517
GAPclaude-codeCPython tarfile data filter vulnerability 2026 CVE security-announce
GAPclaude-codePython 3.10 security release tarfile extraction filter bypass fix
GAPclaude-codeCVE-2026-4360 tarfile extract filter hardlinks Python fixed version
GAPclaude-codeGitHub docs converting a user into an organization warning cannot sign in personal account
GAPclaude-codeGitHub Enterprise Server docs promoting or demoting site administrator converting user to organization
GAPclaude-codepython.org release metadata API authentication vulnerability security advisory 2026
GAPclaude-codepython.org release metadata vulnerability unauthenticated API modify release files 2026
GAPclaude-codepython.org release API authentication bypass DEVCORE Splitline CVE advisory exploitation
GAPclaude-codepython.org release metadata vulnerability unauthenticated modify release files advisory 2026
GAPclaude-codePython Software Foundation security incident python.org downloads release API authentication
GAPcodexGitHub Classroom sunset July 2026 September 2026 partner repositories
GAPcodexGitHub Classroom retirement September 1 2026 repositories partner
GAPcodexsite:github.com/orgs/community/discussions/205975 "September 4"
GAPcodexGitHub Classroom sunset September 2026 repositories partner retirement
GAPcodexsite:github.com/orgs/community/discussions GitHub Classroom "September 4" "2026"
GAPcodexGitHub Models inference BYOK retirement July August 2026 paid existing usage
GAPcodexGitHub Models retiring July August 2026 paid inference BYOK July 10
GAPcodexGitHub Models retirement July August 2026 paid inference BYOK existing usage
GAPcodexsite:docs.github.com converting user into organization January 2026 transfer repositories personal account enterprise server
GAPcodexsite:github.blog/changelog 2026 January converting user organization disabled January 2026
GAPcodexsite:docs.github.com converting a user into an organization January 2026 transfer repositories
GAPcodexsite:github.blog/changelog 2026 January converting personal accounts organizations January 20
GAPcodexsite:docs.github.com converting user into organization January 2026 transfer repositories personal account GHES
GAPcodexsite:github.blog/changelog 2026 January 20 convert account organization transfer personal account
GAPcodexGitHub Copilot Billing Preview app retired individual budgets usage export billing reports
GAPcodexsite:docs.github.com Copilot AI usage page reports limitations billing API user level budgets export usage credits
GAPcodexsite:docs.github.com "AI usage" "report" "does not" billing usage reports
GAPcodexsite:docs.github.com "Setting up budgets" "user-level" "Budgets and alerts"
GAPcodex"Copilot Billing Preview" budgets export usage
GAPcodexsite:docs.github.com AI usage user-level budgets billing reports Copilot AI credits
GAPcodexsite:docs.github.com "AI usage" "does not" reports billing
GAPcodexsite:docs.github.com "Viewing your usage" "AI" "report" billing
GAPcodexsite:docs.github.com "Automating usage reporting" "AI"
GAPcodexsite:docs.github.com Copilot usage metrics billing data coverage IDE chat github CLI
GAPcodexsite:docs.github.com "Setting" "user-level budgets" "Budgets"
GAPcodexCopilot Billing Preview app individual budgets export raw usage finance billing dashboard
GAPcodexsite.github.blog/changelog 2026-08-04 retiring copilot billing preview
GAPcodexsite:docs.github.com AI usage page billing reports user-level budgets coverage limitations Copilot
GAPcodexsite:docs.github.com "AI usage" "does not" reports billing
GAPcodexsite:docs.github.com "AI usage" "coverage"
GAPcodexsite:docs.github.com "Viewing your usage of metered products" "AI usage"
GAPcodexsite:github.blog/changelog "user-level budgets" 2026
GAPcodexsite:docs.github.com copilot usage metrics "billing" "coverage"
GAPcodexsite:github.blog/changelog billing CSV usage reports API 2026
GAPcodexnpm token security granular tokens package access management July 2026 August 2026 GitHub credentials
GAPcodexsite:github.blog/changelog/2026 staged publishing npm May 22 trusted publishing tokens approve
GAPcodexsite:github.blog/changelog/ "2026-05-22" "npm"
GAPcodexnpm token security granular tokens package access github August 2026 publishing tokens
GAPcodexsite:github.blog/changelog npm December 9 2025 classic tokens revoked staged publishing July 2026
GAPcodexnpm token security August 2026 package access granular tokens GitHub credentials
GAPcodexsite:github.blog/changelog npm staged publishing 2026 July 2FA
GAPcodexpython.org release metadata vulnerability unsigned downloads API incident 2026 2025
GAPcodexsite.python.org Sigstore verify Python releases identity PEP 761
GAPcodexsite.blog.python.org/2026/06/mitigated-api-bypass "Timeline" "30" "June"
GAPcodex"python/pythondotorg" "3014" "merged" "June"
GAPcodexpython.org release metadata vulnerability authentication 2026 2025 security incident
GAPcodexsite:blog.python.org/2026/06/mitigated-api-bypass "Remediations" "Timeline"
GAPcodexsite:github.com/python/pythondotorg/issues/3010 OR site:github.com/python/pythondotorg/issues/3011
GAPcodexpython.org release metadata security incident authentication bypass before June 30 2026 Trail of Bits
GAPcodexpython.org release metadata vulnerability authentication 2026 2025 API incident
GAPcodexsite.python.org security incident release metadata 2026 2025 downloads authentication
GAPcodexPython Windows code signing certificates revoked August 2025 update investigation June 2026
bakeoffclaude-codePython 4.0 release date announced by Python Steering Council
bakeoffclaude-codewebpack md4 OpenSSL 3 Node 17 hash wasm md4 fix output.hashFunction xxhash64 sokra comment
bakeoffclaude-codegrype offline air-gapped vulnerability database import GRYPE_DB_AUTO_UPDATE false
bakeoffclaude-codecargo audit --no-fetch --db local advisory database path offline
bakeoffclaude-codeDependency-Track air-gapped offline vulnerability mirror configuration NVD OSV feeds URL
bakeoffclaude-codePython 4.0 release date announced by Python Steering Council
bakeoffclaude-codePython 4.0 release date announced by Python Steering Council
bakeoffclaude-codeNode.js documentation --openssl-legacy-provider option "Enable OpenSSL 3.0 legacy provider"
bakeoffclaude-codeaws.amazon.com data transfer out to internet US East (N. Virginia) first 10 TB per GB price same availability zone free
bakeoffclaude-codegrype air-gapped offline database import GRYPE_DB_AUTO_UPDATE false documentation
bakeoffclaude-codeDependency-Track air-gapped offline NVD mirror configuration documentation
bakeoffclaude-codeSafety CLI 3 requires account login API key vulnerability database commercial license
bakeoffclaude-codeAWS EC2 on-demand pricing data transfer out to internet US East (N. Virginia) first 10 TB per month per GB; data transfer same availability zone free
bakeoffclaude-codeStripe annual letter 2025 total payment volume private company revenue
bakeoffclaude-codeStripe revenue estimate UK Companies House filing Stripe Payments Europe accounts net revenue
bakeoffclaude-codeStripe annual letter 2025 total payment volume private company revenue
bakeoffclaude-codeStripe revenue estimate net revenue report private company does not disclose financials
bakeoffclaude-codeStripe Irish entity accounts filed Companies Registration Office revenue $5.1bn pre-tax profit 2024
bakeoffclaude-codeStripe annual letter 2025 total payment volume private company revenue estimate
bakeoffclaude-codeGrype offline air-gapped vulnerability database GRYPE_DB_AUTO_UPDATE false grype db import
bakeoffclaude-codeDependency-Track air-gapped offline mirror NVD OSV internal mirror documentation
How this was run

Agent search behavior: methodology

The analysis reads the stored transcripts of the web search bakeoff (Claude Code, eight setups, 18 tasks, three repetitions) and the search gap bench (Claude Code and codex, brief tasks, three repetitions per admitted task). It runs no agent and changes no grade. Pass and fail come from each run's recorded evaluation.

scripts/analyze_search_behavior.py extracts every completed provider tool call from a transcript:

  • codex MCP tool calls, and codex built-in web_search items;
  • Claude Code tool_use and tool_result pairs, including vendor MCP tools, WebSearch and WebFetch.

Classification rules

  • Search call: a call to a search tool (web_search_exa, brave_web_search, tavily_search, Parallel web_search, firecrawl_search, the Perplexity search tools, Claude Code WebSearch, and codex built-in search actions).
  • Query: each entry of a multi-query call (Parallel search_queries, codex built-in query lists) counts as one query. Each user message in Perplexity's conversational inputs also counts as one query; system and assistant messages are excluded. Parallel's objective is counted as a structured parameter when search_queries is present, not as another query.
  • Fetch call: a page-retrieval tool call (web_fetch_exa, tavily_extract, firecrawl_scrape, Parallel web_fetch, Claude Code WebFetch, and codex built-in open_page, find_in_page and other page views).
  • Natural-language query: ends with "?", starts with an interrogative or auxiliary verb, or contains at least two function words from a fixed list (the, a, an, is, are, was, were, does, do, did, how, what, which, why, when, where, who, that, this, of, to, for, with, will, should, can, about, on, from, by, as, be, has, have).
  • Contains a year: a token from 2000 to 2099. Quoted phrase: contains a double quote. site:: contains site:.
  • Fetched without searching: the run made at least one fetch call and no search call.
  • Primary source surfaced (GAP only): the task's catalog oracle source_url, normalised (scheme, query, fragment and trailing slash removed), appears among the URLs in any returned search result of the run, never in the search input. It is measured only over runs that searched. A run is unknown when any search lacks exposed results and no observed result contains the primary source. Unknown runs are excluded from both surfaced and not-surfaced pass-rate denominators. Recomputing against returned evidence leaves the historical source counts unchanged; no searched run has unknown coverage.

The rules are deliberately simple so they can be audited. The natural-language rule undercounts descriptive keyword-free phrases and overcounts keyword strings that happen to contain function words. Readers can substitute their own rule in the script and rerun it.

Scope

Pooled figures cover the provider and native setups. Reference setups (no-search, floor and ceiling) make no search calls and are excluded from query statistics. GAP run counts differ by agent because calibration admitted seven brief tasks for Claude Code and six for codex.

See the report, summary data and reproduction. Calibration does not apply: calibration record.

Files: reports/2026-10-05-agent-search-behavior/