Search API study · 2026-09-26
Search API head-to-head
In ten seconds
Exa added the most verified new facts (25 on 40 open questions), Firecrawl read the most pages built-in tools could not (14 of 15), and Parallel filled a template best.
New or corrected facts, 40 open questions
New dated facts, two weeks of company news
2 more measures
Pages it read that built-in tools could not of 15
Correct values filling a fixed template of 133
Counts from one run per question, copied from the report. No ranges were published, so a gap of a few items may be chance.
The full report
Search API head-to-head: Exa, Tavily, Parallel and Firecrawl on a research workload (2026-09-26)
This study called four search APIs directly, not through a coding agent. It asked how much verified new information each adds to a research knowledge base that had already been built with an agent's built-in search tools until they stopped finding new facts, and what each provider surfaces that the others don't. It was pre-registered and cost-capped, results were pooled and graded blind, and every counted claim was verified on its source page. Stages A and B ran on 2026-09-26, Stage C on 2026-09-27, and an Exa-only Monitors probe ran daily from 2026-09-27 to 2026-10-06. The full record (design, pre-registration with twelve amendments, judge briefs and stage write-ups), the code and the sanitized data are published in this directory.
Context. The experiment was run while building a go-to-market knowledge base about Exa, one of the four providers tested. Its questions come from that knowledge base's open questions about Exa's market and competitors, so the workload reflects what that knowledge base needed. The design added workloads where Exa might be weak on purpose (freshness, and pages that need rendering). Weigh the results with that in mind; the record, code and data are here so readers can check them.
Headline
- On top of built-in search, Exa added the most verified new information. On 40 questions the knowledge base couldn't answer, verified new or correcting claims under strict credit: Exa 25, Parallel 19, Tavily 11, Firecrawl 10, with 11 of Exa's found by no other provider. On two weeks of company news: Exa 20 new dated facts, Tavily 12, Parallel 3, Firecrawl 2. Building three entity lists: Exa returned 28 entities new to the knowledge base (21 that no other provider returned), Firecrawl 19 and Parallel 7.
- The other providers led elsewhere. On pages the built-in tools couldn't read, Firecrawl recovered 14 of 15, Parallel 12, Tavily 10 and Exa 5. Filling a 20-company, eight-field schema, Parallel was right on 113 of 133 ground-truth cells, Exa on 105 and Firecrawl on 94; the Parallel–Exa gap is not significant, at the same price.
- Finding a source the knowledge base already cites was a tie: 16 to 19 of 20 for all four, on identical inputs.
- On evidence about four big grounding vendors (Stage C), no lead is significant. Verified new tier-1 claims across a search arm and a research arm, after the post-hoc corrections of Amendment 12: Exa 16, Parallel 14, Tavily 13, Firecrawl 9. Parallel's research agent was the most accurate: 0.6% of its checkable items were unsupported or false, against Exa's 4.6%, Firecrawl's 2.7% and Tavily's 18.1%.
- Exa's Monitors caught real competitor changes, with some noise. Over 49 daily runs on five competitors' pricing and changelog pages, they reported 20 changes: 14 confirmed, 5 false positives. 13 were real, dated pricing or product changes, 12 of them missing from the knowledge base or only partly recorded there.
- One provider for everything leaves gaps. For this workload the record recommends Exa for discovery, list building and news sweeps; Firecrawl or Parallel for pages that need rendering; and Parallel or Exa for filling a fixed schema.
What was tested
Each provider ran with parameters taken from its own best-practice documentation and frozen before the first call, on the same day, with the same top 10. The design is a capability matrix rather than four copies of every job: Stage A's cheap probes ran on all four providers, and Stage B's list and schema sets ran only on providers that sell those capabilities. Tavily sat out S5 and S6 because it sells no list-building or structured-extraction product. See methodology for the endpoints, grading and amendments.
| Set | Stage | Size | What it tests | Providers |
|---|---|---|---|---|
| S1 known unknowns | A | 40 questions | Can a provider answer what the built-in research couldn't? | All four |
| S2 known knowns | A | 20 claims | Is the primary source the knowledge base already cites in the top 10? | All four |
| S3 blocked sources | A | 15 pages | Can it read pages the built-in tools couldn't (JavaScript, rate limits, PDFs)? | All four |
| S4 freshness | A | 10 companies | New, dated facts from the last 14 days | All four |
| S5 entity lists | B | 3 lists | Precision, and entities new to the knowledge base | Exa, Parallel, Firecrawl |
| S6 schema fill | B | 20 companies × 8 fields | Accuracy and completeness on a fixed schema | Exa, Parallel, Firecrawl |
| SP find-similar | B | 10 homepages | New competitor domains | Exa |
| SM Monitors | B | 5 monitors, daily | Does a monitor surface real, dated pricing or product changes? | Exa |
| SV big vendors' evidence | C | 24 questions × 2 arms | Verified new evidence on four grounding vendors, by search and by research agent | All four |
Stage A: four providers, cheap probes
Strict credit counts a claim only when the provider's own result page states it (Amendment 4). S2 was re-run with identical inputs after a review found that hand-written queries leaked answer terms. The Stage A write-up has broad credit, the first grading round and the diagnostics.
| Stage A (v2 re-grade, strict credit) | Exa | Tavily | Parallel | Firecrawl |
|---|---|---|---|---|
| S1 new or correcting claims, strict (known unknowns, 40 questions) | 25 | 11 | 19 | 10 |
| … no other provider supported | 11 | 2 | 4 | 0 |
| S4 new dated facts, strict (10 companies, two weeks) | 20 | 12 | 3 | 2 |
| S3 pages recovered that native tools couldn't read (of 15) | 5 | 10 | 12 | 14 |
| S2 cited sources found in the top 10, identical inputs (of 20) | 18 | 16 | 19 | 17 |
| Stage A run spend | $0.50 | 146 credits ($1.17) | $0.37 | 402 credits ($0.30–$2.01) |
| Median latency, ms | 1,230 | 2,995 | 2,343 | 1,171 |
Stage B: lists, schema fill and find-similar
S5 counts entities verified against the list's criteria and new to the knowledge base, under the fact-level rule; the stricter entity-level rule gives Exa 27, Firecrawl 18 and Parallel 3, in the same order. S6 compares cells with the knowledge base's verified ground truth. Parallel's spend is at list price; Firecrawl's credits are shown as a range across plans. The Stage B write-up has each list, each field and the review fixes (Amendments 6 and 7).
| Stage B | Exa | Parallel | Firecrawl |
|---|---|---|---|
| S5 entities verified and new to the KB (3 lists, fact-level rule) | 28 | 7 | 19 |
| … no other provider returned it | 21 | 5 | 12 |
| S5 precision: entities meeting every criterion | 28/36 (77.8%) | 13/21 (61.9%) | 19/23 (82.6%) |
| S6 right on KB ground truth, strict (of 133 cells) | 105/133 (78.9%) | 113/133 (85.0%) | 94/133 (70.7%) |
| S6 cells filled (of 160) | 149/160 (93.1%) | 156/160 (97.5%) | 157/160 (98.1%) |
| S6 false-claim rate | 7/149 (4.7%) | 6/156 (3.8%) | 16/157 (10.2%) |
| S6 cells where it added a verified new fact (of 40) | 30 | 31 | 25 |
| Stage B spend | $4.09 | $12.16 (list price) | 393 credits ($0.29–$1.97), plus 5 free runs |
Find-similar (SP, Exa only) is a cheap sweep for long-tail competitors, but it needs triage: 50 of the 61 new domains were directories, profiles, clones or unrelated sites. It was graded, but not blind, since only Exa ran.
| Exa find-similar, 10 competitor homepages | Count |
|---|---|
| Results returned | 100 |
| Distinct registrable domains | 69 |
| … already cited in the KB | 8 |
| … new to the KB | 61 |
| New domains: relevant vendors (search, SERP, crawling, or web-data APIs) | 11 |
| New domains: profiles, directories, or reviews of a seed company | 39 |
| New domains: clones or unrelated | 11 |
| Relevant vendors the KB baseline already covers | 0 |
| Spend | $0.07 |
Stage C: evidence on the big grounding vendors
Twenty-four pre-registered questions asked for evidence on four supply lines a market sizing couldn't anchor: Google grounding, Anthropic's and OpenAI's web search tools, and Microsoft Grounding with Bing. Each provider ran a search arm and a research-agent arm, and eight blind judges verified every claim on its page. A tier-1 claim is a figure that enters a line's arithmetic: a volume, a revenue figure or share, a denominator, or a price. After a review, Amendment 12 (post hoc) added a claim-level correction layer for novelty, tiers, duplicates and Tavily's cap-refused runs; the Stage C write-up shows the as-judged figures beside the corrected ones.
| Provider | New tier-1 claims (either arm) | …found by no other provider | All new claims | Cost, both arms | Cost per new tier-1 claim |
|---|---|---|---|---|---|
| Exa | 16 | 6 | 207 | $2.57 | $0.16 |
| Parallel | 14 | 4 | 131 | $2.52 | $0.18 |
| Tavily | 13 | 4 | 79 | $3.96–$6.34 | $0.30–$0.49 |
| Firecrawl | 9 | 0 | 126 | $1.16–$7.75 | $0.13–$0.86 |
By arm, corrected. The research error rate is the share of a research agent's checkable items whose cited page didn't state them or contradicted them.
| Provider | Arm | New tier-1 claims | …found by no other arm | All new claims | Questions with a new claim (of 24) | Useful new tier-1 (2–3) | Research error rate | Median time |
|---|---|---|---|---|---|---|---|---|
| Exa | search | 11 | 2 | 100 | 20 | 6 | — | 1.7 s |
| Parallel | search | 8 | 3 | 75 | 20 | 6 | — | 3.1 s |
| Tavily | search | 11 | 3 | 58 | 21 | 5 | — | 3.2 s |
| Firecrawl | search | 7 | 0 | 57 | 18 | 3 | — | 1.3 s |
| Exa | research | 11 | 2 | 158 | 24 | 8 | 4.6% (21 of 455) | 86.1 s |
| Parallel | research | 10 | 1 | 84 | 17 | 6 | 0.6% (2 of 331) | 107.0 s |
| Tavily | research | 4 | 1 | 36 | 13 | 1 | 18.1% (19 of 105) | 31.4 s |
| Firecrawl | research | 5 | 0 | 85 | 18 | 3 | 2.7% (8 of 298) | 71.7 s |
Is the lead real? Exact two-sided sign tests over the 24 questions, chosen after the results were known, with Holm's adjustment for the six comparisons and question-level bootstraps:
| Arm | Leader | Against | New tier-1 (leader / other) | Questions ahead / behind | Sign test p | Holm-adjusted p (6 tests) | Bootstrap 95% CI of the difference |
|---|---|---|---|---|---|---|---|
| search | Exa | Tavily | 11 / 11 | 3 / 3 | 1.000 | 1.000 | -7 to 7 |
| search | Exa | Parallel | 11 / 8 | 3 / 2 | 1.000 | 1.000 | -5 to 12 |
| search | Exa | Firecrawl | 11 / 7 | 4 / 1 | 0.375 | 1.000 | -1 to 10 |
| research | Exa | Parallel | 11 / 10 | 4 / 2 | 0.688 | 1.000 | -5 to 6 |
| research | Exa | Firecrawl | 11 / 5 | 5 / 1 | 0.219 | 1.000 | 0 to 13 |
| research | Exa | Tavily | 11 / 4 | 5 / 1 | 0.219 | 1.000 | 0 to 15 |
Monitors probe (SM)
Five Exa Monitors watched competitors' pricing and changelog pages once a day. The pre-registered question: does a monitor surface a real, dated pricing or product change on the competitor's own page within the knowledge base's 30-day staleness window, and was the change already in the knowledge base?
- Deviation from the plan. The probe was planned to run to 2026-10-10 and then delete its monitors. It was harvested and the monitors paused on 2026-10-06, four days early, so it covers 49 daily runs rather than about 70. The monitors were paused, not deleted.
- What the monitors reported. 14 of 49 runs reported at least one change, 20 changes in all. Monitors report only what changed since their previous run.
- Graded result. Of the 20 reported changes, 14 were confirmed against the vendor's page and archived snapshots, 5 were false positives and 1 could not be settled either way. 13 were real, dated pricing or product changes, and 12 of those were missing from the knowledge base or only partly recorded there. 3 of 5 monitors answered the pre-registered question yes: Parallel, Firecrawl and Perplexity API.
- Where it worked and where it didn't. Perplexity's monitor was the strongest: its changelog carries exact timestamps, and every confirmed change fell inside its run window. Firecrawl's monitor reported a security feature added, removed and added again on its pricing page; the page was unchanged in every archived capture, so those reports are noise. Both of Brave's reports were false positives: prices were unchanged, and the watched pages had started serving a version without prices. Tavily's monitor reported nothing, and its changelog had nothing new.
- Misses were not measured. The grader noticed one: a Firecrawl changelog entry from 2026-09-29 was never reported. Per-change verdicts, evidence links and notes are in the grading record and its write-up.
- How it was graded. One Claude agent, not blind (only Exa ran), checked each reported change against the vendor's current page and Internet Archive snapshots from before and after the run, and searched the knowledge base for it.
| Monitor (competitor pages watched) | Daily runs | Runs reporting a change | Changes reported | Confirmed real | Real, dated pricing or product change | …missing from the KB or partial |
|---|---|---|---|---|---|---|
| Tavily | 10 | 0 | 0 | 0 | no (0) | no (0) |
| Parallel | 10 | 3 | 4 | 4 | yes (3) | yes (2) |
| Firecrawl | 10 | 4 | 6 | 3 | yes (3) | yes (3) |
| Perplexity API | 9 | 6 | 8 | 7 | yes (7) | yes (7) |
| Brave Search API | 10 | 1 | 2 | 0 | no (0) | no (0) |
Spend
Stages A and B together cost at most $23.29 of the $40 cap, valuing credits at the most they cost (Stage A $5.07, Stage B $18.22). Stage C's costs are in its provider table above; Exa's are billed, Parallel's are list-price estimates and Tavily's and Firecrawl's credits are ranges across plans. The Monitors probe made 49 runs, at $15 per 1,000 runs on Exa's price list.
Read this before comparing providers
- Small samples, one run. Each item ran once on one day. The web and the indexes change, so a re-run will differ.
- Model judges. Claude agents graded blind to the provider and verified every counted claim on its page; no human graded the sets.
- Post-hoc changes are logged, not hidden. Amendments 4, 6, 7 and 12 changed scoring after results were seen. The record keeps the earlier figures beside the corrected ones, and the pre-registration lists every amendment.
- New means new to one knowledge base. Novelty is measured against the knowledge base the questions came from.
- Direct APIs, tuned parameters. Unlike the agent benchmarks, where each provider ran through its MCP server at default settings, this study called each API directly with its best-practice parameters. Results do not transfer between the two setups.
- Vendor terms. The experiment was first run as an internal evaluation. The record's overview summarises each vendor's terms on benchmarking as read on 2026-09-26 (terms).
- What is and isn't published. For Stages A to C, page text, titles, snippets, vendor-written values and raw responses are not published, and social-media URLs are hashed. Those raw responses were deleted after grading (Amendment 7), so their grading cannot be replayed exactly. Every published table rebuilds from
data/except Stage A's S3 rows, which need page text; their per-page outcomes are in the scripted summary. The Monitors probe is the exception: by the owner's decision of 2026-10-06, its raw API responses, including Exa's change titles, summaries and run notes, are published verbatim in data/stage-b/runs/SM/raw/ (each monitor's webhook secret was never stored), and its sanitized record rebuilds from them byte for byte.
Record and data
- The experiment's overview, design and pre-registration with Amendments 1–12.
- Judge briefs: first round, re-grade, Stage B and Stage C.
- Stage write-ups: Stage A, Stage B and Stage C, each with every table and the verified findings.
- Code: the Stage A harness code/harness.py, Stage B code/harness_b.py, Stage C code/stage_c/harness_research.py, the exporter code/export_data.py and the Monitors harvest code/monitors_harvest.py.
- Scores: Stage A, Stage B, Stage C corrected and as judged; the Monitors harvest, raw responses and grading.
How this was run
Search API head-to-head: methodology
This study calls search APIs directly; no coding agent is involved. Its design, query sets, parameters and metrics were frozen in the design and the pre-registration before the first paid call, with SHA-256 hashes of the frozen files. Twelve amendments follow, each dated and marked as made before or after results were seen. This page summarises the method; the record is authoritative.
Providers and endpoints
Every provider's parameters come from its own best-practice documentation, read on 2026-09-26, and are written into the harness configs (code/config.json, code/config_b.json, code/stage_c/config.json and config_c2.json). Every provider got the same top 10 and ran on the same day.
- Exa:
/search(auto, highlights) for S1 and S2;/contentswith a live crawl for S3; the news category with a date filter for S4; the Agent API for S5 (capped) and S6 (with an output schema); find-similar for SP; Monitors for SM; and the Agent API atmediumeffort for Stage C's research arm. - Tavily: search (
advanced, chunks); extract for S3; news with a time range for S4; the Research API (mini) for Stage C's research arm. Tavily sat out S5 and S6: it sells no list-building or structured-extraction product, and search plus our own model would measure our model, not Tavily. - Parallel: search (
advanced, with an objective); extract for S3;after_datefor S4; FindAll for S5; Task with an output schema for S6; Task (pro) for Stage C's research arm. - Firecrawl: search plus a scrape of the top 3; scrape with JavaScript rendering for S3;
tbsfor S4;/agentfor S5 and S6 (with a schema);/agentcapped at 100 credits for Stage C's research arm.
Sets
The query sets are drawn from the knowledge base's open questions and claims (see the design): S1 40 known unknowns, S2 20 known knowns, S3 15 blocked pages, S4 10 companies' last two weeks, S5 three entity lists, S6 20 companies by eight fields, SP 10 competitor homepages, SM 5 monitors and SV 24 questions on four grounding vendors. Their hashes are pinned in code/sets/SHA256SUMS and match the prefixes frozen in the pre-registration.
Grading
- Pooled and blind. Each question's results are pooled across providers, deduplicated and, from Amendment 4, shuffled with a recorded seed. Judges never see provider names; attribution is rejoined only after judging.
- Verified on the page. Judges are Claude agents working from written briefs (first round, re-grade, Stage B, Stage C). Every counted claim must be stated on its source page.
- Strict per-claim credit (Amendment 4): a provider is credited with a claim only when its own result page states it. Broad credit, where any pooled page may support the claim, is reported in the stage write-ups.
- Novelty is judged against the knowledge base at a recorded baseline commit. Claims are weighted for usefulness: 3 corrects a knowledge-base claim, 2 is a new fact for a page, 1 is a new corroborating source and 0 is trivia.
- Scripted sets. S2 counts whether the cited primary source is in the top 10. S3 counts a page as recovered when the fetch returns at least 500 characters and a target keyword, with one recorded correction (a redirect).
- S5 and S6. An S5 entity counts when it meets every criterion of its list and is new to the knowledge base (a fact-level and a stricter entity-level rule are both reported). S6 cells are compared with the knowledge base's verified ground truth where it has one; the false-claim rate is the share of filled cells that are wrong.
- Stage C. Eight blind judges. A tier-1 claim enters a sizing line's arithmetic. The research error rate is the share of a research agent's checkable items whose cited page didn't state or contradicted them. Amendment 12 (post hoc) adds a claim-level correction layer (
data/stage-c/corrections.json): novelty against the knowledge base's working notes, tiers, duplicates across questions, and scoring Tavily's four cap-refused runs as returning nothing, as pre-registered. Sign tests, Holm's adjustment and bootstraps were chosen after the results were known. - Monitors (SM). One Claude agent, not blind (only Exa ran), checked each reported change against the vendor's current page and Internet Archive snapshots from before and after the run, and searched the knowledge base for it. A monitor passes when at least one change is confirmed, is a pricing or product change, and is dated inside the 30 days before the run.
Gates, caps and stop rules
S1 ran on 20 questions first and would have stopped there had no provider resolved at least 3 that the built-in tools couldn't. Stage B ran only for providers and workloads that showed signal in Stage A. Each provider had a hard cap, and the owner set a $40 cap for Stages A and B together (Amendment 5); Stage C had its own caps (Amendment 8).
Amendments that changed scoring after results were seen
- Amendment 4: after a review, Stage A was re-graded with shuffled pools and strict per-claim credit, and S2 was re-run on identical inputs.
- Amendment 6: Stage B scoring choices, written after unblinding.
- Amendment 7: fixes from the Stage B review, and the per-file allowlist for exported data.
- Amendment 12: the Stage C correction layer described above.
The record keeps the earlier figures beside the corrected ones in each case.
Costs
Exa's costs are billed. Parallel's are list-price estimates from the harness config. Tavily's and Firecrawl's credits are shown as ranges across their plans. Stage A and B costs value credits at the most they cost, as the cap guard did.
Published data
code/export_data.py writes the data through an allowlist. It keeps request parameters, status, latency, cost, each result's rank, URL, domain, published date and text length, crawl status codes, the pools (URLs only), attribution maps, judge verdicts and scores. It never exports page titles, snippets, page text, extracted content, vendor-written values or raw responses, and replaces the paths of social-media profile and post URLs with a stable hash. Judge notes may quote short fragments of pages. The Monitors export (code/monitors_harvest.py) follows the same rule: change types and fingerprints are kept, change summaries and titles only as lengths. The Monitors probe is the one exception to the no-vendor-text rule: by the owner's decision of 2026-10-06, its raw API responses, including Exa's change titles, summaries and run notes, are published verbatim in data/stage-b/runs/SM/raw/.
See the report, summary data and reproduction. Calibration does not apply: calibration record.