Searchlight

Learn

How a Searchlight test works.

Five steps, the same for every search setup.

  1. Step 1 of 5

    Give the agent a real job

    A research brief or a code fix about something that changed after the model's training cutoff, or a question with a well-documented answer.

    Eval lesson: Past the cutoff, memory alone cannot pass, so search becomes measurable.

  2. Step 2 of 5

    Change only the search

    The same agent, model and instructions every time. Each run plugs in one search provider at its default settings.

    Eval lesson: Frame every provider with two controls: no search, and the agent's own built-in search.

  3. Step 3 of 5

    Run it sealed, three times

    Each run starts in a fresh, isolated workspace, and every task runs three times: 18–42 runs per setup. Every search and page visit is logged.

    Eval lesson: A few dozen runs is a small sample, so every score carries a range.

  4. Step 4 of 5

    Grade it blind

    Automatic checks run where a task has them. Otherwise an AI judge compares the answer with the saved source pages, never told which setup produced it.

    Eval lesson: Fix the pass bar first: the right decision, 70% of the key facts, few unsupported claims.

  5. Step 5 of 5

    Rank it honestly

    Setups are ranked by how often their runs passed. Every pass rate shows its 95% range.

    Eval lesson: When ranges overlap, the test cannot separate them, so no winner is named.

Try it: honest ranges

How sure is 90%?

A pass rate from a few dozen runs comes with a range. Change the passes and the runs and watch it move; it starts at Firecrawl's score in the web search bakeoff.

Passes38of runs42
90%

95% range: 78% to 96%. Run the same test again and the score could land anywhere in that range.

How the range is worked out

Start with the score p = passes ÷ runs, and z = 1.96 for 95%. Wilson's interval sets the centre at (p + z²/2n) ÷ (1 + z²/n) and the half-width at z × √(p(1 − p)/n + z²/4n²) ÷ (1 + z²/n). It never runs past 0% or 100%, and it stays honest at the edges. To halve a range, run about four times as many tests.

The words

Every idea, in one line

Every term on this site; report terms are mapped below.

Agent

The AI coding agent doing the job: Claude Code or Codex.

Search setup

The one thing that changes: a search provider, the agent's built-in search, or none.

Training cutoff

The date a model's knowledge ends.

Brief

A short research report that ends in a decision and cites its sources.

Run

One task with one search setup, repeated three times.

Blind grading

The grader never sees which setup produced the answer.

95% range

Where the true score most likely sits.

Tie

Two ranges overlap, so the test cannot separate them.

Gap closed

How far a setup got from the no-search score toward the score with the answer handed over.

Tokens

The units AI models bill by.

Study

One fixed set of questions sent to every search and research service.

Verified

The claim holds up on the page it cites.

Strict credit

A service earns a finding only if its own cited page supports it.

Unique

A finding only one service earned.

Insight

A verified finding that would change a decision, valued 1 to 3.

Pre-registered

The questions and scoring rules were fixed before any result was seen.

Words in the reports
Search setup arm

One way of giving the agent search: one provider's connector, the agent's built-in search (native), or no search at all.

Test run cell

One task, one search setup, one repetition, graded once.

Agent harness

The AI coding agent program: Claude Code or Codex.

Benchmark names GAP, WSB, SEW

The search gap bench, the web search bakeoff, and the Search Evaluation Workbench that runs them.

Connector MCP server

The standard connector that plugs a search provider into an agent.

No search, answer handed over floor, ceiling

The same task with no search (floor) and with an excerpt containing the answer handed over (ceiling). A task counts only if the agent fails at the floor and passes at the ceiling.

Gap closed gap closure

How far a search setup got from the floor toward the ceiling: 1.00 means all the way.

Grader, agreement judge, kappa

An AI model that grades answers without knowing the setup. In the search gap bench a Claude Code judge decides and a Codex judge checks; kappa measures their agreement (1 is perfect, 0 is chance).

Tokens per pass tokens per success

Tokens used per passing run; the search gap bench counts failed runs' tokens too.

95% range Wilson 95% CI

The range the true pass rate likely falls in, worked out with Wilson's method so it never runs past 0% or 100%.

Chance of a fluke p-value (McNemar, Fisher, sign test)

How likely a difference this large would be if the setups were really equal. Below 0.05 is the usual bar for a real difference.

Held-back questions holdout

Questions kept back from tuning, so a result cannot be fitted to them.

Official page primary source

The project's own release notes, advisory or docs.

Now read the results