Learn
How a Searchlight test works.
Five steps, the same for every search setup.
Step 1 of 5
Give the agent a real job
A research brief or a code fix about something that changed after the model's training cutoff, or a question with a well-documented answer.
Eval lesson: Past the cutoff, memory alone cannot pass, so search becomes measurable.
Step 2 of 5
Change only the search
The same agent, model and instructions every time. Each run plugs in one search provider at its default settings.
Eval lesson: Frame every provider with two controls: no search, and the agent's own built-in search.
Step 3 of 5
Run it sealed, three times
Each run starts in a fresh, isolated workspace, and every task runs three times: 18–42 runs per setup. Every search and page visit is logged.
Eval lesson: A few dozen runs is a small sample, so every score carries a range.
Step 4 of 5
Grade it blind
Automatic checks run where a task has them. Otherwise an AI judge compares the answer with the saved source pages, never told which setup produced it.
Eval lesson: Fix the pass bar first: the right decision, 70% of the key facts, few unsupported claims.
Step 5 of 5
Rank it honestly
Setups are ranked by how often their runs passed. Every pass rate shows its 95% range.
Eval lesson: When ranges overlap, the test cannot separate them, so no winner is named.
Try it: honest ranges
How sure is 90%?
A pass rate from a few dozen runs comes with a range. Change the passes and the runs and watch it move; it starts at Firecrawl's score in the web search bakeoff.
95% range: 78% to 96%. Run the same test again and the score could land anywhere in that range.
How the range is worked out
Start with the score p = passes ÷ runs, and z = 1.96 for 95%. Wilson's interval sets the centre at (p + z²/2n) ÷ (1 + z²/n) and the half-width at z × √(p(1 − p)/n + z²/4n²) ÷ (1 + z²/n). It never runs past 0% or 100%, and it stays honest at the edges. To halve a range, run about four times as many tests.
The words
Every idea, in one line
Every term on this site; report terms are mapped below.
The AI coding agent doing the job: Claude Code or Codex.
The one thing that changes: a search provider, the agent's built-in search, or none.
The date a model's knowledge ends.
A short research report that ends in a decision and cites its sources.
One task with one search setup, repeated three times.
The grader never sees which setup produced the answer.
Where the true score most likely sits.
Two ranges overlap, so the test cannot separate them.
The control: the agent answers from memory alone.
The search the agent ships with.
How far a setup got from the no-search score toward the score with the answer handed over.
The units AI models bill by.
One fixed set of questions sent to every search and research service.
The claim holds up on the page it cites.
A service earns a finding only if its own cited page supports it.
A finding only one service earned.
A verified finding that would change a decision, valued 1 to 3.
The questions and scoring rules were fixed before any result was seen.
Words in the reports
One way of giving the agent search: one provider's connector, the agent's built-in search (native), or no search at all.
One task, one search setup, one repetition, graded once.
The AI coding agent program: Claude Code or Codex.
The search gap bench, the web search bakeoff, and the Search Evaluation Workbench that runs them.
The standard connector that plugs a search provider into an agent.
The same task with no search (floor) and with an excerpt containing the answer handed over (ceiling). A task counts only if the agent fails at the floor and passes at the ceiling.
How far a search setup got from the floor toward the ceiling: 1.00 means all the way.
An AI model that grades answers without knowing the setup. In the search gap bench a Claude Code judge decides and a Codex judge checks; kappa measures their agreement (1 is perfect, 0 is chance).
Tokens used per passing run; the search gap bench counts failed runs' tokens too.
The range the true pass rate likely falls in, worked out with Wilson's method so it never runs past 0% or 100%.
How likely a difference this large would be if the setups were really equal. Below 0.05 is the usual bar for a real difference.
Questions kept back from tuning, so a result cannot be fitted to them.
The project's own release notes, advisory or docs.