Searchlight

An open benchmark

Does web search make AI agents better at real work?

We give AI agents real jobs and change only the search. On anything recent, search decides it: 0% without it, 71–95% with any provider.

See the results
Latest2026-11-11Leading software factories and agent-platform architectures13 services · 24 questions

What we asked the agents to do

Real tasks, quoted from the catalogs.

Research brief

Security releases the AI has never seen

cutoff

Python shipped these on March 3, 2026, after the models' training data ends: without search, an agent can only guess. The source.

The task and its grading
As of March 10, 2026, research security maintenance for Python 3.10 through 3.12. Cover patch-release status, HTTP header validation, XML parsing and SSL memory safety. Produce a concise change-impact brief for maintainers without assuming they can upgrade minor versions. Return JSON with brief (markdown), claims (text and citation_urls).

The brief must reach the right decision, cover at least 70% of the key facts and keep unsupported claims to 25% or less, checked against the official page.

Research brief

A platform policy that just changed

cutoff

GitHub changed what npm access tokens can do on July 31, 2026; only recent sources have the answer. The source.

The task and its grading
As of August 5, 2026, write a research brief on npm token security for maintainers whose automation both publishes packages and manages package access. Explain which operations their credentials can perform, any dated future commitments, and which GitHub credential types are in scope. Return JSON with brief (markdown), claims (text and citation_urls).

Same bar as every brief: right decision, key facts covered, few unsupported claims.

Code fix

Patch a security advisory in real code

cutoff

The fix shipped after the cutoff: the agent must find the advisory and the fixing release, then change the code. The source.

The task and its grading
Repair the session-token middleware: return claims for valid signed tokens and None for every rejected token. Use a dependency release that addresses the applicable advisory without weakening signature checks.

Hidden tests must pass in an isolated workspace. Weakening the security check to make them pass fails.

Documented knowledge

A well-documented technical question

cutoff

Public for years and probably in the training data: does search add anything beyond memory?

The task and its grading
Kubernetes v1.22 removed a batch of beta API versions and broke clients that still called them. Build the list of API groups whose beta version v1.22 removed. Report at least eight. For each, give the removed apiVersion, the replacement apiVersion clients must migrate to, and a primary-source URL from the Kubernetes deprecation guide or release notes. Also report the release in which the removals took effect.

The list must match the official release notes, scored by a grader that never sees which search provider was used.