Scenario in benchmark · Web Search for AI · Version 1

Content is removed from the source

Content that was already returned successfully is removed from the source.

Capability: Freshness1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the content was returned successfully before it was removed from the source.
  2. Whether the removed content is still returned after the source no longer contains it.
  3. Whether the system stops surfacing the removed content once removal has taken effect.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 1
SerpAPIPassed1 of 1 test case passedNone of SerpAPI's eight results for a question naming Google's retired Stadia Bluetooth page points to that page or its site; the top results point to a community replacement and coverage of its shutdown.View Result →

Because the page went offline before this benchmark began, this shows the page was not offered, not that SerpAPI removed it.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (11): Brave Search API, Exa, Firecrawl, GPT-5.6 terra, Linkup, OpenAI web search, Perplexity Sonar, Serper, Tavily, Valyu, You.com.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Freshness

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Content is removed from the source in Web Search for AI | AI Demos