Scenario in benchmark · Web Search for AI · Version 1
A page on the web answers the question
A public web page directly supports the answer.
What we check
- Whether it locates a public page that directly supports the answer.
- Whether it uses that support when forming the response.
- Whether it treats the case as answered by public evidence rather than as unsupported.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
GPT-5.6 terraFailed1 of 1 test case failedAnswered 512 MB but returned only one result, a link to OpenAI's Files API reference, and that link leads to OpenAI's Files index page.View Result →
It refused automated access on the run day; opened on 9 Oct 2026, it showed the Files index page, which lists the endpoints but does not state the 512 MB limit.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ExaPassed1 of 1 test case passedAnswered the question by returning pages that state the Files API upload limit is 512 MB.View Result →
The OpenAI Files API reference is at rank 1, and all ten returned results carry that figure; the other sizes mentioned on those pages are different limits.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LinkupPassed1 of 1 test case passedFound a page answering the question: OpenAI’s Files API reference at rank 8 stated a maximum individual-file upload size of 512 MB.View Result →
In all, seven of the ten standard-mode results contained that figure.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Perplexity SonarPassed1 of 1 test case passedReturned OpenAI's Files API reference page, which states that individual files can be up to 512 MB, and its answer gave the same figure.View Result →
Most other sizes in the 14 results describe different limits, such as file inputs and storage. One community reply (rank 10) wrongly says files uploaded to OpenAI's API are limited to 5 MB, but the answer did not repeat it.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
SerpAPIPassed1 of 1 test case passedReturned OpenAI's Upload file reference at rank four, a page that states the 512 MB per-file limit the question asked about.View Result →
Seven other results also mention 512 MB (ranks 1, 2, 3, 7, 8, 9 and 10).
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
SerperPassed1 of 1 test case passedPassed this scenario: it returned a page that answered the question.View Result →
The search results included the Files API page stating the per-file limit is 512 MB, and five other results also showed that figure.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ValyuPassed1 of 1 test case passedRetrieved a page answering the OpenAI Files API upload-limit question with a limit of 512 MB per file.View Result →
The answer appeared in the returned page text, and eight of ten results contained that figure.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (5): Brave Search API, Firecrawl, OpenAI web search, Tavily, You.com.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Web Retrieval
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →