Scenario in benchmark · AI Database Agents · Version 1
The user wants several findings in one place
A user asks to combine several distinct findings into one report-like artifact.
What we check
- Whether the agent combines several distinct findings into one coherent artifact.
- Whether each included finding still matches the answer to its own original question.
- Whether the combined output keeps the findings consistent with one another rather than mixing, dropping, or altering them.
- Whether the result remains a report-like summary of already produced answers, not a fresh analysis with changed meaning.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
FutureSmart Database AgentFailed1 of 1 test case failedFailed to keep the assembled findings consistent.View Result →
It kept revenue at USD 20,253.47, but the combined report changed the category answer from storage_and_organization to outdoor_and_patio and the cancellation rate from 12.0% to 13.19%.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Anomaly AIPassed1 of 1 test case passedCombined its three earlier answers into one report and a saved dashboard without changing any of them: August revenue of $8,795.45, Outdoor & Patio as the best-selling category at $1,806, and a 21.75% cancellation rate, 67 of 308 orders.View Result →
The saved dashboard reopens from the Dashboards list into a new chat session.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
AskYourDatabasePassed1 of 1 test case passedBrought several findings together in one response.View Result →
Its report combined revenue, sales category and cancellation figures that agreed with its direct answers to the same questions.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BasedashPassed1 of 1 test case passedPassed this scenario.View Result →
One chat shows three separate answers and a fourth report that repeats the same figures: August revenue $8,795.45, a bedding and rugs/home decor tie at 106 units each, Furniture at $51,248.25, and a 21.75% cancellation rate. Its verification query returned the same values.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BlazeSQLPassed1 of 1 test case passedPassed this one-report task.View Result →
The combined table kept the earlier revenue, category, and cancellation answers intact, with the same values and bases, and brought all five metrics into one report.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DotPassed1 of 1 test case passedAssembled revenue, category, and cancellation into one report.View Result →
Revenue and category matched earlier chat answers; the cancellation figure came after a clarification and matched the PDF.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DraxlrPassed1 of 1 test case passedPassed this request to gather several findings in one place.View Result →
It gave the three direct answers, then combined them into one report without changing the values; the saved report reopened with the same single row.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (4): AI for Database, camelAI, Definite, Querio.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Reporting
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →