Scenario in benchmark · AI Customer Support Chatbots · Version 1
Answer is available in the knowledge base
The relevant source already contains the answer, and the system surfaces it correctly.
What we check
- Whether the system surfaces the answer when the relevant source already contains it.
- Whether the output is grounded in the source content rather than invented.
- Whether the system handles a present answer as a successful source-based result.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
BotpressDifferent test setAnswered shipping, returns and damaged-item questions with the knowledge-base policy figures and remedies.View Result →
It gave Canada's free-shipping threshold as over $120 and return periods of 21 days, or 45 for Plus members. For damage, it requested photos, required reporting within 7 days of delivery and described a replacement if available or a full refund including original shipping, subject to review.
Published on a different test set, so its counts are not compared here.
Freshdesk FreddyDifferent test setAnswered the shipping, returns and damaged-item questions in line with the knowledge base.View Result →
It gave free standard shipping over $120 in Canada and return windows from delivery of 21 days, or 45 for Plus members. For damage, support could offer an in-stock replacement or full refund including original shipping, with reporting within 7 days of delivery and photos required.
Published on a different test set, so its counts are not compared here.
Intercom FinDifferent test setAnswered the shipping, returns and damaged-item questions in line with the knowledge-base articles.View Result →
It gave Canada’s free-shipping threshold as over $120, return windows of 21 days from delivery or 45 days for Plus members, and an in-stock replacement or full refund for damage with photos within 7 days of delivery.
Published on a different test set, so its counts are not compared here.
KommunicateDifferent test setAnswered all three customer questions with the figures its knowledge base gives.View Result →
It said Canadian orders over $120 ship free and named no US figure, gave return windows of 21 days, or 45 days for Plus members, from delivery, and for a damaged rug gave the 7-day photo report and a replacement if available or a full refund including shipping.
Published on a different test set, so its counts are not compared here.
Respond.ioDifferent test setAnswered three policy questions with the knowledge base’s shipping, return and damage terms.View Result →
It gave Canada’s free standard shipping threshold as over $120 and return windows of 21 days from delivery, or 45 days for Plus members. For damage, it offered replacement subject to stock or a full refund including original shipping, with photos required within 7 days of delivery.
Published on a different test set, so its counts are not compared here.
Tidio LyroDifferent test setAnswered all three questions from the store's knowledge base correctly.View Result →
It gave Canada's free-shipping threshold as orders over $120 with no other figure, the return windows as 21 days from delivery and 45 days for Cedarline Plus members, and, for a damaged rug, a replacement or full refund after a report with photos within 7 days of delivery.
Published on a different test set, so its counts are not compared here.
Zendesk AIDifferent test setAnswered the shipping, returns and damaged-item questions in line with the knowledge base.View Result →
It gave Canada's free-shipping threshold as over $120 and return windows of 21 days from delivery, extended to 45 days for Plus members. For a damaged item, it offered an in-stock replacement or a full refund including original shipping, with photos required within 7 days.
Published on a different test set, so its counts are not compared here.
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (7): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 3
- Disclosure
- 0 public · 3 withheld
- Capabilities exercised here
- Knowledge-grounded answering
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 pins a scenario rubric for this scenario. Benchmark methodology →