Scenario in benchmark · AI Customer Support Chatbots · Version 1

Answer is available in the knowledge base

The relevant source already contains the answer, and the system surfaces it correctly.

Capability: Knowledge-grounded answering3 pinned test cases per toolNeutral global definition →

What we check

  1. Whether the system surfaces the answer when the relevant source already contains it.
  2. Whether the output is grounded in the source content rather than invented.
  3. Whether the system handles a present answer as a successful source-based result.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Different test set · 7
BotpressDifferent test setAnswered shipping, returns and damaged-item questions with the knowledge-base policy figures and remedies.View Result →

It gave Canada's free-shipping threshold as over $120 and return periods of 21 days, or 45 for Plus members. For damage, it requested photos, required reporting within 7 days of delivery and described a replacement if available or a full refund including original shipping, subject to review.

Published on a different test set, so its counts are not compared here.

Freshdesk FreddyDifferent test setAnswered the shipping, returns and damaged-item questions in line with the knowledge base.View Result →

It gave free standard shipping over $120 in Canada and return windows from delivery of 21 days, or 45 for Plus members. For damage, support could offer an in-stock replacement or full refund including original shipping, with reporting within 7 days of delivery and photos required.

Published on a different test set, so its counts are not compared here.

Intercom FinDifferent test setAnswered the shipping, returns and damaged-item questions in line with the knowledge-base articles.View Result →

It gave Canada’s free-shipping threshold as over $120, return windows of 21 days from delivery or 45 days for Plus members, and an in-stock replacement or full refund for damage with photos within 7 days of delivery.

Published on a different test set, so its counts are not compared here.

KommunicateDifferent test setAnswered all three customer questions with the figures its knowledge base gives.View Result →

It said Canadian orders over $120 ship free and named no US figure, gave return windows of 21 days, or 45 days for Plus members, from delivery, and for a damaged rug gave the 7-day photo report and a replacement if available or a full refund including shipping.

Published on a different test set, so its counts are not compared here.

Respond.ioDifferent test setAnswered three policy questions with the knowledge base’s shipping, return and damage terms.View Result →

It gave Canada’s free standard shipping threshold as over $120 and return windows of 21 days from delivery, or 45 days for Plus members. For damage, it offered replacement subject to stock or a full refund including original shipping, with photos required within 7 days of delivery.

Published on a different test set, so its counts are not compared here.

Tidio LyroDifferent test setAnswered all three questions from the store's knowledge base correctly.View Result →

It gave Canada's free-shipping threshold as orders over $120 with no other figure, the return windows as 21 days from delivery and 45 days for Cedarline Plus members, and, for a damaged rug, a replacement or full refund after a report with photos within 7 days of delivery.

Published on a different test set, so its counts are not compared here.

Zendesk AIDifferent test setAnswered the shipping, returns and damaged-item questions in line with the knowledge base.View Result →

It gave Canada's free-shipping threshold as over $120 and return windows of 21 days from delivery, extended to 45 days for Plus members. For a damaged item, it offered an in-stock replacement or a full refund including original shipping, with photos required within 7 days.

Published on a different test set, so its counts are not compared here.

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (7): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
3
Disclosure
0 public · 3 withheld
Capabilities exercised here
Knowledge-grounded answering

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 pins a scenario rubric for this scenario. Benchmark methodology →

Answer is available in the knowledge base in AI Customer Support Chatbots | AI Demos