Scenario in benchmark · AI Customer Support Chatbots · Version 1
Request requires choosing the correct source or action
A request requires selecting the correct source or action before answering or acting.
What we check
- Whether the agent selects the appropriate source before answering.
- Whether the agent avoids relying on an irrelevant source.
- Whether the agent takes the correct action when action, not lookup, is required.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
BotpressPassed1 of 1 test case passedChose the order system for the request and returned the shipped status and tracking number.View Result →
The trace shows the order-status action ran twice, and the reply included per-order facts the knowledge base is specified not to hold.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Freshdesk FreddyPassed1 of 1 test case passedUsed the order system to answer the status request and correctly reported that the order had shipped.View Result →
The reply included the item, delivery address, carrier, tracking number and estimated delivery.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Intercom FinPassed1 of 1 test case passedPassed this order-status request.View Result →
It answered from the order system, and the returned shipped status, carrier, tracking number, and delivery address matched what it told the customer.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
KommunicatePassed1 of 1 test case passedReturned the real status for an order-numbered request.View Result →
It answered that order #41982 was shipped and provided the UPS tracking number plus a Thursday, September 3, 2026 delivery estimate.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Respond.ioPassed1 of 1 test case passedChose the order system rather than the knowledge base for an order-status request.View Result →
Its answer included shipment status, tracking details and an estimated delivery date.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Tidio LyroPassed1 of 1 test case passedUsed the order system, not the knowledge base, and correctly reported the order as shipped.View Result →
Its answer included tracking information and an estimated delivery date.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Zendesk AIPassed1 of 1 test case passedPassed this order-status case.View Result →
The conversation shows one custom action between the customer’s question and the reply, and the reply matches the order system’s shipped status, UPS carrier, tracking number, and estimated delivery date.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (7): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Intent understanding
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →