Benchmark · Version 1

AI Customer Support Chatbots

This benchmark evaluates AI support agents for businesses that need customer questions answered and customer requests handled in connected systems.

Benchmark overview

9Capabilities
26Scenarios
14Participating tools
What is included and excluded

Evaluates AI support agents on grounded answers, intent understanding, conversation continuity, human handoff, connected-system actions, multilingual support, configuration and control, analytics and observability, and feedback-driven learning.

A reader learns how a product handles the core support capabilities buyers care about: grounded answers, intent understanding, continuity across turns and sessions, human handoff, connected actions, multilingual use, configuration, observability, and feedback-driven learning.

In scope

  • AI support agents for customer support.
  • Grounded answers from the business’s own knowledge base.
  • Intent understanding before answering or acting.
  • Conversation continuity across turns and sessions.
  • Escalation and human handoff.
  • Actions in connected systems.
  • Multilingual support.
  • Agent configuration and control.
  • Analytics and observability.
  • Feedback-driven learning.

Out of scope

  • General RAG chatbots with no support workflow.
  • Live chat tools with no AI.
  • Voice support agents.
  • Prompt-injection resistance.

Participating tools

How each tool did on the 26 scenarios in this benchmark. Each cell is one scenario, in the same order for every tool and grouped by capability; Pass and Fail come from published Results.

PassedFailedPublished, different test setNo published Result yet
ToolScenario results · 26 scenarios in 9 capabilitiesPassedFailedExplore
Botpress
25 of 26 scenarios published · 20 passed · 2 failed · 3 different test set
20of 22 graded2View tool →
Zendesk AI
25 of 26 scenarios published · 18 passed · 4 failed · 3 different test set
18of 22 graded4View tool →
Intercom Fin
24 of 26 scenarios published · 20 passed · 1 failed · 3 different test set
20of 21 graded1View tool →
Freshdesk Freddy
23 of 26 scenarios published · 18 passed · 3 failed · 2 different test set
18of 21 graded3View tool →
Kommunicate
21 of 26 scenarios published · 14 passed · 4 failed · 3 different test set
14of 18 graded4View tool →
Respond.io
19 of 26 scenarios published · 16 passed · 0 failed · 3 different test set
16of 16 graded0View tool →
Tidio Lyro
10 of 26 scenarios published · 8 passed · 0 failed · 2 different test set
8of 8 graded0View tool →
7 more toolsin the rosterNo published Results yet for Ada · Chatbase · CustomGPT.ai · Decagon · FS Agent (DIY control) · Gorgias · Help Scout.Each tool gets its own row here once its first Result is published.

Capabilities & scenarios

26 scenarios grouped by 9 capabilities. Open a group to explore its scenarios in this benchmark.

Knowledge-grounded answering3 scenarios

The agent answers customer questions from the business’s own knowledge base rather than guessing from model assumptions.

Capability in this benchmark → · Global definition →

  1. Answer is available in the knowledge base7 tools with published Results
  2. Answer is not available in the knowledge base7 tools with published Results
  3. The request uses different wording than the source1 tool with a published Result
Intent understanding3 scenarios

The agent works out what the customer actually wants before it answers or acts.

Capability in this benchmark → · Global definition →

  1. Request requires choosing the correct source or action7 tools with published Results
  2. One message contains multiple requests7 tools with published Results
  3. Request is unclear and needs clarification7 tools with published Results
Conversation continuity3 scenarios

The agent keeps track of earlier context across turns and sessions when later replies depend on it.

Capability in this benchmark → · Global definition →

  1. Information from earlier in the conversation is needed later6 tools with published Results
  2. Customer corrects information given earlier6 tools with published Results
  3. Earlier conversation/session needs to be continued5 tools with published Results
Escalation and human handoff3 scenarios

The agent knows when a conversation should go to a human and hands it over properly.

Capability in this benchmark → · Global definition →

  1. User explicitly asks for a human7 tools with published Results
  2. A configured rule requires escalation6 tools with published Results
  3. Agent cannot resolve the request7 tools with published Results
Action execution5 scenarios

The agent can look things up and make changes in connected systems such as orders, subscriptions, or tickets.

Capability in this benchmark → · Global definition →

  1. Retrieve information from an external system5 tools with published Results
  2. Update something in an external system5 tools with published Results
  3. Requested action cannot be completed5 tools with published Results
  4. Action requires confirmation6 tools with published Results
  5. User is not authorized to perform the action6 tools with published Results
Multilingual support2 scenarios

The agent handles customers who do not write in English and can follow language switches during a conversation.

Capability in this benchmark → · Global definition →

  1. Customer communicates in another supported language4 tools with published Results
  2. Customer switches languages during the conversation4 tools with published Results
Agent configuration and control2 scenarios

The agent follows configured instructions and respects configured restrictions.

Capability in this benchmark → · Global definition →

  1. Configured instruction changes agent behavior7 tools with published Results
  2. Configured restriction is respected7 tools with published Results
Analytics and observability3 scenarios

The operator can inspect conversations and agent actions, and the reported numbers match what actually happened.

Capability in this benchmark → · Global definition →

  1. Conversation activity can be inspected6 tools with published Results
  2. Agent actions/tool calls can be inspected5 tools with published Results
  3. Reported analytics match what actually happened4 tools with published Results
Feedback and learning2 scenarios

The agent accepts corrections and, where the product claims learning, changes future answers.

Capability in this benchmark → · Global definition →

  1. Feedback can be submitted5 tools with published Results
  2. Correction changes future behavior where the product claims learning5 tools with published Results

Results overview

Current published evidence in this benchmark.

147Current published Results
26Scenarios with published Results
7Tools with published Results

Publication availability is separate from test coverage and unpublished research progress.

Explore scenarios →

How the benchmark works

A public summary of the evaluation method. The same defined scope and evidence standard apply to every tool assessed under this version.

Scenario meaning

Scenarios describe conditions, not inputs

A scenario is a condition that makes the capability hard, not an input archetype or a smaller capability.

Initial test case

Simplest direct verification

The first test case under a scenario is the simplest, most natural test that directly verifies it.

Later depth

Add harder variants from execution

Harder variants are added later from observed product differences; depth is earned from execution, not designed upfront.

Full benchmark methodology →

Resources and fixtures

The registered material and systems that create a consistent test environment for this benchmark.

Database

Cedarline commerce database

A shared PostgreSQL 16 commerce database used for benchmark evaluations.

AI Customer Support Chatbots — Benchmark definition | AI Demos