Scenario in benchmark · AI Customer Support Chatbots · Version 1

Customer corrects information given earlier

A later correction supersedes earlier information for the rest of the interaction.

Capability: Conversation continuity1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the later correction replaces the earlier detail in later turns.
  2. Whether the response reflects only the corrected information, not a mix of old and new details.
  3. Whether the agent does not continue reasoning from the superseded detail after the correction.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 6
BotpressPassed1 of 1 test case passedHandled a customer correcting information given earlier in the visible chat.View Result →

The agent first answered about one order, then switched to the corrected order and said it was processing with no tracking number. The first order's shipped status and tracking number did not carry over.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Freshdesk FreddyPassed1 of 1 test case passedSwitched to the corrected order without carrying over details from the first.View Result →

It answered with the dinnerware order's processing status, not the linen-sheet order's details.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Intercom FinPassed1 of 1 test case passedHandled the corrected order lookup correctly.View Result →

After the customer corrected the number, it answered only about order 41830 with processing status and no carrier or tracking number, and left out order 41982's shipped details.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

KommunicatePassed1 of 1 test case passedPassed this conversation-continuity case: after the customer corrected the order number, it answered only about the new order and dropped the first one.View Result →

The final reply names 41830 and leaves out 41982.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Respond.ioPassed1 of 1 test case passedHandled the correction and kept the answer on the updated order.View Result →

After the customer changed the number, the reply covered only 41830: it said the order was processing, gave an arrival date of November 2, 2026, and said there was no tracking number yet.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Zendesk AIPassed1 of 1 test case passedAnswered about the corrected order 41830 and did not carry over 41982.View Result →

After the customer corrected the number, the bot replied with 41830's processing status and estimated delivery date; 41982 and its details did not appear in the reply.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (8): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Tidio Lyro.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Conversation continuity

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Customer corrects information given earlier in AI Customer Support Chatbots | AI Demos