Scenario in benchmark · AI Customer Support Chatbots · Version 1

Earlier conversation/session needs to be continued

A follow-up depends on resuming an earlier conversation or session from prior context.

Capability: Conversation continuity1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the assistant uses earlier context to continue the same issue.
  2. Whether the assistant treats the follow-up as a continuation rather than a fresh request that must be fully re-explained.
  3. Whether the assistant carries the thread forward across turns or across sessions when the earlier context is available.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 3
BotpressFailed1 of 1 test case failedDid not continue the earlier return conversation when the customer came back.View Result →

After reopening, it asked again for the order number and item, even though session 1 had already established them.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

KommunicateFailed1 of 1 test case failedFails to continue an earlier chat after reopening the widget.View Result →

The restored thread still shows the earlier return request, but the bot asks for more details and then asks for the order number again instead of continuing the thread.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Zendesk AIFailed1 of 1 test case failedDid not continue the earlier return conversation in the follow-up session.View Result →

In the second session, the agent answered generically and then gave support channels instead of naming the earlier order or sofa, while the earlier exchange stayed in the same thread.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 2
Freshdesk FreddyPassed1 of 1 test case passedContinued the return conversation when the customer came back in the same thread.View Result →

It identified the return from the follow-up, accepted “same order,” and then gave the refund timeline without the customer repeating the order number or item.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Intercom FinPassed1 of 1 test case passedContinued the earlier return conversation after the widget was closed and reopened.View Result →

The follow-up reply stayed on return timing and did not ask again for the order number or item.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (9): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Respond.io, Tidio Lyro.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Conversation continuity

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Earlier conversation/session needs to be continued in AI Customer Support Chatbots | AI Demos