Scenario in benchmark · AI Customer Support Chatbots · Version 1

Customer switches languages during the conversation

A conversation switches languages mid-stream and must continue without losing context.

Capability: Multilingual support1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the agent preserves earlier context after the language change.
  2. Whether the agent continues the exchange in the new language.
  3. Whether the agent keeps the topic and prior thread intact rather than restarting the conversation.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
Zendesk AIFailed1 of 1 test case failedFailed the language-switch scenario.View Result →

After the customer switched to English, it kept the policy context but answered in French, including the follow-up reply and the holding line before it.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 3
BotpressPassed1 of 1 test case passedKept the conversation context when the customer switched from French to English.View Result →

The same thread shows a French return-policy answer first, then an English follow-up that repeats the 45-day Plus window and the 21-day standard window.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Freshdesk FreddyPassed1 of 1 test case passedKept up when the customer switched from French to English.View Result →

Its French answer gave a 21-day standard return window and 45 days for Cedarline Plus members; asked in English, it answered in English, confirmed the 45-day Plus window and offered to work out the exact last return day.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Respond.ioPassed1 of 1 test case passedContinued in English without losing the return-policy context after the customer changed languages.View Result →

It kept the 45-day Plus return window from delivery and explained that orders placed during active membership still qualify if they arrive after cancellation.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (10): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Intercom Fin, Kommunicate, Tidio Lyro.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Multilingual support

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Customer switches languages during the conversation in AI Customer Support Chatbots | AI Demos