Scenario in benchmark · AI Customer Support Chatbots · Version 1
Configured instruction changes agent behavior
A configured instruction is applied to the agent and changes how it responds.
What we check
- Whether the agent's response reflects the applied instruction.
- Whether the response changes from its default behaviour when the instruction is in force.
- Whether the agent follows the configured instruction on the relevant request.
- Whether the configured instruction is observable in the resulting behaviour.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
BotpressPassed1 of 1 test case passedFollowed the configured instruction in the tested chat.View Result →
The configuration says opened home-decor items are not returnable, and the reply says the opened candle is not eligible for return, matching that rule.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Freshdesk FreddyPassed1 of 1 test case passedFollowed the configured instruction when answering a request to return an opened candle.View Result →
It said the candle was not returnable and cited the policy against returning opened home decor items.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Intercom FinPassed1 of 1 test case passedFollowed the configured instruction in this case.View Result →
It told the customer opened candles were final sale and did not offer a refund, matching the live guidance for home decor items on all channels.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
KommunicatePassed1 of 1 test case passedFollowed the configured instruction in this scenario.View Result →
The store policy said opened candles could not be returned, and the chat reply gave that same rule to the customer.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Respond.ioPassed1 of 1 test case passedFollowed the requested behavior by offering store credit before any cash refund.View Result →
It asked when the cushions were delivered and whether the customer wanted store credit.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Tidio LyroPassed1 of 1 test case passedFollowed a custom instruction when answering a return request.View Result →
It refused an opened-candle return without suggesting that the item could still be returned.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Zendesk AIPassed1 of 1 test case passedFollowed the configured instruction in this return chat.View Result →
The instruction was active, and the customer’s candle-return request got a polite refusal that named Home Decor and did not start or approve a return.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (7): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Agent configuration and control
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →