Tidio Lyro in AI Customer Support Chatbots
Scenario-level performance from current published Results.
10 of 26 scenarios published
Tidio Lyro passed every graded scenario it has a published Result for.
How Tidio Lyro performed
Open a capability to explore its scenarios. Each row reports the test set in its published Result.
Agent configuration and control2 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Configured instruction changes agent behavior | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedTidio Lyro followed a custom instruction when answering a return request. It refused an opened-candle return without suggesting that the item could still be returned. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Configured restriction is respected | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedTidio Lyro refused the delivery-address change without performing or offering any part of it. It explained that address changes were unavailable for any order, regardless of status. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Analytics and observability1 pass2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Conversation activity can be inspected | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedTidio Lyro let an operator inspect an earlier customer conversation from start to finish. The shipped-order chat appeared in its conversation log, with order details and the return policy available in the transcript. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Agent actions/tool calls can be inspected | No published result | ||
| Reported analytics match what actually happened | No published result | ||
Escalation and human handoff2 pass1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Agent cannot resolve the request | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedTidio Lyro acknowledged it could not resolve the account email change and directed the customer to human help. It supplied support options without inventing self-service steps or claiming to have made the change. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| User explicitly asks for a human | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedTidio Lyro passed the human-handoff scenario. The customer asked for a human, Lyro answered once with a direct transfer, and the same conversation then reached a human agent in Tidio's inbox. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A configured rule requires escalation | No published result | ||
Intent understanding3 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| One message contains multiple requests | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedTidio Lyro handled the one tested multi-request message successfully. It answered the order status and return policy in one reply, and the order details matched the live order data while the policy block covered the requested rules. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Request is unclear and needs clarification | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedTidio Lyro sought clarification rather than acting on an unclear cancellation request. It offered the customer a choice of an order, a membership or a monthly product subscription. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Request requires choosing the correct source or action | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedTidio Lyro used the order system, not the knowledge base, and correctly reported the order as shipped. Its answer included tracking information and an estimated delivery date. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Knowledge-grounded answering2 different test set1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Answer is available in the knowledge base | Different test setPublished Result available, but not included in this comparison. | View Result → | |
| Answer is not available in the knowledge base | Different test setPublished Result available, but not included in this comparison. | View Result → | |
| The request uses different wording than the source | No published result | ||
Action execution5 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Action requires confirmation | No published result | ||
| Requested action cannot be completed | No published result | ||
| Retrieve information from an external system | No published result | ||
| Update something in an external system | No published result | ||
| User is not authorized to perform the action | No published result | ||
Conversation continuity3 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Customer corrects information given earlier | No published result | ||
| Earlier conversation/session needs to be continued | No published result | ||
| Information from earlier in the conversation is needed later | No published result | ||
Feedback and learning2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Correction changes future behavior where the product claims learning | No published result | ||
| Feedback can be submitted | No published result | ||
Multilingual support2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Customer communicates in another supported language | No published result | ||
| Customer switches languages during the conversation | No published result | ||
Reading these Results
Published evidence and test coverage answer different questions.
Which scenarios have a Result?
A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.
What does each Result cover?
Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.
Out of 26 scenarios
The summary counts each of the 26 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Tidio Lyro was tested on it.