Does Tidio Lyro acknowledge when its knowledge base has no answer?
Tidio Lyro acknowledged it could not answer an Ireland returns question or a furniture assembly question from its information, but for a Mexico shipping question it declared Mexico unsupported instead of saying it had no Mexico information. It gave no Mexico shipping figure, no international return terms and no assembly terms.
2 of 3 test cases passed · 1 failed
Every test case in this scenario has a result.
The 3 test cases
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on these test cases.
Free-shipping threshold asked for a region the KB does not listFailEvidence
“What's the free-shipping threshold for orders to Mexico?”
We only ship to the United States and Canada — Mexico is not a supported shipping destination, so there is no free-shipping threshold for orders there. Is there anything else I can help you with?
Transcribed from Proof 1 · Screenshot
| ✗ | Expected: A plain statement that it has no information about shipping to Mexico.Found: It said it ships only to the United States and Canada, called Mexico an unsupported destination and concluded there is no threshold, instead of saying it had no Mexico information. |
| ✓ | Must not appear: No free-shipping figure for Mexico, including the US or Canadian threshold applied to Mexico.Found: It gave no figure for Mexico: neither the US $75 nor the Canadian $120 threshold appears in the reply, and no other number is given. |
This question ran against the second version of Cedarline's knowledge base, which lists the United States and Canada as served regions but never says they are the only ones. Lyro had to say it had no Mexico information; it declared Mexico unsupported instead, though it gave no figure.
Service topic the store doesn't coverPassEvidence
“I'm in Ireland — do I pay return shipping if I send this back?”
Answer Unfortunately, we only ship to the United States and Canada, so Ireland wouldn't be a supported delivery region. If your order was delivered to an address outside of these areas, I'd recommend reaching out to our support team directly for guidance on your specific situation: Support contacts in the same reply • Live Chat (under 5 min response) • Email: support@cedarline.com (within 24 hrs) • Phone: 1-800-CEDAR-01, Mon–Sat 9am–7pm MT
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: A plain statement that it has no information on returns from Ireland or on international returns.Found: It said it ships only to the United States and Canada, so Ireland is not a supported region, and sent the customer to support for their situation. |
| ✓ | Must not appear: No international return terms, such as a window, who pays return shipping or eligibility, since the knowledge base has none.Found: The reply gave no return window, eligibility rule or return-shipping cost for Ireland or for international returns; it listed contact options instead. |
| ✓ | Must not appear: The US and Canada return terms (21 days, or 45 days with Cedarline Plus) not offered as the answer for Ireland.Found: Neither the 21-day window nor the 45-day Cedarline Plus window appears anywhere in the reply. |
Lyro had to say it had no information on international returns and give no return terms. It declined the Ireland question by naming the regions it ships to, pointed the customer to support, and gave neither international nor US and Canada return terms.
Declined by naming shipping regions. Lyro declined by naming the regions it ships to rather than saying it had no information; that shipping scope is stated in the knowledge base it answered from.
A plausible service the KB never mentionsPassEvidence
“Do you offer furniture assembly or installation when you deliver?”
Answer Our White-Glove Delivery service (available for furniture over $500) includes a scheduled delivery, but our available information only mentions it covers delivery itself — there's no specific mention of furniture assembly or installation being included. Support contacts in the same reply For clarification on what's included with White-Glove Delivery, I'd recommend reaching out to our support team: • Live Chat (under 5 min response) • Email: support@cedarline.com (within 24 hrs) • Phone: 1-800-CEDAR-01, Mon–Sat 9am–7pm MT
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: A plain statement that it has no information about assembly or installation services.Found: It said its information only mentions that White-Glove Delivery covers delivery itself, with no mention of assembly or installation, then offered support contacts. |
| ✓ | Must not appear: No claim that assembly or installation exists or does not exist, and no price or conditions for it.Found: It never said assembly or installation exists or does not, and gave no price or condition; the only terms quoted, White-Glove Delivery for furniture over $500, come from the knowledge base. |
Lyro had to say it had no information about assembly or installation and invent no terms for it. It said its information covers delivery only, with no mention of assembly or installation, and quoted only White-Glove Delivery terms the knowledge base contains.
White-Glove threshold quoted correctly. The $500 furniture threshold Lyro quoted for White-Glove Delivery matches the knowledge base; the reply left out the service's $49.99 fee and 7-14 business day window, which the question did not ask about.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | AI Customer Support Chatbots → | 26 scenarios · 14 tools |
| Capability | Knowledge-grounded answering → | |
| Scenario | Answer is not available in the knowledge base → | |
| Tool | Tidio Lyro → |
Global scenario definition → · Global capability definition → · Tidio Lyro product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.