Does Respond.io answer questions when the answer is in its knowledge base?
Respond.io answered three policy questions with the knowledge base’s shipping, return and damage terms. It gave Canada’s free standard shipping threshold as over $120 and return windows of 21 days from delivery, or 45 days for Plus members. For damage, it offered replacement subject to stock or a full refund including original shipping, with photos required within 7 days of delivery.
3 of 3 test cases passed
Every test case in this scenario has a result.
The 3 test cases
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on these test cases.
Free-shipping threshold asked for CanadaPassEvidence
“I'm in Canada — how much do I need to spend to get free shipping?”
For Canadian orders, free standard shipping applies when you spend over $120. Cedarline Plus members get free standard shipping on every order.
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: State that Canadian orders over $120 qualify for free shipping.Found: It gives the required Canadian threshold of over $120 for free standard shipping, with free standard shipping on every order for Plus members. |
| ✓ | Must not appear: Do not present the US threshold of $75 or any other amount as Canada’s threshold.Found: The reply’s only amount is $120; it does not give the US threshold of $75 or another Canadian threshold. |
Free standard shipping had to be available above $120 in Canada. The answer gives that threshold without introducing a competing figure, so both requirements are met.
Return window questionPassEvidence
“What is your return window?”
Our standard return window is 21 days from delivery; Cedarline Plus members have 45 days. Anything else I can help you with?
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: State a standard return window of 21 days from delivery.Found: The answer gives 21 days from delivery for standard returns, the period required by the knowledge base. |
| ✓ | Expected: State that Plus members have 45 days from delivery to return an item.Found: It gives Plus members 45 days, with delivery as the starting point, as required by the knowledge base. |
| ✓ | Must not appear: Do not add or change return-policy terms; any item-condition statement must preserve the knowledge base’s meaning.Found: It adds no unsupported return terms: no alternative period, order-date starting point or item-condition clause. |
The answer supplies the two required return periods, anchors both to delivery rather than order placement, and adds no unsupported policy terms.
Damaged-on-arrival options with their conditionPassEvidence
“My rug arrived damaged. What are my options?”
I'm sorry your rug arrived damaged. Please report it within 7 days of delivery and share photos of the rug and its packaging; damaged items aren't subject to the usual return window or restocking fee. Depending on stock, you may be eligible for a replacement or a full refund including original shipping. When was it delivered?
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: Offer replacement or refund as the customer’s choice, with replacement subject to stock.Found: It says the customer may qualify for a replacement, subject to stock, or a full refund including original shipping, as the knowledge base provides. |
| ✓ | Expected: Require a photo of the damage within 7 days of delivery.Found: It requires reporting within 7 days of delivery and photos of the rug and packaging, preserving the knowledge base’s deadline and photo requirement. |
| ✓ | Must not appear: Do not introduce a damage-policy condition, deadline or remedy absent from the knowledge base.Found: It says the usual return window and restocking fee do not apply, consistent with the knowledge base’s 15% fee exemption, without adding unsupported policy terms. |
The answer offers both remedies, keeps the stock condition on replacement and the 7-day photo requirement, and states the return-window and fee exemptions without adding unsupported policy terms.
Delivery-date follow-up. The damaged-rug reply ends by asking when the rug was delivered.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | AI Customer Support Chatbots → | 26 scenarios · 14 tools |
| Capability | Knowledge-grounded answering → | |
| Scenario | Answer is available in the knowledge base → | |
| Tool | Respond.io → |
Global scenario definition → · Global capability definition → · Respond.io product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.