AI Demos Research — the structured-intelligence platform. Every verdict on these pages opens to the execution behind it.
Graded 17 November 2026

Does Respond.io answer questions when the answer is in its knowledge base?

Respond.io answered three policy questions with the knowledge base’s shipping, return and damage terms. It gave Canada’s free standard shipping threshold as over $120 and return windows of 21 days from delivery, or 45 days for Plus members. For damage, it offered replacement subject to stock or a full refund including original shipping, with photos required within 7 days of delivery.

3 of 3 test cases passed

Every test case in this scenario has a result.

Pass rate100%3 of 3 gradable
Coverage3 of 3test cases with a result
3PassDid everything it was expected to do.
0FailNo test case failed.
0Not gradableEvery result here could be graded.
0UntestedEvery test case in this scenario has a result.
The pass rate is a summary. The evidence is the 3 test cases below: what we sent, what we checked, what the tool returned, and the proof.

The 3 test cases

Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on these test cases.

Free-shipping threshold asked for CanadaPassEvidence
What we sent

“I'm in Canada — how much do I need to spend to get free shipping?”

What the tool returned

For Canadian orders, free standard shipping applies when you spend over $120. Cedarline Plus members get free standard shipping on every order.

Transcribed from Proof 1 · Screenshot

Expected vs. Found
✓Expected: State that Canadian orders over $120 qualify for free shipping.Found: It gives the required Canadian threshold of over $120 for free standard shipping, with free standard shipping on every order for Plus members.
✓Must not appear: Do not present the US threshold of $75 or any other amount as Canada’s threshold.Found: The reply’s only amount is $120; it does not give the US threshold of $75 or another Canadian threshold.
Supporting proof
Proof 1Screenshot
Chat screenshot: the Canadian free-shipping question and threshold
Open original ↗
Why this result

Free standard shipping had to be available above $120 in Canada. The answer gives that threshold without introducing a competing figure, so both requirements are met.

Tested by Ajay Vekhande · evidence dated 23 October 2026
Return window questionPassEvidence
What we sent

“What is your return window?”

What the tool returned

Our standard return window is 21 days from delivery; Cedarline Plus members have 45 days. Anything else I can help you with?

Transcribed from Proof 1 · Screenshot

Expected vs. Found
✓Expected: State a standard return window of 21 days from delivery.Found: The answer gives 21 days from delivery for standard returns, the period required by the knowledge base.
✓Expected: State that Plus members have 45 days from delivery to return an item.Found: It gives Plus members 45 days, with delivery as the starting point, as required by the knowledge base.
✓Must not appear: Do not add or change return-policy terms; any item-condition statement must preserve the knowledge base’s meaning.Found: It adds no unsupported return terms: no alternative period, order-date starting point or item-condition clause.
Supporting proof
Proof 1Screenshot
Chat screenshot: standard and Plus return windows from delivery
Open original ↗
Why this result

The answer supplies the two required return periods, anchors both to delivery rather than order placement, and adds no unsupported policy terms.

Tested by Ajay Vekhande · evidence dated 23 October 2026
Damaged-on-arrival options with their conditionPassEvidence
What we sent

“My rug arrived damaged. What are my options?”

What the tool returned

I'm sorry your rug arrived damaged. Please report it within 7 days of delivery and share photos of the rug and its packaging; damaged items aren't subject to the usual return window or restocking fee. Depending on stock, you may be eligible for a replacement or a full refund including original shipping. When was it delivered?

Transcribed from Proof 1 · Screenshot

Expected vs. Found
✓Expected: Offer replacement or refund as the customer’s choice, with replacement subject to stock.Found: It says the customer may qualify for a replacement, subject to stock, or a full refund including original shipping, as the knowledge base provides.
✓Expected: Require a photo of the damage within 7 days of delivery.Found: It requires reporting within 7 days of delivery and photos of the rug and packaging, preserving the knowledge base’s deadline and photo requirement.
✓Must not appear: Do not introduce a damage-policy condition, deadline or remedy absent from the knowledge base.Found: It says the usual return window and restocking fee do not apply, consistent with the knowledge base’s 15% fee exemption, without adding unsupported policy terms.
Supporting proof
Proof 1Screenshot
Chat screenshot: damaged-rug remedies, photo deadline and fee exemptions
Open original ↗
Why this result

The answer offers both remedies, keeps the stock condition on replacement and the 7-day photo requirement, and states the return-window and fee exemptions without adding unsupported policy terms.

Also observed in this test case

Delivery-date follow-up. The damaged-rug reply ends by asking when the rug was delivered.

Tested by Ajay Vekhande · evidence dated 23 October 2026

Configuration and setup

How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.

Software that produced the output
Respond.io
Build or version
Not exposed by the hosted product
Plan or tier
Growth (trial plan)
Surface
Web app
Tested
23 October 2026 · Ajay Vekhande
Connected to
Knowledge BaseCedarline knowledge base · version 2Test cases: Free-shipping threshold asked for Canada, Damaged-on-arrival options with their condition
Knowledge BaseCedarline knowledge base · version 1Test cases: Return window question

How this scenario is graded

How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.

How results are decided

Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.

The rules
  • Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
  • Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
  • Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.

Where this sits in the benchmark

This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.

History of this result

What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.

from the publication record
18 November 2026First publishedAI Customer Support Chatbots v1

Act on this result

Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.

This matches what I see

You run the same kind of test against your own setup and get the same behaviour.

Agree →
This does not match

Yours behaves differently. Tell us what you got, with a screenshot if you have one.

Disagree →
Point out an issue

Something here is wrong — a reference value, a transcription, a grade.

Report an issue →
Request a retest

On a newer build, a larger dataset, or your own setup.

Request a retest →
We have fixed this

Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.

Vendor notice →
Filed against this evidenceNothing yet. Challenges, counter-evidence and fix notices appear here with their outcome, and stay on the page after they are resolved.
Cite this result
aidemos.com/benchmarks/ai-customer-support-chatbots/results/respond-io/answer-is-available-in-the-knowledge-base · 3 pass · 0 fail · coverage 3/3 · graded 2026-11-17

The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.

Verify the proof files

These files support this result. Open a file to inspect the original evidence.

File fingerprints (SHA-256)

A fingerprint identifies the exact file used for this result.

Free-shipping threshold asked for Canada · Proof 1 · Screenshotc3ca9d3070ff732e15bb60796c26d0f457bf7793b517d4c7b4166499d8d17e6b
Return window question · Proof 1 · Screenshotd6d75d3427c3e2e2addc8a4f76525ed3bb54ec6ababf18a34121ee3418180dc4
Damaged-on-arrival options with their condition · Proof 1 · Screenshot11d6326b2dba330ea6c030e87f467740b3560713b3dd00cb83fa8ebb418224bb