AI Demos Research — the structured-intelligence platform. Every verdict on these pages opens to the execution behind it.
superseded result

This is the result as it was published on 18 November 2026. A newer result replaced it on 19 November 2026; this page is kept so that anything that cited it still resolves. Read the current result →

Graded 17 November 2026

Does Botpress answer customer questions when the answer is in its knowledge base?

Botpress answered shipping, returns and damaged-item questions with the knowledge-base policy figures and remedies. It gave Canada's free-shipping threshold as over $120 and return periods of 21 days, or 45 for Plus members. For damage, it requested photos, required reporting within 7 days of delivery and described a replacement if available or a full refund including original shipping, subject to review.

3 of 3 test cases passed

Every test case in this scenario has a result.

Pass rate100%3 of 3 gradable
Coverage3 of 3test cases with a result
3PassDid everything it was expected to do.
0FailNo test case failed.
0Not gradableEvery result here could be graded.
0UntestedEvery test case in this scenario has a result.
The pass rate is a summary. The evidence is the 3 test cases below: what we sent, what we checked, what the tool returned, and the proof.

The 3 test cases

Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on these test cases.

Free-shipping threshold asked for CanadaPassEvidence
What we sent

“I'm in Canada — how much do I need to spend to get free shipping?”

What the tool returned

In Canada, standard shipping is free on orders over $120. Since you're a Cedarline Plus member, you receive free standard shipping on every order, regardless of the order total.

Transcribed from Proof 1 · Screenshot

Expected vs. Found
✓Expected: State that orders over $120 qualify for free shipping in Canada.Found: It stated that standard shipping in Canada is free on orders over $120, the required Canadian threshold.
✓Must not appear: Do not present the US threshold of $75, or another figure, as Canada's free-shipping threshold.Found: It gave no competing Canadian threshold and did not present the US threshold of $75 as Canada's.
Supporting proof
Proof 1Screenshot
Chat screenshot: the Canada shipping question and the threshold in Botpress's reply
Open original ↗
Why this result

The answer met the Canada-specific requirement without substituting another region's figure. Its separate Plus-member benefit did not introduce a competing Canadian threshold.

Also observed in this test case

No shipping source link. The Canada shipping reply has no knowledge-base source link identifying the article behind the $120 threshold.

Tested by Ajay Vekhande · evidence dated 11 September 2026
Return window questionPassEvidence
What we sent

“What is your return window?”

What the tool returned

I'm checking Cedarline's current return policy so I can give you the accurate return window. Cedarline's standard return window is 21 days. As a Cedarline Plus member, you receive an extended 45-day return window instead. Damaged, defective, or incorrect items are handled separately and must be reported within 7 days of delivery, with photos of the item and packaging.

Transcribed from Proof 1 · Screenshot

Expected vs. Found
✓Expected: State a standard return window of 21 days from delivery.Found: It stated a 21-day standard return window, the policy's required length.
✓Expected: State a return window of 45 days from delivery for Plus members.Found: It stated a 45-day return window for Plus members, the policy's required length.
✓Must not appear: Do not add a different return period, an order-date starting point or an unsupported or altered return condition.Found: It added the policy's separate rule for damaged, defective or incorrect items: report within 7 days of delivery with photos, without unsupported return terms.
Supporting proof
Proof 1Screenshot
Chat screenshot: standard and Plus return periods, with knowledge-base source links
Open original ↗
Why this result

The answer supplied both required return periods without adding unsupported policy claims. It omitted their delivery-date starting point rather than giving a conflicting one.

Also observed in this test case

Answer despite search error. The return-window reply appeared after a search-failure notice and still listed two knowledge-base sources.

Tested by Ajay Vekhande · evidence dated 11 September 2026
Damaged-on-arrival options with their conditionPassEvidence
What we sent

Input 1 · Damaged-item question: “My rug arrived damaged. What are my options?”

Input 2 · Order details: “Order number 41290 Name Cotton waffle comforter, king — Sand”

Input 3 · Delivery date: “delivery date 5th September 2026”

What the tool returned

Request for order details

I'm sorry your rug arrived damaged. I can help review replacement or refund options. Please provide your order number and the name or description of the rug.

Follow-up question and photo instructions

Thank you. For order 41290—the Cotton Waffle Comforter, King — Sand—when was the purchase made? Since the item arrived damaged, please also keep photos of the comforter and its packaging available; damaged items must be reported within 7 days of delivery.

Final answer: remedies and conditions

Thank you. Your Cotton Waffle Comforter, King — Sand from order 41290 was delivered on September 5, 2026, and the damage was reported within Cedarline's 7-day reporting period. Because it arrived damaged: The standard return window and restocking fee do not apply. Please provide photos of the comforter and its packaging. After review, Cedarline may offer a replacement if available or a full refund, including the original shipping cost. Your return request is eligible for review. If a refund is approved, it is issued after warehouse inspection—typically within 5–7 business days of receipt. Card refunds generally appear within another 5–7 business days; PayPal refunds take 3–5 business days. Please attach clear photos of the damaged comforter and its packaging so the return request can proceed to review. Once received, Cedarline can assess whether a replacement is available or approve a full refund including the original shipping cost.

Transcribed from Proof 1 · Screen recording at 0:28

Every message the tool sent, in order (3)
  1. Request for order details

    Printed above.

  2. Follow-up question and photo instructions

    Printed above.

  3. Final answer: remedies and conditions

    The final response, printed above.

Expected vs. Found
✓Expected: After review, Cedarline may offer a replacement if available or a full refund including original shipping.Found: It said Cedarline may offer an available replacement or a full refund including original shipping, after review.
✓Expected: A photo of the damage must be provided within 7 days.Found: It required damage to be reported within 7 days of delivery and requested photos of the item and packaging.
✓Must not appear: Do not add a condition, deadline or remedy beyond the knowledge-base policy.Found: It stated policy-backed terms: no standard return window or restocking fee; refunds after warehouse inspection, typically within 5–7 business days of receipt; cards another 5–7, PayPal 3–5 business days.
Supporting proof
Proof 1Screen recording
Screen recording: order details, delivery date and Botpress's conditional damaged-item remedies
Key moments
Open original ↗
Why this result

The knowledge-base policy leaves the remedy to Cedarline's discretion. After receiving order details and a delivery date, Botpress gave the policy's conditional replacement-or-refund options and its photo and reporting requirements.

Tested by Ajay Vekhande · evidence dated 11 September 2026

Configuration and setup

How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.

Software that produced the output
Botpress
Build or version
Not exposed by the hosted product
Plan or tier
Free plan
Surface
Web app
Agent publication state
Free-shipping threshold asked for Canada: Free-shipping threshold asked for Canada: previewing 16 unpublished changes; Damaged-on-arrival options with their condition: Damaged-on-arrival options with their condition: Not published yet
Surface mode
Damaged-on-arrival options with their condition: Damaged-on-arrival options with their condition: Preview
Tested
11 September 2026 · Ajay Vekhande
Connected to
Knowledge BaseCedarline knowledge base · version 2Test cases: Free-shipping threshold asked for Canada, Damaged-on-arrival options with their condition
Knowledge BaseCedarline knowledge base · version 1Test cases: Return window question

How this scenario is graded

How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.

How results are decided

Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.

The rules
  • Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
  • Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
  • Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.

Where this sits in the benchmark

This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.

History of this result

What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.

from the publication record
18 November 2026First publishedAI Customer Support Chatbots v1
19 November 2026Updated — the page was revised
The result is unchanged: 3 pass · 0 fail. The earlier result stays readable. Read the earlier result →

Act on this result

Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.

This matches what I see

You run the same kind of test against your own setup and get the same behaviour.

Agree →
This does not match

Yours behaves differently. Tell us what you got, with a screenshot if you have one.

Disagree →
Point out an issue

Something here is wrong — a reference value, a transcription, a grade.

Report an issue →
Request a retest

On a newer build, a larger dataset, or your own setup.

Request a retest →
We have fixed this

Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.

Vendor notice →
Filed against this evidenceNothing yet. Challenges, counter-evidence and fix notices appear here with their outcome, and stay on the page after they are resolved.
Cite this result
aidemos.com/benchmarks/ai-customer-support-chatbots/results/botpress/answer-is-available-in-the-knowledge-base · 3 pass · 0 fail · coverage 3/3 · graded 2026-11-17

The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.

Verify the proof files

These files support this result. Open a file to inspect the original evidence.

File fingerprints (SHA-256)

A fingerprint identifies the exact file used for this result.

Free-shipping threshold asked for Canada · Proof 1 · Screenshot3b8537998ea605f4b79b6672da3d8e9f38de47f1158df0714fb4718450963e04
Return window question · Proof 1 · Screenshot0f69a0fef4010148fcb93fb98e10e33228c0b3b4542e82377b3e62a462cba13b
Damaged-on-arrival options with their condition · Proof 1 · Screen recordingbf062ba30e2e3c8336129a04585fcceea2d594ff252f20362bf381ffeb42317e