Scenario in benchmark · Managed RAG · Version 1

Retrieval is scoped to part of the documents

When retrieval is limited to a defined subset of documents, it should stay inside that scope.

Capability: Retrieval1 pinned test case per toolNeutral global definition →

What we check

  1. Whether returned results stay within the allowed subset when search is restricted.
  2. Whether the answer-bearing document outside the allowed subset is excluded from retrieval.
  3. Whether the system treats a narrow scope as a true no-result case rather than a successful hit.
  4. Whether the restriction is respected instead of being silently ignored.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
FutureSmart Document IntelligenceFailed1 of 1 test case failedFailed to keep retrieval scoped to the subset that did not hold the answer.View Result →

Both restricted searches still returned the answering depot note first at score 1.0000.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 3
RAGFlowPassed1 of 1 test case passedRespected the document restriction, returning three Nordic service passages without the depot note that held the answer.View Result →

The depot note ranked first in both the unrestricted search and the UK-only comparison search.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

VectaraPassed1 of 1 test case passedKept retrieval within the requested document subset without returning an empty result.View Result →

The document containing the answer ranked first in an unrestricted search and was absent from the restricted results.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Vectorize.ioPassed1 of 1 test case passedReturned only Nordic passages when the search was limited to Nordic-region documents, leaving out the UK document containing the answer.View Result →

The same search without the region limit did return that document.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): Denser AI, Dify, Llama Cloud, Onyx.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Retrieval

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Retrieval is scoped to part of the documents in Managed RAG | AI Demos