Tool in benchmark · Version 1

Vectara in Managed RAG

Scenario-level performance from current published Results.

13Passed
0Failed
2No published Result yet

13 of 15 scenarios published

CapabilityScenariosResult
Citations1 pass · 0 fail · 1 of 1 published
Data Ingestion3 pass · 0 fail · 3 of 4 published
Grounded Generation3 pass · 0 fail · 3 of 3 published
Knowledge Sync2 pass · 0 fail · 2 of 3 published
Retrieval4 pass · 0 fail · 4 of 4 published
PassedFailedPublished, different test setNo published Result yet

Vectara passed every graded scenario it has a published Result for.

How Vectara performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Citations1 pass
ScenarioPublished outcomesTest coverageResult
A claim in the answer traced back to its source
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara cited a passage that supported its answer's claim. It said a named duty engineer would acknowledge the out-of-hours fault within 90 minutes, and the cited passage said the same.

1/1 assessed1/1 gradablePublished test setView Result →
Data Ingestion3 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
A bad file in the upload
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara reported files it could not process during upload. It returned errors for a truncated PDF and a password-protected PDF; the console also showed the truncated file's failure.

1/1 assessed1/1 gradablePublished test setView Result →
A simple, clearly formatted table
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara correctly answered the row-and-column lookup with 487 for an Extended contract’s out-of-hours callout charge. The answer cited the passage containing the charges table.

1/1 assessed1/1 gradablePublished test setView Result →
An ordinary digital text document
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara returned the requested fact in the document's own words. In both configurations, the top result was the passage stating a 22-month recalibration interval, and the next results were different warranty passages; both captures showed HTTP 200.

1/1 assessed1/1 gradablePublished test setView Result →
A cleanly scanned documentNo published result
Grounded Generation3 pass
ScenarioPublished outcomesTest coverageResult
The retrieved material doesn't support an answer
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara declined to answer a question its retrieved material did not support: it said it did not have enough information about how much paid parental leave Fenwake gives and offered no figure or policy. None of the ten passages it retrieved mentions parental leave.

1/1 assessed1/1 gradablePublished test setView Result →
The retrieved material supports an answer
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara gave the supported 90-minute acknowledgement time for an out-of-hours fault. It did not mix in the competing 30-minute target from another retrieved passage.

1/1 assessed1/1 gradablePublished test setView Result →
The retrieved sources disagree with each other
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara said it could not determine the warranty period on a new Tolvan A3 because its search results conflict, and named neither 24 nor 36 months as the fact. Both conflicting warranty documents were among the passages it retrieved, at ranks 1 and 2.

1/1 assessed1/1 gradablePublished test setView Result →
Knowledge Sync2 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
Content is removed from the source
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara stopped using the deleted source and declined to answer the question. Across four repeat queries, the deleted document was absent from the top ten results and the supplier was not cited.

1/1 assessed1/1 gradablePublished test setView Result →
New content is added to the source
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara made a newly added document findable right away: before the add no retrieved passage mentioned the Tromso depot, and afterwards all four identical queries retrieved the passage giving its opening date. Only two of those four answers stated the date; the other two declined.

1/1 assessed1/1 gradablePublished test setView Result →
Existing content is changedNo published result
Retrieval4 pass
ScenarioPublished outcomesTest coverageResult
Answer is available in the knowledge base
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara returned the answer from the knowledge base for this naturally phrased question. The out-of-hours passage ranked first in three independent captures, including a live notebook run, while the business-hours near-miss stayed lower each time. The corpus view shows the relevant document was present.

1/1 assessed1/1 gradablePublished test setView Result →
Answer is not available in the knowledge base
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara signaled the unavailable paid-parental-leave answer through clearly lower relevance scores. In a second default-reranker call with no context window specified, the top score was 0.0820, versus 0.9986 for an answerable question on the same documents.

1/1 assessed1/1 gradablePublished test setView Result →
Retrieval is scoped to part of the documents
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara kept retrieval within the requested document subset without returning an empty result. The document containing the answer ranked first in an unrestricted search and was absent from the restricted results.

1/1 assessed1/1 gradablePublished test setView Result →
The request uses different wording than the source
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Vectara returned the matching passage even though the question used different wording from the source. In both retrieval runs, the annual leave passage was rank 1, and the returned passages did not contain the query's alternative terms.

1/1 assessed1/1 gradablePublished test setView Result →

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 15 scenarios

The summary counts each of the 15 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Vectara was tested on it.

Vectara in Managed RAG | AI Demos