Tool in benchmark · Version 1

RAGFlow in Managed RAG

Scenario-level performance from current published Results.

12Passed
2Failed
1No published Result yet

14 of 15 scenarios published

CapabilityScenariosResult
Citations1 pass · 0 fail · 1 of 1 published
Data Ingestion3 pass · 1 fail · 4 of 4 published
Grounded Generation2 pass · 1 fail · 3 of 3 published
Knowledge Sync2 pass · 0 fail · 2 of 3 published
Retrieval4 pass · 0 fail · 4 of 4 published
PassedFailedPublished, different test setNo published Result yet

Where RAGFlow failed

2 of 14 graded scenarios. Open a Result to see the conversation and the evidence.

Data Ingestion

A bad file in the upload

RAGFlow did not flag the two unreadable uploads as document failures. A truncated PDF and a password-protected PDF received the same success status as a readable comparison document.

View Result →
Grounded Generation

The retrieved sources disagree with each other

RAGFlow chose one warranty period instead of flagging conflicting sources. Its answer gave 24 months, although it had retrieved documents stating 24 and 36 months.

View Result →

How RAGFlow performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Citations1 pass
ScenarioPublished outcomesTest coverageResult
A claim in the answer traced back to its source
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

RAGFlow’s out-of-hours Priority 1 answer cited text supporting all four claims, including acknowledgement within 90 minutes of the ticket being raised. The retrieved table giving 30 minutes was not cited.

1/1 assessed1/1 gradablePublished test setView Result →
Data Ingestion3 pass1 fail
ScenarioPublished outcomesTest coverageResult
A bad file in the upload
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

RAGFlow did not flag the two unreadable uploads as document failures. A truncated PDF and a password-protected PDF received the same success status as a readable comparison document.

1/1 assessed1/1 gradablePublished test setView Result →
A cleanly scanned document
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

RAGFlow recovered the requested fact from an image-only calibration certificate. Its first-ranked result contained an as-found bias of -0.037 psu, matching the reference value of -0.037 psu.

1/1 assessed1/1 gradablePublished test setView Result →
A simple, clearly formatted table
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

RAGFlow preserved the structure of a simple table on retrieval. Its first result placed the reference value, 487, at the requested row and column; it returned the table rather than a separate answer naming the value.

1/1 assessed1/1 gradablePublished test setView Result →
An ordinary digital text document
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

RAGFlow passes this plain-text document scenario. The ingested document contains the target sentence, and the default retrieval call returns that chunk first with a clear score gap.

1/1 assessed1/1 gradablePublished test setView Result →
Grounded Generation2 pass1 fail
ScenarioPublished outcomesTest coverageResult
The retrieved material doesn't support an answer
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

RAGFlow declined to answer when the retrieved material did not support an answer. Across four responses to a paid parental leave question, it said the information was unavailable and supplied no entitlement figure or policy.

1/1 assessed1/1 gradablePublished test setView Result →
The retrieved material supports an answer
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

RAGFlow gave a fully supported answer to the out-of-hours acknowledgement question. It stated the policy’s 90-minute target, and the cited passage supported all three claims.

1/1 assessed1/1 gradablePublished test setView Result →
The retrieved sources disagree with each other
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

RAGFlow chose one warranty period instead of flagging conflicting sources. Its answer gave 24 months, although it had retrieved documents stating 24 and 36 months.

1/1 assessed1/1 gradablePublished test setView Result →
Knowledge Sync2 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
Content is removed from the source
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

RAGFlow stopped using the supplier list after it was deleted. It had previously answered the supplier question with a citation to that list; afterwards, it said the information was not found and gave no citations.

1/1 assessed1/1 gradablePublished test setView Result →
Existing content is changed
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

After an indexed document was edited and processed again, RAGFlow returned the new value and no longer returned the old one. Asked the same lead-time question before and after the change, it returned 11 working days before and 6 working days after, and a second full cycle repeated that result.

1/1 assessed1/1 gradablePublished test setView Result →
New content is added to the sourceNo published result
Retrieval4 pass
ScenarioPublished outcomesTest coverageResult
Answer is available in the knowledge base
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

RAGFlow retrieved the passage answering the knowledge-base question. The first result contained the 90-minute out-of-hours acknowledgement target; the business-hours passage ranked fifth.

1/1 assessed1/1 gradablePublished test setView Result →
Answer is not available in the knowledge base
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

RAGFlow signalled the missing answer through lower relevance scores, not an explicit no-match message. Its ten returned passages did not address paid parental leave, and the highest score was 0.2865 versus 0.4826 for an answerable question.

1/1 assessed1/1 gradablePublished test setView Result →
Retrieval is scoped to part of the documents
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

RAGFlow respected the document restriction, returning three Nordic service passages without the depot note that held the answer. The depot note ranked first in both the unrestricted search and the UK-only comparison search.

1/1 assessed1/1 gradablePublished test setView Result →
The request uses different wording than the source
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

RAGFlow retrieved the supporting passage despite the question using different wording from the source. It ranked the annual-leave passage first in both requests.

1/1 assessed1/1 gradablePublished test setView Result →

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 15 scenarios

The summary counts each of the 15 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether RAGFlow was tested on it.

RAGFlow in Managed RAG | AI Demos