Tool in benchmark · Version 1

LlamaParse in Structured Document Extraction

Scenario-level performance from current published Results.

13Passed
5Failed
2No published Result yet

18 of 20 scenarios published

CapabilityScenariosResult
Field Extraction5 pass · 0 fail · 5 of 5 published
Nested & Repeated Fields4 pass · 0 fail · 4 of 4 published
OCR2 pass · 0 fail · 2 of 2 published
Querying0 pass · 2 fail · 2 of 3 published
Review and Correction0 pass · 3 fail · 3 of 3 published
Source Grounding2 pass · 0 fail · 2 of 2 published
Training0 of 1 published
PassedFailedPublished, different test setNo published Result yet

Where LlamaParse failed

5 of 18 graded scenarios. Open a Result to see the conversation and the evidence.

Querying

Aggregate across documents

LlamaParse did not produce the required combined total of 474.93 across three receipts in Extract on the Free plan. It filled the requested cross-document sum with individual receipt totals instead.

View Result →
Querying

The answer was never in the schema

LlamaParse offered no place to ask about a fact printed on the receipt that the extraction schema left out. Neither the Parse result screen and its menu nor the Extract results panel has a question or chat box, and Chat cannot start until an index exists.

View Result →
Review and Correction

Correct a field value

LlamaParse offered no way to correct a returned field. No edit control was found in the Results view, JSON view, History menu or Details drawer, and the saved receipt number stayed RHS-70234.

View Result →
Review and Correction

Corrected data goes downstream

LlamaParse did not carry the corrected receipt_number RHS-99999 downstream on the Extract screen. On the Free plan, the agentic four-field setup stayed read-only, and the downloaded JSON and opened export still showed receipt_number RHS-70234 with subtotal, tax, and total_due.

View Result →
Review and Correction

Repair a record

LlamaParse did not let the user add or remove a member of an extracted set. The saved result stayed at three line items: Site fencing remained, and Concrete mixer was absent after the edit attempts.

View Result →

How LlamaParse performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Field Extraction5 pass
ScenarioPublished outcomesTest coverageResult
The field is absent
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse passed this absent-field case by leaving the missing loyalty field empty and ungrounded. The receipt has no loyalty wording, the extractor returned an empty value for that field, and the job record gave that field no citation metadata while the present receipt number had a citation.

1/1 assessed1/1 gradablePublished test setView Result →
The same field across layouts
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse passed this scenario. It returned the correct identifier, subtotal, tax, and total_due values for all three differently laid out documents, and the three outputs matched the values read from the pages while keeping the same four-field schema.

1/1 assessed1/1 gradablePublished test setView Result →
The value is directly available
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse correctly extracted the directly available receipt number as text. It also returned subtotal, tax and total due as $85.00, $5.10 and $90.10, respectively—the same amounts printed on the receipt.

1/1 assessed1/1 gradablePublished test setView Result →
The value must be derived
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse passed this derived-value task. It read the two page-1 operands and returned £81.00; the derived field had no citation, and the source contains no 81.00.

1/1 assessed1/1 gradablePublished test setView Result →
The value needs a supplied definition
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse returned the consignment reference CFS-CN-40821, as the field definition asked, rather than the invoice number CFS-7710. The invoice number is printed more prominently in the header and on every page.

1/1 assessed1/1 gradablePublished test setView Result →
Nested & Repeated Fields4 pass
ScenarioPublished outcomesTest coverageResult
A repeated set of records
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse handled the repeated set correctly. It returned all three records with the right values, and the per-value citations stayed in separate rows, so nothing was bound to the wrong record.

1/1 assessed1/1 gradablePublished test setView Result →
The set contains a non-record
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse left out the invoice’s Total row carrying 1,982.00 rather than treating it as an item. It returned five items, preserving all five genuine rows.

1/1 assessed1/1 gradablePublished test setView Result →
The set continues across a page break
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse kept all ten invoice line items together in one array across the page break. It retained the six items from page one and four from page two, with no boundary duplicate and all 40 values correct.

1/1 assessed1/1 gradablePublished test setView Result →
The set is legitimately empty
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse passed this empty-set extraction case. It returned an empty required array for the damage-exceptions field, with the source page saying nothing was reported and no invented member or marker text added.

1/1 assessed1/1 gradablePublished test setView Result →
OCR2 pass
ScenarioPublished outcomesTest coverageResult
A cleanly scanned document
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse passed this OCR baseline. It returned the same invoice values for the digital file and the image-only twin, including the page 2 totals, and the twin had no text layer. The two runs differed only in job details such as timestamps and confidence flags.

1/1 assessed1/1 gradablePublished test setView Result →
A document mixing digital and scanned pages
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse extracted the requested values from a document mixing digital and scanned pages. It returned all five values correctly, including two found only on the scanned page.

1/1 assessed1/1 gradablePublished test setView Result →
Querying2 fail1 no Result yet
ScenarioPublished outcomesTest coverageResult
Aggregate across documents
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

LlamaParse did not produce the required combined total of 474.93 across three receipts in Extract on the Free plan. It filled the requested cross-document sum with individual receipt totals instead.

1/1 assessed1/1 gradablePublished test setView Result →
The answer was never in the schema
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

LlamaParse offered no place to ask about a fact printed on the receipt that the extraction schema left out. Neither the Parse result screen and its menu nor the Extract results panel has a question or chat box, and Chat cannot start until an index exists.

1/1 assessed1/1 gradablePublished test setView Result →
Filter records across documentsNo published result
Review and Correction3 fail
ScenarioPublished outcomesTest coverageResult
Correct a field value
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

LlamaParse offered no way to correct a returned field. No edit control was found in the Results view, JSON view, History menu or Details drawer, and the saved receipt number stayed RHS-70234.

1/1 assessed1/1 gradablePublished test setView Result →
Corrected data goes downstream
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

LlamaParse did not carry the corrected receipt_number RHS-99999 downstream on the Extract screen. On the Free plan, the agentic four-field setup stayed read-only, and the downloaded JSON and opened export still showed receipt_number RHS-70234 with subtotal, tax, and total_due.

1/1 assessed1/1 gradablePublished test setView Result →
Repair a record
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

LlamaParse did not let the user add or remove a member of an extracted set. The saved result stayed at three line items: Site fencing remained, and Concrete mixer was absent after the edit attempts.

1/1 assessed1/1 gradablePublished test setView Result →
Source Grounding2 pass
ScenarioPublished outcomesTest coverageResult
The field is absent
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse passed this absent-field case by leaving the missing loyalty field empty and ungrounded. The receipt has no loyalty wording, the extractor returned an empty value for that field, and the job record gave that field no citation metadata while the present receipt number had a citation.

1/1 assessed1/1 gradablePublished test setView Result →
Trace a value to its location
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse traced the value to the right place in the document. The value appears once in the two-page invoice; the citation names page 2 and the viewer box lands on the value.

1/1 assessed1/1 gradablePublished test setView Result →
Training1 no Result yet
ScenarioPublished outcomesTest coverageResult
A field the tool can only get right from examplesNo published result

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 19 scenarios

The summary counts each of the 19 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether LlamaParse was tested on it.

LlamaParse in Structured Document Extraction | AI Demos