LlamaParse in Structured Document Extraction
Scenario-level performance from current published Results.
18 of 20 scenarios published
Where LlamaParse failed
5 of 18 graded scenarios. Open a Result to see the conversation and the evidence.
Aggregate across documents
LlamaParse did not produce the required combined total of 474.93 across three receipts in Extract on the Free plan. It filled the requested cross-document sum with individual receipt totals instead.
View Result →The answer was never in the schema
LlamaParse offered no place to ask about a fact printed on the receipt that the extraction schema left out. Neither the Parse result screen and its menu nor the Extract results panel has a question or chat box, and Chat cannot start until an index exists.
View Result →Correct a field value
LlamaParse offered no way to correct a returned field. No edit control was found in the Results view, JSON view, History menu or Details drawer, and the saved receipt number stayed RHS-70234.
View Result →Corrected data goes downstream
LlamaParse did not carry the corrected receipt_number RHS-99999 downstream on the Extract screen. On the Free plan, the agentic four-field setup stayed read-only, and the downloaded JSON and opened export still showed receipt_number RHS-70234 with subtotal, tax, and total_due.
View Result →Repair a record
LlamaParse did not let the user add or remove a member of an extracted set. The saved result stayed at three line items: Site fencing remained, and Concrete mixer was absent after the edit attempts.
View Result →How LlamaParse performed
Open a capability to explore its scenarios. Each row reports the test set in its published Result.
Field Extraction5 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The field is absent | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse passed this absent-field case by leaving the missing loyalty field empty and ungrounded. The receipt has no loyalty wording, the extractor returned an empty value for that field, and the job record gave that field no citation metadata while the present receipt number had a citation. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The same field across layouts | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse passed this scenario. It returned the correct identifier, subtotal, tax, and total_due values for all three differently laid out documents, and the three outputs matched the values read from the pages while keeping the same four-field schema. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The value is directly available | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse correctly extracted the directly available receipt number as text. It also returned subtotal, tax and total due as $85.00, $5.10 and $90.10, respectively—the same amounts printed on the receipt. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The value must be derived | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse passed this derived-value task. It read the two page-1 operands and returned £81.00; the derived field had no citation, and the source contains no 81.00. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The value needs a supplied definition | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse returned the consignment reference CFS-CN-40821, as the field definition asked, rather than the invoice number CFS-7710. The invoice number is printed more prominently in the header and on every page. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Nested & Repeated Fields4 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| A repeated set of records | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse handled the repeated set correctly. It returned all three records with the right values, and the per-value citations stayed in separate rows, so nothing was bound to the wrong record. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The set contains a non-record | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse left out the invoice’s Total row carrying 1,982.00 rather than treating it as an item. It returned five items, preserving all five genuine rows. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The set continues across a page break | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse kept all ten invoice line items together in one array across the page break. It retained the six items from page one and four from page two, with no boundary duplicate and all 40 values correct. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The set is legitimately empty | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse passed this empty-set extraction case. It returned an empty required array for the damage-exceptions field, with the source page saying nothing was reported and no invented member or marker text added. | 1/1 assessed1/1 gradablePublished test set | View Result → |
OCR2 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| A cleanly scanned document | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse passed this OCR baseline. It returned the same invoice values for the digital file and the image-only twin, including the page 2 totals, and the twin had no text layer. The two runs differed only in job details such as timestamps and confidence flags. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A document mixing digital and scanned pages | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse extracted the requested values from a document mixing digital and scanned pages. It returned all five values correctly, including two found only on the scanned page. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Querying2 fail1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Aggregate across documents | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedLlamaParse did not produce the required combined total of 474.93 across three receipts in Extract on the Free plan. It filled the requested cross-document sum with individual receipt totals instead. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The answer was never in the schema | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedLlamaParse offered no place to ask about a fact printed on the receipt that the extraction schema left out. Neither the Parse result screen and its menu nor the Extract results panel has a question or chat box, and Chat cannot start until an index exists. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Filter records across documents | No published result | ||
Review and Correction3 fail
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Correct a field value | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedLlamaParse offered no way to correct a returned field. No edit control was found in the Results view, JSON view, History menu or Details drawer, and the saved receipt number stayed RHS-70234. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Corrected data goes downstream | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedLlamaParse did not carry the corrected receipt_number RHS-99999 downstream on the Extract screen. On the Free plan, the agentic four-field setup stayed read-only, and the downloaded JSON and opened export still showed receipt_number RHS-70234 with subtotal, tax, and total_due. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Repair a record | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedLlamaParse did not let the user add or remove a member of an extracted set. The saved result stayed at three line items: Site fencing remained, and Concrete mixer was absent after the edit attempts. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Source Grounding2 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The field is absent | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse passed this absent-field case by leaving the missing loyalty field empty and ungrounded. The receipt has no loyalty wording, the extractor returned an empty value for that field, and the job record gave that field no citation metadata while the present receipt number had a citation. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Trace a value to its location | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedLlamaParse traced the value to the right place in the document. The value appears once in the two-page invoice; the citation names page 2 and the viewer box lands on the value. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Training1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| A field the tool can only get right from examples | No published result | ||
Reading these Results
Published evidence and test coverage answer different questions.
Which scenarios have a Result?
A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.
What does each Result cover?
Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.
Out of 19 scenarios
The summary counts each of the 19 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether LlamaParse was tested on it.