Benchmark · Version 1

Structured Document Extraction

This benchmark evaluates document extraction platforms that take a user-defined schema and return structured records.

Benchmark overview

7Capabilities
19Scenarios
10Participating tools
What is included and excluded

Evaluates whether a document extraction platform can turn a user-defined schema into correct structured records across layouts, scans and repeated sets, while staying honest about what it could not find.

A buyer learns how far the platform can go in turning documents into usable structured output, and whether it clearly separates extracted facts from missing ones or values found elsewhere in the document.

In scope

  • User-defined schema to structured records from many documents
  • Scalar fields and repeated or nested records
  • Digital, scanned and mixed documents
  • Training from labelled examples
  • Correcting returned data
  • Working with extracted records: filtering, totalling and asking questions

Out of scope

  • Document classification
  • Packet splitting
  • Routing document types to different schemas
  • Transcription quality
  • Non-English documents

Participating tools

How each tool did on the 19 scenarios in this benchmark. Each cell is one scenario, in the same order for every tool and grouped by capability; Pass and Fail come from published Results.

PassedFailedPublished, different test setNo published Result yet
ToolScenario results · 19 scenarios in 7 capabilitiesPassedFailedExplore
FutureSmart Document Intelligence
20 of 20 scenarios published · 15 passed · 4 failed · 1 different test set
15of 19 graded4View tool →
LlamaParse
18 of 20 scenarios published · 13 passed · 5 failed
13of 18 graded5View tool →
Extend AI
17 of 20 scenarios published · 14 passed · 3 failed
14of 17 graded3View tool →
Landing AI
17 of 20 scenarios published · 12 passed · 5 failed
12of 17 graded5View tool →
Nanonets
17 of 20 scenarios published · 16 passed · 1 failed
16of 17 graded1View tool →
Docsumo
16 of 20 scenarios published · 15 passed · 1 failed
15of 16 graded1View tool →
Reducto
11 of 20 scenarios published · 8 passed · 3 failed
8of 11 graded3View tool →
Datalab
6 of 20 scenarios published · 6 passed · 0 failed
6of 6 graded0View tool →
2 more toolsin the rosterNo published Results yet for Retab · Unstract.Each tool gets its own row here once its first Result is published.

Capabilities & scenarios

19 scenarios grouped by 7 capabilities. Open a group to explore its scenarios in this benchmark.

Field Extraction5 scenarios

Returns the scalar fields the schema asks for — present, interpreted, derived, absent or differently laid out — correctly typed

Capability in this benchmark → · Global definition →

  1. The value is directly available6 tools with published Results
  2. The value needs a supplied definition7 tools with published Results
  3. The value must be derived8 tools with published Results
  4. The same field across layouts6 tools with published Results
  5. The field is absent7 tools with published Results
Nested & Repeated Fields4 scenarios

Returns repeated sets of records completely, with each value under the right record: none dropped, none invented, none cross-bound

Capability in this benchmark → · Global definition →

  1. A repeated set of records8 tools with published Results
  2. The set continues across a page break6 tools with published Results
  3. The set contains a non-record6 tools with published Results
  4. The set is legitimately empty6 tools with published Results
Training1 scenario

Learns from labeled examples — documents paired with their expected output — and applies what it learned to a document it hasn't seen

Capability in this benchmark → · Global definition →

  1. A field the tool can only get right from examples1 tool with a published Result
OCR2 scenarios

The same schema returns the same values when the document is an image, and a mixed document is handled page by page

Capability in this benchmark → · Global definition →

  1. A cleanly scanned document6 tools with published Results
  2. A document mixing digital and scanned pages8 tools with published Results
Source Grounding2 scenarios

Points to where a returned value came from — and attaches no pointer to a value that wasn't there

Capability in this benchmark → · Global definition →

  1. Trace a value to its location6 tools with published Results
  2. The field is absent7 tools with published Results
Review and Correction3 scenarios

A human fix reaches the stored record and the systems downstream of it

Capability in this benchmark → · Global definition →

  1. Correct a field value4 tools with published Results
  2. Repair a record7 tools with published Results
  3. Corrected data goes downstream7 tools with published Results
Querying3 scenarios

Operates over the extracted records — filters, totals, answers questions — and is clear when an answer didn't come from them

Capability in this benchmark → · Global definition →

  1. Filter records across documents4 tools with published Results
  2. Aggregate across documents6 tools with published Results
  3. The answer was never in the schema6 tools with published Results

Results overview

Current published evidence in this benchmark.

115Current published Results
19Scenarios with published Results
8Tools with published Results

Publication availability is separate from test coverage and unpublished research progress.

Explore scenarios →

How the benchmark works

A public summary of the evaluation method. The same defined scope and evidence standard apply to every tool assessed under this version.

Scoring

Unsupported capabilities

A capability a tool doesn't support scores 0, and the 0 stays in the denominator. No tool ranks higher for having less product.

Full benchmark methodology →

Resources and fixtures

The registered material and systems that create a consistent test environment for this benchmark.

Structured Document Extraction — Benchmark definition | AI Demos