Tool in benchmark · Version 1

LlamaParse in Converting a complex PDF into clean Markdown with a hosted API

Scenario-level performance from current published Results.

8Passed
2Failed
2No published Result yet

10 of 12 scenarios published

CapabilityScenariosResult
Code Extraction1 pass · 0 fail · 1 of 1 published
Equations & Mathematical Notation1 pass · 0 fail · 1 of 1 published
Figures & Charts0 pass · 1 fail · 1 of 2 published
Heading & Section Structure1 pass · 1 fail · 2 of 2 published
Reading Order & Layout1 pass · 0 fail · 1 of 1 published
Scanned Document OCR2 pass · 0 fail · 2 of 2 published
Table Extraction1 pass · 0 fail · 1 of 2 published
Text Fidelity1 pass · 0 fail · 1 of 1 published
PassedFailedPublished, different test setNo published Result yet

Where LlamaParse failed

2 of 10 graded scenarios. Open a Result to see the conversation and the evidence.

Figures & Charts

A document with figures and captions

LlamaParse replaced the figure with prose rather than preserving it in the default Markdown output. Its caption remained in place, but the image was neither embedded nor linked.

View Result →
Heading & Section Structure

A document with styled headings and subheadings

LlamaParse turned every heading into a Markdown heading but kept only two of the source's three heading levels. On page 1, the subsections Chain of custody and Condition on arrival came out at the same level as their parent, Reception, and the third-tier headings shared a level with the second. A separate run with a custom prompt asking for three heading levels did not preserve the hierarchy either.

View Result →

How LlamaParse performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Code Extraction1 pass
ScenarioPublished outcomesTest coverageResult
A document containing a code block
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse returned both code listings in the document as fenced blocks tagged as Python and shell, and kept the Python body at its three indent levels of 4, 8 and 12 spaces. Two small whitespace differences remain inside the fences: one blank line was dropped, and the shell continuation line sits at four spaces rather than six.

1/1 assessed1/1 gradablePublished test setView Result →
Equations & Mathematical Notation1 pass
ScenarioPublished outcomesTest coverageResult
A document containing mathematical equations
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse returned all six numbered equations of a three-page technical note as LaTeX math, each matching the printed page symbol by symbol, and none came out as garbled prose. Inline math stayed inline and Greek letters came back as LaTeX commands.

1/1 assessed1/1 gradablePublished test setView Result →
Figures & Charts1 fail1 no Result yet
ScenarioPublished outcomesTest coverageResult
A document with figures and captions
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

LlamaParse replaced the figure with prose rather than preserving it in the default Markdown output. Its caption remained in place, but the image was neither embedded nor linked.

1/1 assessed1/1 gradablePublished test setView Result →
A document with a data chartNo published result
Heading & Section Structure1 pass1 fail
ScenarioPublished outcomesTest coverageResult
A document with styled headings and subheadings
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

LlamaParse turned every heading into a Markdown heading but kept only two of the source's three heading levels. On page 1, the subsections Chain of custody and Condition on arrival came out at the same level as their parent, Reception, and the third-tier headings shared a level with the second. A separate run with a custom prompt asking for three heading levels did not preserve the hierarchy either.

1/1 assessed1/1 gradablePublished test setView Result →
Footnotes at the bottom of the page
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse kept all 13 footnotes out of the body text and tied each one to its reference. Each page's notes come after that page's body, every note on its own line, and every reference mark in the body is a superscript that matches a note, with every note matching a mark.

1/1 assessed1/1 gradablePublished test setView Result →
Reading Order & Layout1 pass
ScenarioPublished outcomesTest coverageResult
A page laid out in multiple columns
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse read the two-column bulletin in natural order on all three pages. It finished each column before continuing to the next, without interleaving the text.

1/1 assessed1/1 gradablePublished test setView Result →
Scanned Document OCR2 pass
ScenarioPublished outcomesTest coverageResult
A cleanly scanned document
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse recovered every printed word of the image-only certificate correctly, including the station code VQ1, all five mg/l readings, the four method codes and the dates. The six-row results block came back as a table, and the only addition was a line standing in for the handwritten signature.

1/1 assessed1/1 gradablePublished test setView Result →
A document mixing digital and scanned pages
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse processed the scanned page in a PDF that also contained digital pages. Its Markdown included the scanned worksheet’s text, readings, identifiers and times.

1/1 assessed1/1 gradablePublished test setView Result →
Table Extraction1 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
A simple, clearly formatted table
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse returned the page-1 charges table intact: the four headers in order, all eight rows and every value under the right column. The cells most likely to drift held: 'Metals suite, eleven elements' stayed one cell, and '10 days' stayed out of the charge column. The table came back as an HTML table inside the Markdown.

1/1 assessed1/1 gradablePublished test setView Result →
A table that continues across a page breakNo published result
Text Fidelity1 pass
ScenarioPublished outcomesTest coverageResult
An ordinary digital text document
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

LlamaParse preserved the briefing note’s body text word for word: 3,122 words in the source and 3,122 in Markdown. All eleven money and measurement figures were present.

1/1 assessed1/1 gradablePublished test setView Result →

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 12 scenarios

The summary counts each of the 12 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether LlamaParse was tested on it.

LlamaParse in Converting a complex PDF into clean Markdown with a hosted API | AI Demos