Scenario in benchmark · Converting a complex PDF into clean Markdown with a hosted API · Version 1

A document containing mathematical equations

A document containing mathematical equations is handled as mathematics, not flattened into ordinary prose.

What we check

  1. Whether the equations remain represented as math rather than being flattened into ordinary prose.
  2. Whether the mathematical structure is preserved well enough to stay readable and recoverable.
  3. Whether any limitation is stated honestly through a clear fallback instead of silent corruption.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 4
Extend AIFailed1 of 1 test case failedFailed to keep equations as math markup or an honest fallback in Markdown.View Result →

The returned Markdown turned equations (1) to (4) into garbled prose and never showed math markup or a stated fallback, even though the source equations were available in the PDF.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

PDF VectorFailed1 of 1 test case failedDid not keep the equations of a three-page technical note as math.View Result →

At each of the six equations only its number remained, the bodies reappeared at the end of each page as broken fragments, and the returned Markdown holds no math markup. Sentences that used the symbols lost them too.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ReductoFailed1 of 1 test case failedFailed a document with mathematical equations: the returned Markdown contained no math markup and no fallback, and the equations were flattened into prose with missing symbols and substituted characters.View Result →

This is the case: the source page shows set mathematics, while the output preserves only garbled lines instead of LaTeX, MathML, or an explicit fallback.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Upstage AIFailed1 of 1 test case failedSilently changed a mathematical symbol when converting the PDF to Markdown.View Result →

Five of the six equations retained faithful LaTeX, while equation (5) turned the source’s plain b into a negated b.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 6
Adobe PDF Extract APIPassed1 of 1 test case passedPreserved the equations in the three-page PDF as linked figure images in the generated Markdown.View Result →

The page shows each equation in place with its number beneath it, and the figure images keep fractions, superscripts, subscripts, and absolute-value bars intact; there are no LaTeX, MathML, or $ delimiters.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FirecrawlPassed1 of 1 test case passedPreserved the equations as math in this document.View Result →

It returned all six displayed equations as correct LaTeX.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

GPT-5.6 terraPassed1 of 1 test case passedKept the document's equations as math markup rather than flattening them into prose.View Result →

The source note had six numbered equations plus one unnumbered relation, and the returned Markdown carried all seven as LaTeX with numbering preserved; the title, byline, sections, body prose, and references also carried over.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIPassed1 of 1 test case passedConverted all six display equations into correct LaTeX rather than garbled prose.View Result →

Equation (1) appeared three times, compared with once in the source.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LlamaParsePassed1 of 1 test case passedReturned all six numbered equations of a three-page technical note as LaTeX math, each matching the printed page symbol by symbol, and none came out as garbled prose.View Result →

Inline math stayed inline and Greek letters came back as LaTeX commands.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Mistral AIPassed1 of 1 test case passedKept the mathematical equations as LaTeX math in the Markdown for this PDF.View Result →

The returned Markdown preserved all six display equations as math blocks with tags, kept inline symbols inside math delimiters, and carried through the page-break text and running heads. The page-by-page comparison sheets matched the source pages shown there.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (3): Nutrient.io, Tensorlake, Unstructured.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
1 public · 0 withheld
Capabilities exercised here
Equations & Mathematical Notation

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A document containing mathematical equations in Converting a complex PDF into clean Markdown with a hosted API | AI Demos