bench 1.0.0 · generated 2026-11-05
Scored against Word, failures included.
The oracle is Word’s own PDF export of each document (SHA-pinned). A converter’s PDF is scored 0–100 by pixel similarity; redlines are additionally opened in Word and compared against Word’s own compare. Medians and means count a failed document as 0, so no tool gains by skipping a hard one. Every table is a ranking, and the tool that leads it is named above the table.
Accepted and randomized redline DOCX to PDF vs Word export
Document set `c28442e9bb5a5deb` (400 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.8`.Each converter turns accepted and randomized-redline DOCX documents into PDF; its PDF is scored pixel-wise against the SHA-pinned PDF Word itself exported from the same file. A failed conversion scores 0 for that document (intent-to-treat).
Leader: jubarte 0.12.0 — median 82.45
Source DOCX → PDF, no redlines
Document set `c947065c59b2` (398 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.6`.Each converter turns the clean source DOCX documents, with no tracked changes in them, into PDF; its PDF is scored pixel-wise against the SHA-pinned PDF Word itself exported from the same source. A failed conversion scores 0 (intent-to-treat).
Leader: office2pdf — median 84.34
Every corpus/word DOCX → PDF vs Word's own PDF
Document set `a89ffc217bba` (6427 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `1.0`.Word's PDF export of each DOCX in all four corpus/word states (clean; tracking without comments; comments with tracking; comments clean) is the oracle; the candidate is each converter's PDF of the same file, scored pixel-wise. A failed conversion scores 0 (intent-to-treat).
Leader: jubarte — median 82.36
The same documents under docxide-pdf's own metrics
Document set `a89ffc217bba` (6427 documents), renderer `oracle:microsoft_word`, scorer `docxide-150dpi`, bench `1.0`.The same documents as the corpus-wide table, scored instead with docxide-pdf's own ink Jaccard (the ranked number) and text-boundary metrics at 150 DPI against the same Word PDF exports, so no converter is measured only by this bench's pixel scorer.
Leader: jubarte — median 70.68
Below 3 pages: every converter on the same short documents
Document set `27122f4d945e` (2626 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.7`.The corpus-wide conversion benchmark restricted to the documents Word lays out in 1 to 3 pages, so a converter that renders only the first 3 pages (unlicensed PyMuPDF Pro) is compared with every other converter on the same documents. The oracle stays Word's own PDF export of each document, scored pixel-wise; a failed conversion scores 0 (intent-to-treat).
Leader: jubarte — median 83.15
Below 3 pages: docxide-pdf's metrics on the same short documents
Document set `27122f4d945e` (2626 documents), renderer `oracle:microsoft_word`, scorer `docxide-150dpi`, bench `0.7`.The docxide-pdf metrics on the same 1-to-3-page documents: each converter's PDF scored with ink Jaccard (the ranked number) and text-boundary metrics at 150 DPI against Word's own PDF exports of the same files.
Leader: jubarte — median 74.34
Redlines vs Word's compare
Each tool redlines a document pair; Word opens that redline and exports it to PDF, and the PDF is scored with the pixel scorer against Word's own compare of the same pair. A pair with no scored PDF — no redline, or one Word could not open — counts 0 in the ITT columns; the = 100, ≥ 90 and < 50 counts and the docxide-pdf Ink Jaccard and Text boundary means are over the scored PDFs only.
Leader: jubarte-0.17.0 — median 84.56
Redlines accepted or rejected by Word
Each tool redlines a pair; Word accepts — or rejects — every change in that redline and exports the result to PDF, scored with the pixel scorer against Word's own compare of the pair resolved the same way. A pair with no scored PDF counts 0 (intent-to-treat); the calibration row is Word's own compare against itself.
Leader: word-identity — median 100.00
600-document samples
the release samples, scored on the same rulesDOCX → PDF vs Word's own PDF — the 860-document set
860 docs · 380/375/22/83 by state · set 2026-11-03.1 · scorer numpy · 2026-11-05The fixed 860-document conversion set (every path and sha256 in release_info/sample_conversion_0.17.0_11-05-26_03-23.csv): each converter turns the same DOCX into PDF, scored pixel-wise against Word's own export of the same file. jubarte converts and is scored by this run, intent-to-treat — a document it fails scores 0; the comparators are the vendor refresh's runs on the same set, the newest version per tool, and a document a converter did not run is not run, never 0. Sorted by median; 95% CI is a percentile bootstrap of the median (2,000 resamples, seed 42).
Leader: jubarte — median 81.88
Redlines vs Word's compare — the 600-pair set
600 pairs · one Word compare each · pixel scorer · opened in Word · set 2026-11-03.1 · 2026-11-05The fixed 600-pair redline set (every path and sha256 in release_info/sample_redline_0.17.0_11-05-26_03-23.csv): every jubarte.pro case pair plus pairs drawn from the release sample, state-balanced, one Word compare each. Each tool redlines the pair; Word opens that redline and exports it to PDF, scored against Word's own compare. jubarte is scored by this run, intent-to-treat — a pair without a scored PDF counts 0; the comparators are the vendor refresh's runs on the same set, the newest version per tool, and a pair a comparator did not run is not run, never 0.
Leader: jubarte-0.17.0 — median 82.99
By corpus state
median · documents per state in bracketsclean (380)
tracking, no comments (375)
comments, clean (22)
comments + tracking (83)
clean (100)
tracking, no comments (100)
comments + tracking (100)
comments, clean (100)
clean (2,480)
tracking, no comments (3,410)
comments + tracking (420)
comments, clean (117)
clean (2,480)
tracking, no comments (3,410)
comments + tracking (420)
comments, clean (117)
clean (1,188)
tracking, no comments (1,366)
comments + tracking (69)
comments, clean (3)
clean (1,188)
tracking, no comments (1,366)
comments + tracking (69)
comments, clean (3)
Oracle: real Word
Every reference PDF is exported by Microsoft Word and SHA-pinned. Redlines are opened in Word too: if Word refuses the file, the tool scores 0 for that pair.
Intent-to-treat
Failures are not dropped. A crash, a timeout or an unopenable file counts as 0 in ITT mean and median, so a tool cannot improve its rank by skipping hard documents.
Paired bootstrap
Ranks tie (n=) when the paired bootstrap interval of the median difference to the row above includes 0. 2,000 resamples, seed 42.
Fixed sets
From 0.17.0 the tables on this page are the bench's fixed evaluation sets (set version 2026-11-03.1), the same documents for every release; every row's inputs are sha256-listed in the engine's release_info/. The comparators are the vendor refresh's runs on the same sets, newest version per tool — a tool the refresh did not run shows as not run, never 0. Full-corpus runs remain in the bench's RESULTS.md.
Provenance
where every number came from- docsets 27122f4d945e · a89ffc217bba · c28442e9bb5a5deb · c947065c59b2
- renderers oracle:microsoft_word
- scorers docxide-150dpi · v1