bench 1.0.0 · generated 2026-11-05

Scored against Word, failures included.

The oracle is Word’s own PDF export of each document (SHA-pinned). A converter’s PDF is scored 0–100 by pixel similarity; redlines are additionally opened in Word and compared against Word’s own compare. Medians and means count a failed document as 0, so no tool gains by skipping a hard one. Every table is a ranking, and the tool that leads it is named above the table.

Accepted and randomized redline DOCX to PDF vs Word export

Document set `c28442e9bb5a5deb` (400 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.8`.

Each converter turns accepted and randomized-redline DOCX documents into PDF; its PDF is scored pixel-wise against the SHA-pinned PDF Word itself exported from the same file. A failed conversion scores 0 for that document (intent-to-treat).

Leader: jubarte 0.12.0 — median 82.45

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte 0.12.0
0.12.0@3c81203f1f66
[81.13, 83.23]
82.45 79.93 400 0
1=
jubarte 0.11.0
0.11.0@838d0e9603c8
[77.54, 81.89]
79.75 76.24 400 0
3
Wordcraft CLI 0.7.0
wordcraft-cli 0.7.0@45fa52a260fb400857c6033c3cbfa90f8e9c67d05aeb9417cccbb3d6c51c0960
[52.41, 55.60]
53.82 56.59 400 0
3=
Wordcraft CLI
0.6.1@b71aef4ee85a
[51.41, 54.53]
52.82 55.46 400 0
3=
docxide-pdf 0.18.4
docxide-pdf 0.18.4@503ea7e085ae9acd16a4f263c7802325cad254eedeff4cb2509e88714019e00b
[50.63, 54.45]
52.37 55.90 400 0
3=
libreoffice_convert_rust 0.1.0
libreoffice_convert_rust 0.1.0@99552eb9ead6dc5dd2bbb3c69cbed958888dbb037553d586f08b5aa80794e172
[47.43, 48.97]
47.87 54.82 398 2
3=
soffice
LibreOffice 26.8.1.1 bf6ef2662401c7d6dce671ab4deb26ee0af3a7c1
[47.43, 48.97]
47.87 54.82 398 2
8
rdocx 0.16.0
rdocx-cli 0.16.0@8c4d40caffb12177ac8ab21d568a5dd6c389f9cb4bd742653c2b180102d0abee
[45.14, 47.11]
45.90 50.69 397 3
9
office2pdf 0.8.1
office2pdf-cli 0.8.1@ec266239b0bdbf71275e79fde2ffaaa1155321050d716eab45ef43f60d6ff9f9
[42.81, 45.81]
44.45 48.66 398 2
10
pdfitdown 4.2.0
pdfitdown 4.2.0@6a8275fe29d0c7fd61465f56ca55c21fc22c84fbd228ceef4288bfb28bceac36
[40.89, 43.34]
41.95 43.48 397 3

Source DOCX → PDF, no redlines

Document set `c947065c59b2` (398 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.6`.

Each converter turns the clean source DOCX documents, with no tracked changes in them, into PDF; its PDF is scored pixel-wise against the SHA-pinned PDF Word itself exported from the same source. A failed conversion scores 0 (intent-to-treat).

Leader: office2pdf — median 84.34

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
office2pdf
office2pdf 0.6.7
[79.83, 86.03]
84.34 73.82 390 8
1=
pdfitdown
pdfitdown 4.0.0
[79.83, 86.03]
84.34 73.82 390 8
3
libreoffice_convert_rust
LibreOffice Convert Rust v0.1.0
[72.14, 79.47]
75.26 73.98 398 0
4
jubarte
jubarte 0.7.0
[66.94, 68.10]
67.39 66.25 398 0
4=
docxide-pdf
docxide-pdf v0.17.0
[63.27, 64.73]
63.97 65.65 398 0
6
rdocx
rdocx 0.7.0
[53.38, 55.31]
54.49 59.11 398 0
7
dxpdf
dxpdf 0.5.1
[52.15, 52.94]
52.52 56.73 381 17

Every corpus/word DOCX → PDF vs Word's own PDF

Document set `a89ffc217bba` (6427 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `1.0`.

Word's PDF export of each DOCX in all four corpus/word states (clean; tracking without comments; comments with tracking; comments clean) is the oracle; the candidate is each converter's PDF of the same file, scored pixel-wise. A failed conversion scores 0 (intent-to-treat).

Leader: jubarte — median 82.36

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte
jubarte 0.17.0+c607894
[82.19, 82.54]
82.36 80.12 6,427 0

The same documents under docxide-pdf's own metrics

Document set `a89ffc217bba` (6427 documents), renderer `oracle:microsoft_word`, scorer `docxide-150dpi`, bench `1.0`.

The same documents as the corpus-wide table, scored instead with docxide-pdf's own ink Jaccard (the ranked number) and text-boundary metrics at 150 DPI against the same Word PDF exports, so no converter is measured only by this bench's pixel scorer.

Leader: jubarte — median 70.68

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte
jubarte 0.17.0+c607894
[70.20, 71.23] · text 100.00
70.68 66.84 6,397 30

Below 3 pages: every converter on the same short documents

Document set `27122f4d945e` (2626 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.7`.

The corpus-wide conversion benchmark restricted to the documents Word lays out in 1 to 3 pages, so a converter that renders only the first 3 pages (unlicensed PyMuPDF Pro) is compared with every other converter on the same documents. The oracle stays Word's own PDF export of each document, scored pixel-wise; a failed conversion scores 0 (intent-to-treat).

Leader: jubarte — median 83.15

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte
jubarte 0.9.3
[83.01, 83.28]
83.15 81.35 2,626 0
2
soffice
LibreOffice 26.8.0.3 bce0998afefdbc355585ca324285661a2170ba77
[67.84, 68.83]
68.40 66.26 2,611 15
3
PyMuPDF Pro
pymupdf-pro 1.28.2 (unlicensed, first 3 pages)
[59.09, 60.77]
60.06 60.02 2,601 25
4
docxide-pdf
docxide-pdf v0.17.1
[53.81, 54.89]
54.24 56.00 2,626 0

Below 3 pages: docxide-pdf's metrics on the same short documents

Document set `27122f4d945e` (2626 documents), renderer `oracle:microsoft_word`, scorer `docxide-150dpi`, bench `0.7`.

The docxide-pdf metrics on the same 1-to-3-page documents: each converter's PDF scored with ink Jaccard (the ranked number) and text-boundary metrics at 150 DPI against Word's own PDF exports of the same files.

Leader: jubarte — median 74.34

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte
jubarte 0.9.3
[73.30, 75.08] · text 100.00
74.34 68.08 2,626 0
2
soffice
LibreOffice 26.8.0.3 bce0998afefdbc355585ca324285661a2170ba77
[23.36, 24.50] · text 99.58
24.03 28.66 2,611 15
3
PyMuPDF Pro
pymupdf-pro 1.28.2 (unlicensed, first 3 pages)
[20.96, 22.71] · text 36.36
21.86 26.77 2,595 31
4
docxide-pdf
docxide-pdf v0.17.1
[11.17, 11.96] · text 24.32
11.42 19.93 2,619 7

Redlines vs Word's compare

Each tool redlines a document pair; Word opens that redline and exports it to PDF, and the PDF is scored with the pixel scorer against Word's own compare of the same pair. A pair with no scored PDF — no redline, or one Word could not open — counts 0 in the ITT columns; the = 100, ≥ 90 and < 50 counts and the docxide-pdf Ink Jaccard and Text boundary means are over the scored PDFs only.

Leader: jubarte-0.17.0 — median 84.56

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte-0.17.0
jubarte 0.17.0 (release)
= 100: 342 · ≥ 90: 729 · < 50: 500 · ink 0.75 · text 0.82
84.56 75.81 3,454 48
2
jubarte-0.14.0
jubarte 0.14.0 (release)
= 100: 344 · ≥ 90: 704 · < 50: 546 · ink 0.74 · text 0.81
84.46 74.33 3,414 88
3
jubarte-0.10.1
jubarte 0.10.1 (release)
= 100: 241 · ≥ 90: 615 · < 50: 619 · ink 0.73 · text 0.80
84.15 74.77 3,496 6
4
jubarte-rust
jubarte 0.10.0 (86b6b5d3)
= 100: 310 · ≥ 90: 658 · < 50: 793 · ink 0.67 · text 0.77
81.73 72.23 3,499 3
5
jubarte-093
jubarte 0.9.3 (release)
= 100: 309 · ≥ 90: 634 · < 50: 780 · ink 0.67 · text 0.76
79.07 68.62 3,338 164
6
docxodus-15.0.0
docxodus 15.0.0 (npm/WASM, set redline_600)
= 100: 46 · ≥ 90: 98 · < 50: 135 · ink 0.65 · text 0.71
75.70 64.86 543 57
7
docxodus
Docxodus 12.6.5 (C#)
= 100: 204 · ≥ 90: 482 · < 50: 614 · ink 0.65 · text 0.73
62.87 53.94 2,612 890
8
apryse
apryse-sdk 12.2.1 (PDFNet 12.2.0-079a1c0268, demo key: output watermarked)
= 100: 0 · ≥ 90: 39 · < 50: 1285 · ink 0.18 · text 0.16
43.17 37.50 2,524 978
9
apryse-12.2.1
apryse-sdk 12.2.1 (PDFNet 12.2.0-079a1c0268, demo key: output watermarked; set redline_600)
= 100: 0 · ≥ 90: 4 · < 50: 190 · ink 0.18 · text 0.14
37.87 28.42 336 264
10
superdoc-2.25.0
superdoc-sdk 2.25.0 (set redline_600)
= 100: 1 · ≥ 90: 1 · < 50: 3 · ink 0.21 · text 0.20
0.00 6.92 67 533
11
superdoc
superdoc-sdk 2.16.0
= 100: 4 · ≥ 90: 7 · < 50: 20 · ink 0.21 · text 0.29
0.00 6.66 380 3,122

Redlines accepted or rejected by Word

Each tool redlines a pair; Word accepts — or rejects — every change in that redline and exports the result to PDF, scored with the pixel scorer against Word's own compare of the pair resolved the same way. A pair with no scored PDF counts 0 (intent-to-treat); the calibration row is Word's own compare against itself.

Leader: word-identity — median 100.00

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
word-identity
Word's own compare (calibration) · accept all
= 100: 400 · ≥ 90: 400 · < 50: 0 · ink 1.00 · text 1.00
100.00 100.00 400 0
2
docxodus
docxodus 12.6.4 (npm) · accept all
= 100: 166 · ≥ 90: 299 · < 50: 16 · ink 0.85 · text 0.96
99.87 90.07 394 6
3
jubarte-rust
jubarte 0.9.3 (673aff74) · accept all
= 100: 181 · ≥ 90: 261 · < 50: 33 · ink 0.80 · text 0.94
99.70 87.56 400 0
4
superdoc
superdoc-sdk 2.15.0 · accept all
= 100: 4 · ≥ 90: 52 · < 50: 1 · ink 0.95 · text 0.98
0.00 13.95 58 342
1
word-identity
Word's own compare (calibration) · reject all
= 100: 400 · ≥ 90: 400 · < 50: 0 · ink 1.00 · text 1.00
100.00 100.00 400 0
2
jubarte-rust
jubarte 0.9.3 (673aff74) · reject all
= 100: 186 · ≥ 90: 260 · < 50: 35 · ink 0.77 · text 0.91
99.85 86.22 400 0
3
docxodus
docxodus 12.6.4 (npm) · reject all
= 100: 155 · ≥ 90: 275 · < 50: 26 · ink 0.80 · text 0.94
99.80 85.38 385 15
4
superdoc
superdoc-sdk 2.15.0 · reject all
= 100: 1 · ≥ 90: 36 · < 50: 1 · ink 0.83 · text 0.95
0.00 11.07 49 351

600-document samples

the release samples, scored on the same rules

DOCX → PDF vs Word's own PDF — the 860-document set

860 docs · 380/375/22/83 by state · set 2026-11-03.1 · scorer numpy · 2026-11-05

The fixed 860-document conversion set (every path and sha256 in release_info/sample_conversion_0.17.0_11-05-26_03-23.csv): each converter turns the same DOCX into PDF, scored pixel-wise against Word's own export of the same file. jubarte converts and is scored by this run, intent-to-treat — a document it fails scores 0; the comparators are the vendor refresh's runs on the same set, the newest version per tool, and a document a converter did not run is not run, never 0. Sorted by median; 95% CI is a percentile bootstrap of the median (2,000 resamples, seed 42).

Leader: jubarte — median 81.88

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte
jubarte 0.17.0
[81.2, 82.44]
81.88 78.25 860 0
2
apryse
apryse-sdk 12.2.1 (PDFNet 12.2.0-079a1c0268, demo key: output watermarked)
[74.79, 76.42]
75.55 69.26 860 0
3
docxide
docxide-pdf v0.18.4
[56.51, 61.21]
58.57 61.01 860 0
4
soffice
LibreOffice 26.8.2.2 9dea406ab7695df6325b11b0e19fdca8db4b7276
[53.3, 57.76]
55.39 58.07 860 0
5
docxodus
@docxodus/export 15.0.0 (docxodus 15.0.0)
[42.19, 43.91]
43.04 46.12 860 38

Redlines vs Word's compare — the 600-pair set

600 pairs · one Word compare each · pixel scorer · opened in Word · set 2026-11-03.1 · 2026-11-05

The fixed 600-pair redline set (every path and sha256 in release_info/sample_redline_0.17.0_11-05-26_03-23.csv): every jubarte.pro case pair plus pairs drawn from the release sample, state-balanced, one Word compare each. Each tool redlines the pair; Word opens that redline and exports it to PDF, scored against Word's own compare. jubarte is scored by this run, intent-to-treat — a pair without a scored PDF counts 0; the comparators are the vendor refresh's runs on the same set, the newest version per tool, and a pair a comparator did not run is not run, never 0.

Leader: jubarte-0.17.0 — median 82.99

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1=
jubarte-0.17.0
jubarte 0.17.0
= 100: 41 · ≥ 90: 96
82.99 74.06 600 3
1=
docxodus
docxodus 15.0.0
= 100: 46 · ≥ 90: 98 · not run: 57
80.37 71.67 543 0
3
superdoc
superdoc 2.25.0
= 100: 1 · ≥ 90: 1 · not run: 533
61.89 61.98 67 0
4
apryse
apryse 12.2.1
= 100: 0 · ≥ 90: 4 · not run: 264
47.50 50.76 336 0

By corpus state

median · documents per state in brackets

clean (380)

jubarte87.30
soffice62.26

tracking, no comments (375)

jubarte81.11
soffice57.85

comments, clean (22)

jubarte80.01
soffice40.79

comments + tracking (83)

jubarte79.74
soffice43.21

clean (100)

jubarte 0.12.090.78
jubarte 0.11.090.72
Wordcraft CLI 0.7.068.29
Wordcraft CLI63.87
docxide-pdf 0.18.466.93
libreoffice_convert_rust 0.1.069.33
soffice69.33
rdocx 0.16.062.29
office2pdf 0.8.154.42
pdfitdown 4.2.044.02

tracking, no comments (100)

jubarte 0.12.082.16
jubarte 0.11.081.85
Wordcraft CLI 0.7.059.18
Wordcraft CLI55.56
docxide-pdf 0.18.450.25
libreoffice_convert_rust 0.1.065.47
soffice65.47
rdocx 0.16.049.35
office2pdf 0.8.145.48
pdfitdown 4.2.041.13

comments + tracking (100)

jubarte 0.12.080.11
jubarte 0.11.071.20
Wordcraft CLI 0.7.050.58
Wordcraft CLI48.46
docxide-pdf 0.18.446.71
libreoffice_convert_rust 0.1.042.12
soffice42.12
rdocx 0.16.041.75
office2pdf 0.8.140.73
pdfitdown 4.2.040.84

comments, clean (100)

jubarte 0.12.080.63
jubarte 0.11.068.97
Wordcraft CLI 0.7.051.94
Wordcraft CLI51.82
docxide-pdf 0.18.456.18
libreoffice_convert_rust 0.1.045.33
soffice45.33
rdocx 0.16.045.21
office2pdf 0.8.144.45
pdfitdown 4.2.044.23

clean (2,480)

jubarte89.84

tracking, no comments (3,410)

jubarte81.11

comments + tracking (420)

jubarte78.19

comments, clean (117)

jubarte81.22

clean (2,480)

jubarte81.13

tracking, no comments (3,410)

jubarte66.44

comments + tracking (420)

jubarte60.24

comments, clean (117)

jubarte68.56

clean (1,188)

jubarte88.11
soffice69.81
PyMuPDF Pro67.45
docxide-pdf57.57

tracking, no comments (1,366)

jubarte82.44
soffice68.39
PyMuPDF Pro57.81
docxide-pdf53.88

comments + tracking (69)

jubarte72.84
soffice46.56
PyMuPDF Pro46.51
docxide-pdf47.24

comments, clean (3)

jubarte69.88
soffice40.69
PyMuPDF Pro41.46
docxide-pdf40.71

clean (1,188)

jubarte79.24
soffice26.33
PyMuPDF Pro31.69
docxide-pdf15.60

tracking, no comments (1,366)

jubarte70.50
soffice22.90
PyMuPDF Pro18.55
docxide-pdf9.95

comments + tracking (69)

jubarte41.71
soffice1.17
PyMuPDF Pro0.26
docxide-pdf0.89

comments, clean (3)

jubarte33.91
soffice4.02
PyMuPDF Pro3.75
docxide-pdf4.51

Oracle: real Word

Every reference PDF is exported by Microsoft Word and SHA-pinned. Redlines are opened in Word too: if Word refuses the file, the tool scores 0 for that pair.

Intent-to-treat

Failures are not dropped. A crash, a timeout or an unopenable file counts as 0 in ITT mean and median, so a tool cannot improve its rank by skipping hard documents.

Paired bootstrap

Ranks tie (n=) when the paired bootstrap interval of the median difference to the row above includes 0. 2,000 resamples, seed 42.

Fixed sets

From 0.17.0 the tables on this page are the bench's fixed evaluation sets (set version 2026-11-03.1), the same documents for every release; every row's inputs are sha256-listed in the engine's release_info/. The comparators are the vendor refresh's runs on the same sets, newest version per tool — a tool the refresh did not run shows as not run, never 0. Full-corpus runs remain in the bench's RESULTS.md.

Provenance

where every number came from