HeOCR benchmarks
How HeOCR compares with other open tools, per kind of document. Every figure gives its sample size, a 95% interval, the measure, the reference and the run it comes from. The page is built from one file, results.json, which is updated after each scored run.
Updated 2026-11-19.
Documents by format
Character error by document type
lower is betterOn the held-out set, heocr has the lowest character error on all 9 document types that another tool also reads; its 95% interval is clear of every other reader on 7 of them. DOC is read by heocr alone.
- heocr ours
- Best other open tool the lowest of Docling, MarkItDown and PyMuPDF4LLM
- Stock Tesseract heb, tessdata_best
Run gate-rc1-synth2026-11-19Release gate: 7 of 10 supported
Held-out set: the release gate's fresh generated documents, made after heocr 0.1.0rc1's code was frozen.
One linear axis from 0 to 100%; bars start at zero. Whiskers show the interval (95% percentile bootstrap over documents, 2000 resamples, seed 0). Where an interval is narrower than its marker, read it from the numbers beside it.
Modern printed Hebrew
No data yetHistorical and rabbinic printed Hebrew
Not measured on a held-out set yetHandwriting
No data yetDocuments by format
Scored 2026-11-19Generated documents with exact gold: Word files and PDFs of every kind HeOCR's document reader handles, made by our generators so the correct text is known exactly. Each table is one kind of document. These figures come from the release gate's fresh set: generated with new seeds after heocr 0.1.0rc1's code was frozen, so heocr was never tuned on these documents. They are drawn by the same generators heocr is developed against, so a good figure means good on generated documents of this kind, not more.
| Reader | Character errorlower is better | Word errorlower is better | Numbers foundhigher is better | Text lost without a warninglower is better | |
|---|---|---|---|---|---|
| Word (.docx) · DOCX · 80 documentsSupported | Section numbers correcthigher is bettern = 60 | ||||
| heocrours | 0.30%0.21–0.38% | 0.36%0.26–0.46% | 100.0%100.0–100.0% | 100.0%100.0–100.0% | |
| Docling | 3.3%2.8–3.7% | 8.1%7.3–8.8% | 93.9%91.9–95.6% | 7.9%6.1–10.0% | |
| MarkItDown | 4.0%3.6–4.5% | 7.5%7.0–7.9% | 77.3%73.7–80.7% | 2.9%2.2–3.8% | |
| Legacy Word (.doc) · DOC · 40 documentsSupported | |||||
| heocrours | 0.77%0.60–0.94% | 1.9%1.4–2.5% | 91.4%88.3–94.2% | ||
| PDF with a correct text layer · PDF T1 · 316 documentsSupported | |||||
| heocrours | 0.02%0.02–0.03% | 0.20%0.16–0.25% | 100.0%100.0–100.0% | 0.02%0.02–0.03% | |
| Tesseract (tessdata_best heb)n = 80 | 8.6%7.9–9.3% | 16.9%15.7–18.2% | 47.2%45.5–48.9% | 7.9%7.3–8.5% | |
| Docling | 21.6%19.2–24.2% | 72.5%71.4–73.7% | 96.6%95.6–97.6% | 13.5%10.9–16.4% | |
| MarkItDown | 56.7%53.5–59.8% | 73.6%69.7–77.2% | 99.1%98.7–99.3% | 40.9%38.5–43.2% | |
| PyMuPDF4LLM | 32.1%31.6–32.6% | 45.6%44.7–46.5% | 91.1%90.6–91.6% | 21.8%21.3–22.2% | |
| PDF whose text layer is in visual (reversed) order · PDF T2 · 40 documentsSupported | |||||
| heocrours | 0.02%0.01–0.05% | 0.05%0.01–0.09% | 100.0%100.0–100.0% | 0.02%0.01–0.04% | |
| Docling | 10.6%9.7–11.6% | 19.0%17.9–20.2% | 95.5%94.4–96.5% | 7.0%6.3–7.7% | |
| MarkItDown | 78.3%78.0–78.6% | 98.7%98.5–98.9% | 100.0%100.0–100.0% | 60.9%58.5–63.5% | |
| PyMuPDF4LLM | 71.2%70.9–71.5% | 91.4%91.2–91.6% | 100.0%100.0–100.0% | 51.5%50.6–52.7% | |
| PDF with a broken but recoverable character map · PDF T3 · 146 documentsSupported | |||||
| heocrours | 0.16%0.10–0.24% | 0.63%0.43–0.86% | 99.9%99.6–100.0% | 0.14%0.10–0.19% | |
| Tesseract (tessdata_best heb)n = 66 | 9.2%8.5–10.0% | 19.4%18.2–20.8% | 46.9%45.1–48.7% | 8.5%7.8–9.2% | |
| Docling | 78.7%70.6–87.2% | 276.6%251.7–303.3% | 99.5%99.3–99.7% | 40.6%35.9–45.5% | |
| MarkItDown | 70.7%66.7–74.5% | 88.2%84.2–92.2% | 100.0%100.0–100.0% | 59.2%55.4–62.9% | |
| PyMuPDF4LLM | 65.7%61.6–70.2% | 87.5%81.2–94.8% | 95.0%94.2–95.7% | 53.7%49.8–58.0% | |
| Scan with a wrong OCR text layer · PDF T4 · 40 documentsNot supportedT4 and T5 use the same scanned pages; heocr and Tesseract read both by OCR, so their figures match. T4 adds a wrong text layer, which the other readers use. | |||||
| heocrours | 5.8%2.8–11.3% | 13.7%7.5–25.5% | 78.8%75.9–81.6% | 2.1%1.6–2.8% | |
| Tesseract (tessdata_best heb) | 8.0%7.0–9.0% | 14.5%13.0–16.1% | 47.5%45.2–49.8% | 7.4%6.5–8.3% | |
| Docling | 81.5%81.1–81.8% | 97.4%97.2–97.6% | 69.2%66.7–71.5% | 79.3%78.6–80.0% | |
| MarkItDown | 81.4%80.1–82.5% | 102.1%101.2–103.0% | 16.5%14.6–18.4% | 65.9%63.9–68.0% | |
| PyMuPDF4LLM | 80.8%80.1–81.6% | 99.7%99.4–100.0% | 14.6%12.8–16.5% | 61.9%60.7–63.4% | |
| Scan, image only · PDF T5 · 40 documentsNot supported | |||||
| heocrours | 5.5%2.7–10.9% | 13.3%7.3–25.1% | 79.1%76.2–81.8% | 1.8%1.5–2.1% | |
| Tesseract (tessdata_best heb) | 8.0%7.0–9.0% | 14.5%13.0–16.1% | 47.5%45.2–49.8% | 7.4%6.5–8.3% | |
| Docling | 84.1%83.5–84.7% | 97.1%96.9–97.4% | 68.0%65.3–70.5% | 82.3%81.5–83.0% | |
| MarkItDown | 100.0%100.0–100.0% | 100.0%100.0–100.0% | 0.00%0.00–0.00% | 100.0%100.0–100.0% | |
| PyMuPDF4LLM | 82.9%82.8–83.1% | 99.0%98.9–99.2% | 73.4%69.9–76.7% | 78.6%78.0–79.2% | |
| Text layer plus text drawn as images · PDF T6 · 40 documentsNot supported | Words in image regions foundhigher is better | ||||
| heocrours | 0.23%0.08–0.44% | 0.34%0.19–0.55% | 99.7%99.4–99.9% | 0.14%0.01–0.34% | 95.8%92.5–98.1% |
| Tesseract (tessdata_best heb) | – | – | – | – | 61.9%58.8–66.1% |
| Docling | 13.4%12.6–14.2% | 68.2%66.1–70.3% | 99.9%99.6–100.0% | 4.6%3.9–5.2% | 67.5%61.2–73.8% |
| MarkItDown | 44.3%35.7–52.8% | 58.0%47.4–68.4% | 89.2%87.6–90.7% | 32.2%26.1–38.4% | 15.5%9.8–21.7% |
| PyMuPDF4LLM | 33.2%31.8–34.5% | 44.5%42.8–46.3% | 80.3%78.6–82.1% | 23.9%22.4–25.5% | 25.4%19.5–31.6% |
| Two pages on one sheet · PDF spread · 40 documentsSupported | |||||
| heocrours | 0.02%0.01–0.05% | 0.05%0.01–0.09% | 100.0%100.0–100.0% | 0.02%0.00–0.04% | |
| Docling | 71.8%66.3–76.0% | 121.4%115.8–125.6% | 99.8%99.5–100.0% | 4.3%3.6–5.0% | |
| MarkItDown | 74.7%70.7–77.9% | 91.8%86.4–96.0% | 100.0%100.0–100.0% | 25.4%19.5–31.5% | |
| PyMuPDF4LLM | 74.6%70.9–77.3% | 91.5%86.8–94.9% | 91.0%89.6–92.4% | 20.4%19.2–21.4% | |
| Reader | Table rows readhigher is better | Rows lost without a warninglower is better | Character errorlower is better | Numbers foundhigher is better |
|---|---|---|---|---|
| Scanned pages with a ruled table · Scanned tables · 120 documentsSupported | ||||
| heocrours | 95.9%94.6–97.0% | 0.06%0.00–0.19% | 1.4%1.2–1.6% | 97.6%97.1–98.2% |
| Tesseract (tessdata_best heb) | 2.3%1.4–3.1% | 97.7%96.9–98.6% | 59.5%57.0–61.9% | 13.3%12.4–14.1% |
| Docling | 2.3%1.7–2.9% | 97.7%97.0–98.3% | 69.7%69.2–70.1% | 96.1%95.4–96.7% |
| MarkItDown | 0.00%0.00–0.00% | 100.0%100.0–100.0% | 100.0%100.0–100.0% | 0.00%0.00–0.00% |
| PyMuPDF4LLM | 0.00%0.00–0.00% | 100.0%100.0–100.0% | 90.9%90.6–91.2% | 3.2%2.6–3.9% |
A dash means the reader has no score on that measure. An empty cell means the measure does not apply to that document type. The last column holds a measure that exists for one type only; its name is in that type's heading row.
Each table lists the readers that handle that format. Tesseract reads images only, so it appears on scans; on PDF T6 it is scored on the image regions only.
Word error counts inserted words as well as wrong and missing ones, so it can exceed 100%.
Method, reference and run
- Measure:
line_fold_v1(text normalised the same way for every reader; ratios of sums over documents) - Reference: the release gate's fresh generated set (the cases of D-bench sets A, B and C, new seeds after the code freeze): generated documents with exact gold; text from the public-domain Israeli official gazette (Reshumot)
- Licence: PD-IL (text)
- Interval: 95% percentile bootstrap over documents, 2000 resamples, seed 0
- Run: gate-rc1-synth; heocr at 57e58de
- Reader versions: Tesseract (tessdata_best heb) tesseract 5.5, heb tessdata_best e12c65a (sha256 dbaa827a), psm 3, pages at 300 dpi gray (pdftoppm); Docling 2.150.1; MarkItDown 0.1.9 [pdf,docx]; PyMuPDF4LLM 2.1 (AGPL; benchmark baseline only, decision 0020 item 2)
- Left out: two office-scan cases rendered with a proprietary font (their scores cannot be published), and the provisional section-number case from the label measure
- Scoring code: HeOCR's evaluation repository (not yet public; it is published with the release gate)
Commercial OCR vendors
PendingRelease gate
The release gate scores a fresh set, made after the code is frozen. Each document type then gets one verdict: supported, or not supported. Run gate-rc1: heocr 0.1.0rc1 (commit 57e58de), scored 2026-11-19. The verdict is set by the one-sided 97.5% upper bound of heocr's difference from each other reader, against a fixed margin.
7Supported·3Not supported
- DOCX Word (.docx)Supported
- DOC legacy Word (.doc)Supported
- PDF T1 correct text layerSupported
- PDF T2 visual-order text layerSupported
- PDF T3 broken character mapSupported
- PDF T4 scan, wrong OCR layervs Tesseract: Character error, difference −2.23, upper bound 3.63 > margin 0.50 pointsNot supported
- PDF T5 scan, image onlyvs Tesseract: Character error, difference −3.38, upper bound 0.51 > margin 0.50 points; vs PyMuPDF4LLM: Numbers missed, difference 3.11, upper bound 9.41 > margin 1.00 pointsNot supported
- PDF T6 text plus text as imagesheld back by the real-document check: reversed amountsNot supported
- PDF spread two pages on one sheetSupported
- Scanned tables pages with a ruled tableSupported