HeOCR benchmarks

How HeOCR compares with other open tools, per kind of document. Every figure gives its sample size, a 95% interval, the measure, the reference and the run it comes from. The page is built from one file, results.json, which is updated after each scored run.

Updated 2026-11-19.

Documents by format

Character error by document type

lower is better

On the held-out set, heocr has the lowest character error on all 9 document types that another tool also reads; its 95% interval is clear of every other reader on 7 of them. DOC is read by heocr alone.

  • heocr ours
  • Best other open tool the lowest of Docling, MarkItDown and PyMuPDF4LLM
  • Stock Tesseract heb, tessdata_best
0255075100%Document typeReaderCharacter error95% intervalDOCXWord (.docx)n = 80heocr0.30%0.21–0.38Docling3.3%2.8–3.7Tesseractnot scored on this type–DOClegacy Word (.doc)n = 40heocr0.77%0.60–0.94Other open toolsno other open tool reads this format–Tesseractnot scored on this type–PDF T1correct text layern = 316heocr0.02%0.02–0.03Docling21.6%19.2–24.2Tesseract (n=80)8.6%7.9–9.3PDF T2visual-order text layern = 40heocr0.02%0.01–0.05Docling10.6%9.7–11.6Tesseractnot scored on this type–PDF T3broken character mapn = 146heocr0.16%0.10–0.24PyMuPDF4LLM65.7%61.6–70.2Tesseract (n=66)9.2%8.5–10.0PDF T4scan, wrong OCR layern = 40heocr5.8%2.8–11.3PyMuPDF4LLM80.8%80.1–81.6Tesseract8.0%7.0–9.0PDF T5scan, image onlyn = 40heocr5.5%2.7–10.9PyMuPDF4LLM82.9%82.8–83.1Tesseract8.0%7.0–9.0PDF T6text plus text as imagesn = 40heocr0.23%0.08–0.44Docling13.4%12.6–14.2Tesseractno character-error score; see table–PDF spreadtwo pages on one sheetn = 40heocr0.02%0.01–0.05Docling71.8%66.3–76.0Tesseractnot scored on this type–Scanned tablespages with a ruled tablen = 120heocr1.4%1.2–1.6Docling69.7%69.2–70.1Tesseract59.5%57.0–61.9 050100%DOCX  Word (.docx)n = 80heocr0.30%0.21–0.38Docling3.3%2.8–3.7Tesseractnot scored on this typeDOC  legacy Word (.doc)n = 40heocr0.77%0.60–0.94Other open toolsno other open tool reads this formatTesseractnot scored on this typePDF T1  correct text layern = 316heocr0.02%0.02–0.03Docling21.6%19.2–24.2Tesseract (n=80)8.6%7.9–9.3PDF T2  visual-order text layern = 40heocr0.02%0.01–0.05Docling10.6%9.7–11.6Tesseractnot scored on this typePDF T3  broken character mapn = 146heocr0.16%0.10–0.24PyMuPDF4LLM65.7%61.6–70.2Tesseract (n=66)9.2%8.5–10.0PDF T4  scan, wrong OCR layern = 40heocr5.8%2.8–11.3PyMuPDF4LLM80.8%80.1–81.6Tesseract8.0%7.0–9.0PDF T5  scan, image onlyn = 40heocr5.5%2.7–10.9PyMuPDF4LLM82.9%82.8–83.1Tesseract8.0%7.0–9.0PDF T6  text plus text as imagesn = 40heocr0.23%0.08–0.44Docling13.4%12.6–14.2Tesseractno character-error score; see tablePDF spread  two pages on one sheetn = 40heocr0.02%0.01–0.05Docling71.8%66.3–76.0Tesseractnot scored on this typeScanned tables  pages with a ruled tablen = 120heocr1.4%1.2–1.6Docling69.7%69.2–70.1Tesseract59.5%57.0–61.9

Run gate-rc1-synth2026-11-19Release gate: 7 of 10 supported

Held-out set: the release gate's fresh generated documents, made after heocr 0.1.0rc1's code was frozen.

One linear axis from 0 to 100%; bars start at zero. Whiskers show the interval (95% percentile bootstrap over documents, 2000 resamples, seed 0). Where an interval is narrower than its marker, read it from the numbers beside it.

Documents by format: every reader and every measure

Modern printed Hebrew

No data yet
No data yet — reference pages are under human review. No number is shown until a reviewed set is scored.

Historical and rabbinic printed Hebrew

Not measured on a held-out set yet
Not measured on a held-out set yet. The historical-print test set is waiting for volunteer references; no figure is shown until it is scored.

Handwriting

No data yet
No data yet — there is no scored handwriting set with real, human-checked references.

Documents by format

Scored 2026-11-19

Generated documents with exact gold: Word files and PDFs of every kind HeOCR's document reader handles, made by our generators so the correct text is known exactly. Each table is one kind of document. These figures come from the release gate's fresh set: generated with new seeds after heocr 0.1.0rc1's code was frozen, so heocr was never tuned on these documents. They are drawn by the same generators heocr is developed against, so a good figure means good on generated documents of this kind, not more.

ReaderCharacter errorlower is betterWord errorlower is betterNumbers foundhigher is betterText lost without a warninglower is better
Word (.docx) · DOCX · 80 documentsSupportedSection numbers correcthigher is bettern = 60
heocrours0.30%0.21–0.38%0.36%0.26–0.46%100.0%100.0–100.0%100.0%100.0–100.0%
Docling3.3%2.8–3.7%8.1%7.3–8.8%93.9%91.9–95.6%7.9%6.1–10.0%
MarkItDown4.0%3.6–4.5%7.5%7.0–7.9%77.3%73.7–80.7%2.9%2.2–3.8%
Legacy Word (.doc) · DOC · 40 documentsSupported
heocrours0.77%0.60–0.94%1.9%1.4–2.5%91.4%88.3–94.2%
PDF with a correct text layer · PDF T1 · 316 documentsSupported
heocrours0.02%0.02–0.03%0.20%0.16–0.25%100.0%100.0–100.0%0.02%0.02–0.03%
Tesseract (tessdata_best heb)n = 808.6%7.9–9.3%16.9%15.7–18.2%47.2%45.5–48.9%7.9%7.3–8.5%
Docling21.6%19.2–24.2%72.5%71.4–73.7%96.6%95.6–97.6%13.5%10.9–16.4%
MarkItDown56.7%53.5–59.8%73.6%69.7–77.2%99.1%98.7–99.3%40.9%38.5–43.2%
PyMuPDF4LLM32.1%31.6–32.6%45.6%44.7–46.5%91.1%90.6–91.6%21.8%21.3–22.2%
PDF whose text layer is in visual (reversed) order · PDF T2 · 40 documentsSupported
heocrours0.02%0.01–0.05%0.05%0.01–0.09%100.0%100.0–100.0%0.02%0.01–0.04%
Docling10.6%9.7–11.6%19.0%17.9–20.2%95.5%94.4–96.5%7.0%6.3–7.7%
MarkItDown78.3%78.0–78.6%98.7%98.5–98.9%100.0%100.0–100.0%60.9%58.5–63.5%
PyMuPDF4LLM71.2%70.9–71.5%91.4%91.2–91.6%100.0%100.0–100.0%51.5%50.6–52.7%
PDF with a broken but recoverable character map · PDF T3 · 146 documentsSupported
heocrours0.16%0.10–0.24%0.63%0.43–0.86%99.9%99.6–100.0%0.14%0.10–0.19%
Tesseract (tessdata_best heb)n = 669.2%8.5–10.0%19.4%18.2–20.8%46.9%45.1–48.7%8.5%7.8–9.2%
Docling78.7%70.6–87.2%276.6%251.7–303.3%99.5%99.3–99.7%40.6%35.9–45.5%
MarkItDown70.7%66.7–74.5%88.2%84.2–92.2%100.0%100.0–100.0%59.2%55.4–62.9%
PyMuPDF4LLM65.7%61.6–70.2%87.5%81.2–94.8%95.0%94.2–95.7%53.7%49.8–58.0%
Scan with a wrong OCR text layer · PDF T4 · 40 documentsNot supportedT4 and T5 use the same scanned pages; heocr and Tesseract read both by OCR, so their figures match. T4 adds a wrong text layer, which the other readers use.
heocrours5.8%2.8–11.3%13.7%7.5–25.5%78.8%75.9–81.6%2.1%1.6–2.8%
Tesseract (tessdata_best heb)8.0%7.0–9.0%14.5%13.0–16.1%47.5%45.2–49.8%7.4%6.5–8.3%
Docling81.5%81.1–81.8%97.4%97.2–97.6%69.2%66.7–71.5%79.3%78.6–80.0%
MarkItDown81.4%80.1–82.5%102.1%101.2–103.0%16.5%14.6–18.4%65.9%63.9–68.0%
PyMuPDF4LLM80.8%80.1–81.6%99.7%99.4–100.0%14.6%12.8–16.5%61.9%60.7–63.4%
Scan, image only · PDF T5 · 40 documentsNot supported
heocrours5.5%2.7–10.9%13.3%7.3–25.1%79.1%76.2–81.8%1.8%1.5–2.1%
Tesseract (tessdata_best heb)8.0%7.0–9.0%14.5%13.0–16.1%47.5%45.2–49.8%7.4%6.5–8.3%
Docling84.1%83.5–84.7%97.1%96.9–97.4%68.0%65.3–70.5%82.3%81.5–83.0%
MarkItDown100.0%100.0–100.0%100.0%100.0–100.0%0.00%0.00–0.00%100.0%100.0–100.0%
PyMuPDF4LLM82.9%82.8–83.1%99.0%98.9–99.2%73.4%69.9–76.7%78.6%78.0–79.2%
Text layer plus text drawn as images · PDF T6 · 40 documentsNot supportedWords in image regions foundhigher is better
heocrours0.23%0.08–0.44%0.34%0.19–0.55%99.7%99.4–99.9%0.14%0.01–0.34%95.8%92.5–98.1%
Tesseract (tessdata_best heb)––––61.9%58.8–66.1%
Docling13.4%12.6–14.2%68.2%66.1–70.3%99.9%99.6–100.0%4.6%3.9–5.2%67.5%61.2–73.8%
MarkItDown44.3%35.7–52.8%58.0%47.4–68.4%89.2%87.6–90.7%32.2%26.1–38.4%15.5%9.8–21.7%
PyMuPDF4LLM33.2%31.8–34.5%44.5%42.8–46.3%80.3%78.6–82.1%23.9%22.4–25.5%25.4%19.5–31.6%
Two pages on one sheet · PDF spread · 40 documentsSupported
heocrours0.02%0.01–0.05%0.05%0.01–0.09%100.0%100.0–100.0%0.02%0.00–0.04%
Docling71.8%66.3–76.0%121.4%115.8–125.6%99.8%99.5–100.0%4.3%3.6–5.0%
MarkItDown74.7%70.7–77.9%91.8%86.4–96.0%100.0%100.0–100.0%25.4%19.5–31.5%
PyMuPDF4LLM74.6%70.9–77.3%91.5%86.8–94.9%91.0%89.6–92.4%20.4%19.2–21.4%
ReaderTable rows readhigher is betterRows lost without a warninglower is betterCharacter errorlower is betterNumbers foundhigher is better
Scanned pages with a ruled table · Scanned tables · 120 documentsSupported
heocrours95.9%94.6–97.0%0.06%0.00–0.19%1.4%1.2–1.6%97.6%97.1–98.2%
Tesseract (tessdata_best heb)2.3%1.4–3.1%97.7%96.9–98.6%59.5%57.0–61.9%13.3%12.4–14.1%
Docling2.3%1.7–2.9%97.7%97.0–98.3%69.7%69.2–70.1%96.1%95.4–96.7%
MarkItDown0.00%0.00–0.00%100.0%100.0–100.0%100.0%100.0–100.0%0.00%0.00–0.00%
PyMuPDF4LLM0.00%0.00–0.00%100.0%100.0–100.0%90.9%90.6–91.2%3.2%2.6–3.9%

A dash means the reader has no score on that measure. An empty cell means the measure does not apply to that document type. The last column holds a measure that exists for one type only; its name is in that type's heading row.

Each table lists the readers that handle that format. Tesseract reads images only, so it appears on scans; on PDF T6 it is scored on the image regions only.

Word error counts inserted words as well as wrong and missing ones, so it can exceed 100%.

Method, reference and run
  • Measure: line_fold_v1 (text normalised the same way for every reader; ratios of sums over documents)
  • Reference: the release gate's fresh generated set (the cases of D-bench sets A, B and C, new seeds after the code freeze): generated documents with exact gold; text from the public-domain Israeli official gazette (Reshumot)
  • Licence: PD-IL (text)
  • Interval: 95% percentile bootstrap over documents, 2000 resamples, seed 0
  • Run: gate-rc1-synth; heocr at 57e58de
  • Reader versions: Tesseract (tessdata_best heb) tesseract 5.5, heb tessdata_best e12c65a (sha256 dbaa827a), psm 3, pages at 300 dpi gray (pdftoppm); Docling 2.150.1; MarkItDown 0.1.9 [pdf,docx]; PyMuPDF4LLM 2.1 (AGPL; benchmark baseline only, decision 0020 item 2)
  • Left out: two office-scan cases rendered with a proprietary font (their scores cannot be published), and the provisional section-number case from the label measure
  • Scoring code: HeOCR's evaluation repository (not yet public; it is published with the release gate)

Commercial OCR vendors

Pending
Scores pending human review of the reference pages.

Release gate

The release gate scores a fresh set, made after the code is frozen. Each document type then gets one verdict: supported, or not supported. Run gate-rc1: heocr 0.1.0rc1 (commit 57e58de), scored 2026-11-19. The verdict is set by the one-sided 97.5% upper bound of heocr's difference from each other reader, against a fixed margin.

7Supported·3Not supported