htext

Openly licensed Hebrew text for training HeOCR's recognizers, cut into lines, plain and with niqqud: public-domain text first, and attribution and share-alike text in their own releases. Every work or document carries its own rights record: its status in Israel and in the US, its licence and its credit line. htext is published as three releases, one per licence, so you can pick the terms you need; the core release is CC0 and can be used without conditions.

Updated 2026-11-18. Figures come from each corpus card and are rebuilt from corpora.json.

What htext is

htext collects Hebrew text whose rights are clear enough to publish openly, and turns it into training lines for hocrsyngen, HeOCR's synthetic page generator. The generator renders each line onto synthetic pages, so the recognizers learn real Hebrew vocabulary, spelling and punctuation, not only made-up text.

Each source has its own builder, its own corpus and its own card. A build reads a pinned copy of the source, computes rights per entry, cleans and cuts the text, and checks that no line overlaps HeOCR's evaluation text. Only the files a card lists are published.

14,252,079lines
27,310works, laws and books
693.9 MBof text
3corpora

Releases

htext ships one release per licence. Pick the release whose terms fit your use. Models trained on any of them can still be released under an open licence, with credit in their NOTICE.

htext-cc0

publishedCC0 1.0

htext-by

plannedCC BY 4.0 (per source)

htext-by-sa

plannedCC BY-SA 4.0

Corpora

Project Ben-Yehuda public-domain lines

pby-v3htext-cc0

Prose, articles, memoirs, letters, fables, poetry, reference works and drama transcribed by Project Ben-Yehuda volunteers, cut into lines, plain and with niqqud.

FileVariantSubsetLinesWorksSize
pby_lines_il_plain_v3.txtIsraelplain12,278,84816,409559.0 MB
pby_lines_il_niqqud_v3.txtIsraelniqqud1,680,29015,738119.9 MB
pby_lines_il_us_plain_v3.txtIsrael + USplain407,45656419.0 MB
pby_lines_il_us_niqqud_v3.txtIsrael + USniqqud18,6133251.3 MB

Israel: every line comes from a work that is public domain in Israel. Israel + US: the subset that is also public domain in the US. niqqud lines are pointed; plain lines carry no points.

Knesset laws as published

knesset-laws-v1htext-cc0

The enacted laws of the first eight Knessets (1949–1977), as printed in Reshumot (Sefer HaChukim), cut into lines, with margin notes as lines of their own.

FileVariantSubsetLinesLawsSize
knesset_laws_lines_plain_v1.txtIsraelplain234,8212,00410.4 MB

Tanakh: the Westminster Leningrad Codex

tanakh-v1htext-cc0

The Hebrew Bible, all 39 books, as the Westminster Leningrad Codex (WLC 4.20) gives it, cut into lines in two forms: with niqqud (no cantillation) and plain.

FileVariantSubsetLinesBooksSize
tanakh_lines_niqqud_v1.txtIsraelniqqud58,120394.6 MB
tanakh_lines_plain_v1.txtIsraelplain58,074392.7 MB

Rights

Every entry has a rights record with two fields, pd_israel and pd_us, each with the basis it was computed from and the year it holds for. A corpus variant includes an entry only if its record passes that variant's rule. The records are published with each corpus.

A translation counts as public domain only if both its author and its translator are. Every entry also records its source licence and credit line, so each release can be rebuilt and filtered. No release takes text under a non-commercial (NC) or no-derivatives (ND) licence, and text never crosses releases: a CC0 build never contains a CC BY or CC BY-SA line.

The rights fields are the project's reading of the law, not legal advice.

Download and use

The corpus files are assets of each corpus's GitHub release, and the card, checksums and rights index are in the repository. Every file is UTF-8 with one line per row, NFC normalised, with no blank lines.

Download the files and check them against the published checksums:

gh release download pby-v3 -R HeOCR/htext
curl -LO https://raw.githubusercontent.com/HeOCR/htext/main/data/pby/v3/SHA256SUMS
grep -E '_v3\.txt$' SHA256SUMS | sha256sum -c

Use a file as training text for hocrsyngen, which records it as external:sha256:<hex>:

hocrsyngen ... --text-corpus pby_lines_il_plain_v3.txt

Or rebuild a corpus from its source: each corpus has a build guide in the repository, for example the PBY guide.

Credit

Credit is not required by the licence, but some sources ask for it, and HeOCR gives it. Please carry these lines into any dataset or model card that uses a corpus:

Credit: Project Ben-Yehuda volunteers

Credit: The Knesset of Israel (Knesset database, laws as published in Reshumot)

Credit: Westminster Leningrad Codex (public domain), via the Open Scriptures Hebrew Bible

Licence

Each release carries its own licence, listed under Releases; the CC BY and CC BY-SA terms bind the source text. HeOCR's own processing and metadata, meaning the rights and line indexes, exclusion lists, cards and checksums, are dedicated to the public domain under CC0 1.0 in every release, as far as they can be separated from the text. The builder code is under the MIT licence.