What htext is
htext collects Hebrew text whose rights are clear enough to publish openly, and turns it into training lines for hocrsyngen, HeOCR's synthetic page generator. The generator renders each line onto synthetic pages, so the recognizers learn real Hebrew vocabulary, spelling and punctuation, not only made-up text.
Each source has its own builder, its own corpus and its own card. A build reads a pinned copy of the source, computes rights per entry, cleans and cuts the text, and checks that no line overlaps HeOCR's evaluation text. Only the files a card lists are published.
Releases
htext ships one release per licence. Pick the release whose terms fit your use. Models trained on any of them can still be released under an open licence, with credit in their NOTICE.
htext-cc0
publishedCC0 1.0- Contains: Public-domain text, CC0 text, and official texts that are not protected by copyright (laws, regulations, Knesset records, under section 6 of the Israeli Copyright Act).
- You may: Any use, with no conditions. This is the default release for training and benchmarks.
- Licence: CC0 1.0
htext-by
plannedCC BY 4.0 (per source)- Contains: Text whose source licence asks only for attribution (CC BY or equivalent).
- You may: Any use, as long as you credit the sources. Each entry carries its credit line, and the release ships a consolidated NOTICE.
- Licence: CC BY 4.0 (per source)
htext-by-sa
plannedCC BY-SA 4.0- Contains: Share-alike text, mainly Hebrew Wikisource transcriptions and Sefaria versions licensed CC BY-SA.
- You may: Any use, with credit, and anything you share that adapts the text must be under CC BY-SA too.
- Licence: CC BY-SA 4.0
Corpora
Project Ben-Yehuda public-domain lines
pby-v3htext-cc0Prose, articles, memoirs, letters, fables, poetry, reference works and drama transcribed by Project Ben-Yehuda volunteers, cut into lines, plain and with niqqud.
| File | Variant | Subset | Lines | Works | Size |
|---|---|---|---|---|---|
pby_lines_il_plain_v3.txt | Israel | plain | 12,278,848 | 16,409 | 559.0 MB |
pby_lines_il_niqqud_v3.txt | Israel | niqqud | 1,680,290 | 15,738 | 119.9 MB |
pby_lines_il_us_plain_v3.txt | Israel + US | plain | 407,456 | 564 | 19.0 MB |
pby_lines_il_us_niqqud_v3.txt | Israel + US | niqqud | 18,613 | 325 | 1.3 MB |
Israel: every line comes from a work that is public domain in Israel. Israel + US: the subset that is also public domain in the US. niqqud lines are pointed; plain lines carry no points.
- Release: htext-cc0
- Source: Project Ben-Yehuda (Public domain (PBY's dump licence))
- Release: pby-v3, 2026-11-18
- Corpus card: CORPUS_CARD.md
- Earlier version: pby-v2 (known issues)
- Earlier version: pby-v1 (known issues)
- Text: transcribed by people.
- 25,267 of 26,455 catalogue works have lines in this corpus; 701 also in the Israel + US variant.
Knesset laws as published
knesset-laws-v1htext-cc0The enacted laws of the first eight Knessets (1949–1977), as printed in Reshumot (Sefer HaChukim), cut into lines, with margin notes as lines of their own.
| File | Variant | Subset | Lines | Laws | Size |
|---|---|---|---|---|---|
knesset_laws_lines_plain_v1.txt | Israel | plain | 234,821 | 2,004 | 10.4 MB |
- Release: htext-cc0
- Source: The Knesset of Israel (Not protected: Israel Copyright Act 2007, section 6)
- Release: knesset-laws-v1, 2026-11-18
- Corpus card: CORPUS_CARD.md
- Text: the OCR layer of the scans, filtered against a lexicon. An eye check of 100 random lines against the scans found an error in 9 (9%) and 9 wrong words of 454 (2.0%).
- 2,004 of 2,146 laws in the catalogue have lines in this corpus.
Tanakh: the Westminster Leningrad Codex
tanakh-v1htext-cc0The Hebrew Bible, all 39 books, as the Westminster Leningrad Codex (WLC 4.20) gives it, cut into lines in two forms: with niqqud (no cantillation) and plain.
| File | Variant | Subset | Lines | Books | Size |
|---|---|---|---|---|---|
tanakh_lines_niqqud_v1.txt | Israel | niqqud | 58,120 | 39 | 4.6 MB |
tanakh_lines_plain_v1.txt | Israel | plain | 58,074 | 39 | 2.7 MB |
- Release: htext-cc0
- Source: Westminster Leningrad Codex, via the Open Scriptures Hebrew Bible (Public domain)
- Release: tanakh-v1, 2026-11-18
- Corpus card: CORPUS_CARD.md
- Text: transcribed by people.
- All 39 books have lines in this corpus, in both forms. The pointed and plain files hold the same verses, so the totals above count them once.
Rights
Every entry has a rights record with two fields, pd_israel and pd_us, each with the basis it was computed from and the year it holds for. A corpus variant includes an entry only if its record passes that variant's rule. The records are published with each corpus.
A translation counts as public domain only if both its author and its translator are. Every entry also records its source licence and credit line, so each release can be rebuilt and filtered. No release takes text under a non-commercial (NC) or no-derivatives (ND) licence, and text never crosses releases: a CC0 build never contains a CC BY or CC BY-SA line.
The rights fields are the project's reading of the law, not legal advice.
- Project Ben-Yehuda: Public domain in Israel: every author and every translator died at least 71 years before the build year (2026), with death years from Wikidata or, where Wikidata has none, from a reviewed source such as the author's PBY page. US status is recorded per work: public domain in the US when the transcribed edition appeared before 1931.
- The Knesset of Israel: Laws are not protected by copyright in Israel (section 6 of the Copyright Act 2007). US status is not assessed.
- Westminster Leningrad Codex, via the Open Scriptures Hebrew Bible: The Leningrad Codex is about a thousand years old, and the WLC text is in the public domain ("The text of the WLC remains in the Public Domain", OSHB). Public domain in Israel and in the US. Only the words are used; OSHB's CC BY lemma and morphology layers are not. The edition is WLC, not UXLC.
Download and use
The corpus files are assets of each corpus's GitHub release, and the card, checksums and rights index are in the repository. Every file is UTF-8 with one line per row, NFC normalised, with no blank lines.
Download the files and check them against the published checksums:
gh release download pby-v3 -R HeOCR/htext
curl -LO https://raw.githubusercontent.com/HeOCR/htext/main/data/pby/v3/SHA256SUMS
grep -E '_v3\.txt$' SHA256SUMS | sha256sum -c
Use a file as training text for hocrsyngen, which records it as external:sha256:<hex>:
hocrsyngen ... --text-corpus pby_lines_il_plain_v3.txt
Or rebuild a corpus from its source: each corpus has a build guide in the repository, for example the PBY guide.
Credit
Credit is not required by the licence, but some sources ask for it, and HeOCR gives it. Please carry these lines into any dataset or model card that uses a corpus:
Credit: Project Ben-Yehuda volunteers
Credit: The Knesset of Israel (Knesset database, laws as published in Reshumot)
Credit: Westminster Leningrad Codex (public domain), via the Open Scriptures Hebrew Bible
Licence
Each release carries its own licence, listed under Releases; the CC BY and CC BY-SA terms bind the source text. HeOCR's own processing and metadata, meaning the rights and line indexes, exclusion lists, cards and checksums, are dedicated to the public domain under CC0 1.0 in every release, as far as they can be separated from the text. The builder code is under the MIT licence.