latin-bert / tests /test_tokenizer.py

Commit History

fix: whitespace-split before subword encoding (full-string tokenization)
b8d6c6c

diyclassics Claude Opus 4.8 (1M context) commited on

fix: add do_lower_case=True to tokenizer (v1.1.1)
c7b1be1

diyclassics Claude Opus 4.6 (1M context) commited on

Fix tokenizer ID offset: reserve IDs 0-4 for BERT special tokens
ce59834

diyclassics Claude Opus 4.6 commited on

Initial: HF-compatible Latin BERT tokenizer (Bamman & Burns 2020)
68d8806

diyclassics Claude Opus 4.6 commited on