Hugging Face
Models
Datasets
Spaces
Community
Docs
Enterprise
Pricing
Log In
Sign Up
ggunio
/
intelligent-tokenizer-v6
like
0
PyTorch
flores200
11 languages
intelligent_tokenizer
tokenization
byte-level
neural-tokenizer
pattern-learning
vocabulary-free
Eval Results
License:
mit
Model card
Files
Files and versions
xet
Community
main
intelligent-tokenizer-v6
/
src
8.66 kB
1 contributor
History:
1 commit
ggunio
Upload src/core/byte_tokenizer_v6.py with huggingface_hub
0a81e70
verified
3 months ago
core
Upload src/core/byte_tokenizer_v6.py with huggingface_hub
3 months ago