Hugging Face
Models
Datasets
Spaces
Community
Docs
Enterprise
Pricing
Log In
Sign Up
ggunio
/
intelligent-tokenizer-v6
like
0
PyTorch
flores200
11 languages
intelligent_tokenizer
tokenization
byte-level
neural-tokenizer
pattern-learning
vocabulary-free
Eval Results
License:
mit
Model card
Files
Files and versions
xet
Community
main
intelligent-tokenizer-v6
/
app.py
Commit History
Upload app.py with huggingface_hub
2d68769
verified
ggunio
commited on
Sep 14