Update README.md

90462a3 verified 4 months ago

6.06 kB

language:
  - ks
library_name: pytorch
pipeline_tag: text-generation
tags:
  - kashmiri
  - byte-level
  - causal-lm
  - spacebyte
  - pytorch
license: apache-2.0
datasets:
  - >-
    Omarrran/KS-PRET-5M_5_million_kashmiri_Pretrainning_LLM_dataset_12M_tokens_2026
model-index:
  - name: ks_byte_lm SpaceByte v1
    results:
      - task:
          type: text-generation
          name: Byte-level causal language modeling
        dataset:
          type: >-
            Omarrran/KS-PRET-5M_5_million_kashmiri_Pretrainning_LLM_dataset_12M_tokens_2026
          name: Kashmiri pretraining corpus
          split: validation
        metrics:
          - type: bpb
            name: Best validation bits-per-byte
            value: 0.9593
          - type: bpb
            name: Final validation bits-per-byte
            value: 0.9862
          - type: cross_entropy
            name: Final validation cross entropy
            value: 0.6836
          - type: accuracy
            name: Validation next-byte top-1 accuracy
            value: 0.7642
          - type: perplexity
            name: Final validation word perplexity estimate
            value: 849
          - type: train_tokens
            name: Training byte tokens
            value: 45362173
          - type: validation_tokens
            name: Validation byte tokens
            value: 1622371
          - type: test_tokens
            name: Test byte tokens
            value: 3074698
          - type: parameters
            name: Model parameters
            value: 15837440
          - type: training_steps
            name: Early-stopped training steps
            value: 4751

ks_byte_lm SpaceByte v1

Small byte-level causal language model trained from scratch on a Kashmiri pretraining corpus.

This repository is a full project/checkpoint release: source code, Modal training artifacts, metrics, and training checkpoints are included.

checkpoints/best.pt — best validation checkpoint; recommended for generation.
checkpoints/latest.pt — final early-stopped checkpoint.
checkpoints/ckpt_*.pt — retained periodic checkpoints.
artifacts/final_report.json — final training report.
artifacts/train.log — full training log.
artifacts/accuracy_val_best.json and accuracy_val_latest.json — held-out next-byte accuracy estimates.
artifacts/data_meta.json — cached byte-shard metadata.
source/ — full ks_byte_lm project source used for the run.

Key metrics

Validation next-byte accuracy using best.pt: 76.42% on 3,276,800 evaluated byte tokens.

Training summary:

Best validation BPB: 0.9593
Final validation BPB: 0.9862
Final CE: 0.6836
Final word PPL estimate: 849.0
Early stopped at step 4751 / 5000

Dataset/cache metadata:

Train byte tokens: 45,362,173
Validation byte tokens: 1,622,371
Test byte tokens: 3,074,698
Words: 5,074,066

Usage

This is not a standard Transformers-format model. Do not use AutoModel.from_pretrained. It is a custom PyTorch checkpoint release for the included ksbyte source code.

Recommended checkpoint:

checkpoints/best.pt

best.pt is preferred over latest.pt because it had the best validation BPB.

1. Download and install

If the repository is private, first login with an account that has access:

pip install -U huggingface_hub
huggingface-cli login

Download the full snapshot:

python - <<'PY'
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Omarrran/ks-byte-lm-spacebyte-v1",
    repo_type="model",
    local_dir="ks-byte-lm-spacebyte-v1",
    local_dir_use_symlinks=False,
)
PY

Install dependencies:

cd ks-byte-lm-spacebyte-v1
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

2. Generate text

Make the bundled source package importable and run generation:

export PYTHONPATH="$PWD/source:$PYTHONPATH"

PYTHONUTF8=1 python -m ksbyte.generate \
  --ckpt checkpoints/best.pt \
  --prompt "کٲشِرۍ پَلو چھِ اکثر خٕطہٕ کِس تٲریٖخٕچ دٔلیٖل ونان، یُس مُختٔلِف ثقافتَن ہٕنٛدۍ اثرات ظٲہِر کران" \
  --n 200 \
  --temperature 0.8 \
  --top_k 50 \
  --device auto

--device auto uses CUDA when available and falls back to CPU otherwise.

Example verified output from the trained checkpoint:

 کٲشِرۍ پَلو چھِ اکثر خٕطہٕ کِس تٲریٖخٕچ دٔلیٖل ونان، یُس مُختٔلِف ثقافتَن ہٕنٛدۍ اثرات ظٲہِر کران

3. Minimal Python API

import torch
from ksbyte.generate import load_model, encode_prompt, decode_tokens
from ksbyte.config import EOS_ID

ckpt = "checkpoints/best.pt"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model, cfg = load_model(ckpt, device)
idx = torch.tensor([encode_prompt("کٲشِرۍ پَلو چھِ اکثر خٕطہٕ کِس تٲریٖخٕچ دٔلیٖل ونان، یُس مُختٔلِف ثقافتَن ہٕنٛدۍ اثرات ظٲہِر کران", cfg)], dtype=torch.long, device=device)

out = model.generate(
    idx,
    max_new_tokens=200,
    temperature=0.8,
    top_k=50,
    eos_id=EOS_ID,
)
print(decode_tokens(out[0].tolist()))

Troubleshooting:

If ModuleNotFoundError: No module named 'ksbyte', run export PYTHONPATH="$PWD/source:$PYTHONPATH" from the repo root.
If CUDA is unavailable, use --device cpu or keep --device auto.
Lower --temperature such as 0.7 for more conservative generation.

Sample Outputs

Checkpoint notes

Use checkpoints/best.pt for generation and evaluation. checkpoints/latest.pt is the final early-stopped checkpoint and is slightly weaker than best.pt.

Caveats

This is a small experimental byte-level LM. It produces Kashmiri-looking text and has learned script/byte structure, but generations can be incomplete, hallucinated, or semantically weak.

License

No explicit open-source license is declared for this experimental release. Contact the repository owner for reuse permissions.

Omarrran
/

ks-byte-lm-spacebyte-v1