---
language:
- ks
library_name: pytorch
pipeline_tag: text-generation
tags:
- kashmiri
- byte-level
- causal-lm
- spacebyte
- pytorch
license: apache-2.0
datasets:
- Omarrran/KS-PRET-5M_5_million_kashmiri_Pretrainning_LLM_dataset_12M_tokens_2026

model-index:
- name: ks_byte_lm SpaceByte v1
  results:
  - task:
      type: text-generation
      name: Byte-level causal language modeling
    dataset:
      type: Omarrran/KS-PRET-5M_5_million_kashmiri_Pretrainning_LLM_dataset_12M_tokens_2026
      name: Kashmiri pretraining corpus
      split: validation
    metrics:
    - type: bpb
      name: Best validation bits-per-byte
      value: 0.9593
    - type: bpb
      name: Final validation bits-per-byte
      value: 0.9862
    - type: cross_entropy
      name: Final validation cross entropy
      value: 0.6836
    - type: accuracy
      name: Validation next-byte top-1 accuracy
      value: 0.7642
    - type: perplexity
      name: Final validation word perplexity estimate
      value: 849.0
    - type: train_tokens
      name: Training byte tokens
      value: 45362173
    - type: validation_tokens
      name: Validation byte tokens
      value: 1622371
    - type: test_tokens
      name: Test byte tokens
      value: 3074698
    - type: parameters
      name: Model parameters
      value: 15837440
    - type: training_steps
      name: Early-stopped training steps
      value: 4751
---

# ks_byte_lm SpaceByte v1

Small byte-level causal language model trained from scratch on a Kashmiri pretraining corpus.

This repository is a full project/checkpoint release: source code, Modal training artifacts, metrics, and training checkpoints are included.

## Contents

- `checkpoints/best.pt` — best validation checkpoint; recommended for generation.
- `checkpoints/latest.pt` — final early-stopped checkpoint.
- `checkpoints/ckpt_*.pt` — retained periodic checkpoints.
- `artifacts/final_report.json` — final training report.
- `artifacts/train.log` — full training log.
- `artifacts/accuracy_val_best.json` and `accuracy_val_latest.json` — held-out next-byte accuracy estimates.
- `artifacts/data_meta.json` — cached byte-shard metadata.
- `source/` — full `ks_byte_lm` project source used for the run.

## Key metrics

Validation next-byte accuracy using `best.pt`: **76.42%** on 3,276,800 evaluated byte tokens.

Training summary:

- Best validation BPB: **0.9593**
- Final validation BPB: **0.9862**
- Final CE: **0.6836**
- Final word PPL estimate: **849.0**
- Early stopped at step **4751 / 5000**

Dataset/cache metadata:

- Train byte tokens: `45,362,173`
- Validation byte tokens: `1,622,371`
- Test byte tokens: `3,074,698`
- Words: `5,074,066`

## Usage

This is **not** a standard Transformers-format model. Do **not** use
`AutoModel.from_pretrained`. It is a custom PyTorch checkpoint release for the
included `ksbyte` source code.

Recommended checkpoint:

```text
checkpoints/best.pt
```

`best.pt` is preferred over `latest.pt` because it had the best validation BPB.

### 1. Download and install

If the repository is private, first login with an account that has access:

```bash
pip install -U huggingface_hub
huggingface-cli login
```

Download the full snapshot:

```bash
python - <<'PY'
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Omarrran/ks-byte-lm-spacebyte-v1",
    repo_type="model",
    local_dir="ks-byte-lm-spacebyte-v1",
    local_dir_use_symlinks=False,
)
PY
```

Install dependencies:

```bash
cd ks-byte-lm-spacebyte-v1
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

### 2. Generate text

Make the bundled source package importable and run generation:

```bash
export PYTHONPATH="$PWD/source:$PYTHONPATH"

PYTHONUTF8=1 python -m ksbyte.generate \
  --ckpt checkpoints/best.pt \
  --prompt "کٲشِرۍ پَلو چھِ اکثر خٕطہٕ کِس تٲریٖخٕچ دٔلیٖل ونان، یُس مُختٔلِف ثقافتَن ہٕنٛدۍ اثرات ظٲہِر کران" \
  --n 200 \
  --temperature 0.8 \
  --top_k 50 \
  --device auto
```

`--device auto` uses CUDA when available and falls back to CPU otherwise.

Example verified output from the trained checkpoint:

```text
 کٲشِرۍ پَلو چھِ اکثر خٕطہٕ کِس تٲریٖخٕچ دٔلیٖل ونان، یُس مُختٔلِف ثقافتَن ہٕنٛدۍ اثرات ظٲہِر کران 
```

### 3. Minimal Python API

```python
import torch
from ksbyte.generate import load_model, encode_prompt, decode_tokens
from ksbyte.config import EOS_ID

ckpt = "checkpoints/best.pt"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model, cfg = load_model(ckpt, device)
idx = torch.tensor([encode_prompt("کٲشِرۍ پَلو چھِ اکثر خٕطہٕ کِس تٲریٖخٕچ دٔلیٖل ونان، یُس مُختٔلِف ثقافتَن ہٕنٛدۍ اثرات ظٲہِر کران", cfg)], dtype=torch.long, device=device)

out = model.generate(
    idx,
    max_new_tokens=200,
    temperature=0.8,
    top_k=50,
    eos_id=EOS_ID,
)
print(decode_tokens(out[0].tolist()))
```

Troubleshooting:

- If `ModuleNotFoundError: No module named 'ksbyte'`, run
  `export PYTHONPATH="$PWD/source:$PYTHONPATH"` from the repo root.
- If CUDA is unavailable, use `--device cpu` or keep `--device auto`.
- Lower `--temperature` such as `0.7` for more conservative generation.



### Sample Outputs


![image](https://cdn-uploads.huggingface.co/production/uploads/66afb3f1eaf3e876595627bf/R5cqHlpPwhb6qMvMOlGYw.png)

## Checkpoint notes

Use `checkpoints/best.pt` for generation and evaluation. `checkpoints/latest.pt` is the final early-stopped checkpoint and is slightly weaker than `best.pt`.

## Caveats

This is a small experimental byte-level LM. It produces Kashmiri-looking text and has learned script/byte structure, but generations can be incomplete, hallucinated, or semantically weak.

## License

No explicit open-source license is declared for this experimental release. Contact the repository owner for reuse permissions.