Sai-Slice1-60M-base

---

Sai-Slice1-60M-base is a small Mixture-of-Experts (MoE) decoder-only language model, pretrained from scratch on ~1.2B tokens of English web text. It is the first release (slice 1) of the Sai series.

This is a base model: it continues text and has not been instruction-tuned. A chat fine-tune (Sai-Slice1-60M-Chatter) is coming soon.

Total parameters	~60.6M
Active parameters per token	~19.3M
Training tokens	~1.19B (1 epoch)
Context length	1,024
Training compute	24.9 h on a single RTX 4060 (8 GB)
Test cross-entropy	2.101 (perplexity ≈ 8.2)

Model details

Component	Setting
Architecture	Decoder-only transformer, pre-norm
Layers	15 (1 dense + 14 MoE)
Hidden size	256
Attention	Multi-head attention, 8 heads (head dim 32)
Position embedding	RoPE (base 10,000)
Normalization	RMSNorm
Feed-forward	SwiGLU
FFN hidden dim	1,024 (dense layer) / 192 (each expert)
Experts per MoE layer	24 routed + 1 shared
Router top-k	4
Vocabulary	16,384 (byte-level BPE)
Embeddings	Tied input/output
KV cache	Yes
Weights dtype	float32

Each token is routed to 4 of the 24 routed experts, and always passes through the shared expert. The first layer uses a dense FFN; the other 14 use MoE. Routing is trained with a load-balancing auxiliary loss.

Parameter breakdown: token embeddings 4.19M (tied with the output head), attention 3.93M, dense FFN 0.79M, MoE layers 51.7M (14 × 25 experts × 147K + routers). Active per token counts 5 of the 25 experts in each MoE layer.

Usage

The model uses custom code, so trust_remote_code=True is required. It was built and tested with transformers==5.16.1 and torch==2.11.0.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "OliverSundaram/Sai-Slice1-60M-base"

model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(repo)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

text, tokens_per_sec, time_to_first_token = model.run(
    "The history of the printing press",
    tokenizer,
    device,
    max_new_tokens=100,
    top_k=20,
    temp=0.9,
    use_cached=True,
    format_prompt=False,  # plain text continuation for the base model
)
print(text)

model.run returns the prompt plus its continuation, the generation speed in tokens/sec, and the time to first token in seconds. Generation stops at <|end|>, at max_new_tokens, or at the 1,024-token context limit.

Notes:

• Keep format_prompt=False with this base model. format_prompt=True wraps the prompt in the chat template (<|system|>…<|end|><|user|>…<|end|><|assistant|>), which is intended for the upcoming fine-tuned model.
• Use model.run for generation. The Hugging Face generate() method is not supported, because the model manages its own KV cache instead of returning past_key_values.
• A standard forward pass (model(input_ids, attention_mask=...)) returns logits and works with tools such as lm-evaluation-harness.

Training data

Source	openbmb/Ultra-FineWeb-L3, subset Ultra-FineWeb-L3-en-Multi-Style-Synthetic
Tokens	~1.2B, streamed and shuffled (seed 42)
Split	99.5% train / 0.25% validation / 0.25% test
Document separator	<|end|>

Tokenizer: byte-level BPE trained on the first 1M documents of the same subset. Vocabulary size is 16,384, maximum token length is 16, and there are five special tokens: <|end|> (EOS), <|pad|>, <|user|>, <|assistant|> and <|system|>.

Training procedure

Hyperparameter	Value
Epochs	1
Optimizer steps	18,219
Sequence length	1,024
Batch size	8 × 8 gradient accumulation = 64 sequences (65,536 tokens) per step
Optimizer	AdamW (β₁ 0.9, β₂ 0.95, ε 1e-8)
Weight decay	0.1 (on weight matrices only)
Peak learning rate	3e-4
LR schedule	Warmup-stable-decay: 3% linear warmup, then constant, then 1 − √ decay to 0 over the final 20%
Gradient clipping	1.0
Aux (load-balancing) loss	coefficient 0.01
Precision	bf16 autocast
Dropout	0.0
Seed	42
Hardware	1× NVIDIA RTX 4060 (8 GB)
Training time	24.9 hours

Final results: test cross-entropy 2.101 (perplexity ≈ 8.2), test auxiliary loss 1.033.

Evaluation

Evaluated with lm-evaluation-harness v0.4.13 on the full test sets.

Task	Metric	Shots	Sai-Slice1-60M-base	Chance
SciQ	acc	0	62.3 ± 1.5	25.0
COPA	acc	0	58.0 ± 5.0	50.0
PIQA	acc_norm	0	56.5 ± 1.2	50.0
WinoGrande	acc	5	52.2 ± 1.4	50.0
TruthfulQA (MC2)	acc	0	47.6 ± 1.6	—
BoolQ	acc	0	47.5 ± 0.9	50.0
Social IQa	acc	0	34.4 ± 1.1	33.3
HellaSwag	acc_norm	5	28.2 ± 0.4	25.0
OpenBookQA	acc_norm	0	26.2 ± 2.0	25.0
ARC-Challenge	acc_norm	15	24.4 ± 1.3	25.0
RACE	acc	0	23.3 ± 1.3	25.0
MMLU	acc	0	23.2 ± 0.4	25.0
CommonsenseQA	acc	0	20.0 ± 1.1	20.0
LAMBADA (OpenAI)	acc	0	9.4 ± 0.4	—
LAMBADA (standard)	acc	0	8.3 ± 0.4	—

Task	Metric	Value
WikiText	word perplexity	148.35
WikiText	byte perplexity	2.547
LAMBADA (OpenAI)	perplexity	1,081

MMLU by category: humanities 24.3, STEM 21.9, social sciences 21.7, other 24.2.

MMLU by subject

Inference speed

Measured with model.run on 100 prompts from gharezlak/autocomplete, generating up to 100 new tokens (top-k 20, temperature 0.9, KV cache on), in float32 on an RTX 4060 (8 GB).

Metric	Mean	Median
Tokens / sec	10.7	10.7
Time to first token	0.18 s	0.16 s

The first prompt includes CUDA warmup (1.85 s time to first token). Throughput is currently limited by the MoE layer, which loops over experts in Python, rather than by model size.

Limitations

• Base model only. It continues text and does not follow instructions or hold a conversation.
• Small. With ~60M parameters and ~1.2B training tokens, it has very limited world knowledge and reasoning, and it often produces fluent but incorrect or incoherent text. Do not rely on its outputs for facts.
• English only, with a 1,024-token context window.
• Data bias. It was trained on filtered and synthetically rewritten web text and may reproduce the biases and errors in that data.
• It is intended for research, education and experimentation, not for production use.

Files

File	Description
model.safetensors	Model weights (float32)
config.json	Model configuration
modules.py	Model code (SaiConfig, Sai), loaded via trust_remote_code
tokenizer.json, tokenizer_config.json	Tokenizer
trainer_state.pt	Optimizer and scheduler state plus loss history, for resuming training or re-plotting
assets/	Charts used in this card

Citation

@misc{sundaram2026sai,
  title        = {Sai-Slice1-60M-base: A Small Mixture-of-Experts Language Model},
  author       = {Sundaram, Oliver},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/OliverSundaram/Sai-Slice1-60M-base}}
}

License

Released under the Apache 2.0 License.