ayse-solmaz/glm53-flash-from-scratch

GLM-5.3-Flash-inspired 25.7M model, trained from scratch

A 25,730,592-parameter byte-level language model (about 9.8M active per token). It was built from scratch in PyTorch, pretrained on synthetic Python code and then improved with executable-reward reinforcement learning (RLOO). All training ran on a single laptop CPU.

The architecture follows the ideas of GLM-5.3-Flash at toy scale:

• hybrid linear + sparse attention in a 3:1 pattern
• Mixture-of-Experts: 8 routed experts, top-2, plus 1 shared expert
• 4 learned residual streams (hyper-connections)
• RoPE

This is an educational model, not a general-purpose assistant. It only knows how to complete 8 tiny families of Python functions, such as increment, double and even. It is not a reproduction of GLM-5.3-Flash and is not affiliated with Z.ai.

📄 Code, full research report and lessons: https://github.com/ayse-solmaz/glm53-flash-learning

Results

Final evaluation used a held-out confirm set (seed 7) of 64 unseen functions, 8 per family. The table reports greedy pass@1:

	before RL	RL (group 64)	RL + solution-only replay (this model)
RL-trained families (24)	8	17	22
untrained families (40)	24	16	31
total (64)	32	33	53

• On RL-trained families this model gains 16 tasks and loses 2 relative to before RL (McNemar exact p = 0.0013).
• Known failure: is_positive gets 0/8 greedy and 6/8 pass@8. RL on even pushed up the % token and flipped a fragile > decision. The full analysis is in the GitHub report.

Files

Path	What
model.safetensors, config.json	main model: RL with group size 64 and solution-only replay, group 96
pretrain-100/	the pretraining checkpoint (step 100) that RL started from, for comparison

Usage

The architecture is custom, so transformers cannot load it. Use the code from the GitHub repo:

git clone https://github.com/ayse-solmaz/glm53-flash-learning
cd glm53-flash-learning
pip install -r requirements.txt

from glm53_flash.model import GLM53FlashFromScratch
from glm53_flash.generate import generate, completion_text
from glm53_flash.tokenizer import BOS, encode

model = GLM53FlashFromScratch.from_pretrained("ayse-solmaz/glm53-flash-from-scratch")
# before-RL checkpoint: from_pretrained("ayse-solmaz/glm53-flash-from-scratch", subfolder="pretrain-100")

prompt = 'def increment_9f3c(x):\n    """Add one to x."""\n'
out = generate(model, [BOS] + encode(prompt), max_new_tokens=32)[0]
print(prompt + completion_text(out))
# def increment_9f3c(x):
#     """Add one to x."""
#     return x + 1

Training summary

• Tokenizer: raw UTF-8 bytes plus PAD, BOS, EOS and SEP (vocab 260). Context length 192.
• Pretraining: 400 steps, batch 32 × 128, AdamW at lr 3e-4, 1.64M tokens of synthetic Python, about 26 minutes on CPU.
• RL (RLOO): started from pretraining step 100.
  • 96 groups, 64 samples per prompt, temperature 0.35, lr 5e-5.
  • Only the last block, final norm and the tied embedding/head are updated (2.19M parameters).
  • Reward is computed by executing the code against hidden tests: +1 if all pass, 0 if wrong, −0.1 if the code is invalid. A fail-closed AST whitelist runs before execution.
  • Each step adds a replay cross-entropy loss on pretraining examples, computed only on solution tokens (coefficient 1.0).

Limitations

• The tasks are tiny and easy. Each family has one canonical solution, and the function name reveals the family.
• Evaluation uses 8 tasks per family. Family-level numbers are noisy, and each run used a single seed.
• The model only writes single-line return bodies for the 8 families. Anything else produces junk.

Credits

The architecture and experimental design are inspired by vukrosic/glm-5.3-flash-from-scratch (MIT). The code was written independently as a learning project.