Pascaline-3M

An experimental 3M language model with units-first number tokens. Pascaline-3M is a 2,984,322-parameter base model trained from scratch on one laptop. It scores 69.2 on ArithMark-3. That score comes mostly from synthetic arithmetic in its final training phase, and it does not hold up on held-out arithmetic (see the note below). We publish it as a research artifact, not as a general arithmetic model.

Held-out check. The Open SLM Leaderboard maintainers evaluated Pascaline-3M on their private arithmetic set, and it failed: it scored about 35% there against 69.2 on ArithMark-3, no better than 3M models trained without synthetic arithmetic (GPT-S3-3M 38.5, Tokle-3M 37.0), so its ArithMark-3 score does not carry over to differently formatted problems. It is therefore not listed on the leaderboard (discussion #242). Our own "held-out" checks came from the same generators as the training data, so they could not detect this. Without our synthetic data, the same model scores 39.0 on ArithMark-3, in line with other models of this size.

It uses the hybrid architecture of GPT-S3-3M (Gated DeltaNet-2 + gated GQA + Full Attention Residuals) with length-prefixed, units-first number tokens, and a final training phase in which 30% of the data was synthetic arithmetic generated by our own programs. It is named after Blaise Pascal's 1642 calculating machine, whose wheels carry from the units digit upward, the same direction this model writes numbers.

Parameters	2,984,322 (tied embeddings)
Layers	8 blocks of width 144: [Gated DeltaNet-2 ×3, gated GQA (3 query / 1 KV heads, head dim 48, partial RoPE 24/48)] ×2, SwiGLU 352
Residual stream	Full Attention Residuals (each sublayer reads a softmax mix of the embedding and all earlier outputs)
Numbers	Learned digit-place embedding (66 × 144); tokenizer writes <lenN> + digits units-first (see below)
Vocabulary	4,096 byte-level BPE ids (one token per digit) + 8 number-length tokens = 4,104
Context	1,024 tokens
Training tokens	10.0B (constant LR) + 1.2B decay = 11.2B
Hardware	one laptop RTX 5070 Ti, ~12.2 h of training

Results (zero-shot)

Benchmark	Metric	Score
PIQA	acc_norm	56.15
HellaSwag	acc_norm	27.79
ARC-Easy	acc_norm	34.18
ARC-Challenge	acc_norm	22.44
ArithMark-3	acc_norm (bencharithmark-3.py)	69.20
Open SLM Leaderboard Intelligence Index		16.09

Self-reported with the official tools. These figures are not leaderboard-verified, and the ArithMark-3 score does not generalize (see the held-out check above). For comparison, the same 10B-token model decayed without any self-generated data: PIQA 56.91, HellaSwag 27.84, ARC-E 33.21, ARC-C 20.22, ArithMark-3 39.0 (Index 8.77).

PIQA / HellaSwag / ARC: lm-evaluation-harness 0.4.13, --model hf, trust_remote_code=True, dtype=float32. ArithMark-3: AxiomicLabs' bencharithmark-3.py, --dtype float32. Run on a Tesla T4 (transformers 5.16.1, torch 2.11). float16 gives the same numbers within 0.2 points (II 16.08). Our training-time scorer gave PIQA 56.15, HellaSwag 27.91, ARC-E 34.22, ARC-C 22.53, ArithMark-3 69.1 (II 16.14).

How this checkpoint was chosen. The constant-LR run was branched at 4B, 8B and 10B tokens into eight 1.2B-token decay runs with different decay data. This is the one with the highest Intelligence Index on the benchmarks above, like GPT-S3-3M's release of its best internal checkpoint. The eight decay branches scored between 11.5 and 16.1; the four branches with the final synthetic-arithmetic recipe scored 15.4–16.1.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("ReckonerLabs/Pascaline-3M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("ReckonerLabs/Pascaline-3M", trust_remote_code=True)
ids = tok("Ava had 43 shells and found 19 more, so she had", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=6, do_sample=False, pad_token_id=tok.pad_token_id)
print(tok.decode(out[0][ids.shape[1]:]))   # " 62 shells"

• Remote code. Both the model and the tokenizer use remote code. The model is pure PyTorch: no flash-linear-attention needed, no KV cache (generation recomputes the prefix).
• Padding. Pad on the right (the default of lm-eval and bencharithmark-3.py); attention_mask is ignored.

Number tokens

Digits are single tokens. Every run of digits is written as a length token followed by its digits in reverse order: 62 → <len2> 2 6, 1234 → <len4> 4 3 2 1. The model learns the size of a number first, then produces it units digit first, so carries flow forward in time. Digit runs next to : (clock times) are left as they are. The tokenizer applies this on encode and undoes it on decode, and the model computes each digit's place (units, tens, …) from the ids. Units-first digits inside the tokenizer follow Atom2.7m; the length prefix is new here. In our ablation (4 seeds, 1B tokens, with 40% synthetic arithmetic in the decay) it raised ArithMark-3 by 9.2 points and the Index by 2.7 over normal digits, with the other benchmarks unchanged. Without synthetic data the effect on ArithMark-3 is small: the model computes bare sums like 43 + 19 = correctly ~99% of the time, but that does not carry over to word problems.

A note for interpreting ArithMark-3: its wrong options are the answer ±1 or ±2, so every option has a different units digit, which a units-first model emits first. This representation therefore helps there beyond full arithmetic ability.

Training data

Constant-LR phase:

Source	Share
ClimbMix	30%
TxT360-v2	28%
OpenMathInstruct-2 (MATH part)	12%
FineWeb-Edu	10%
UltraData-Math	8%
FactBase-DCLM	5%
OpenMathInstruct-2 (GSM8K part)	4%
Cosmopedia WikiHow	3%

Decay (1.2B tokens), mixed per step:

Source	Share
Cosmopedia WikiHow	40%
Our synthetic arithmetic	30%
OpenMathInstruct-2 GSM8K	9.4%
Cosmopedia OpenStax / Khan / Stanford	3.7%
ClimbMix	3.4%
FineWeb-Edu	3.4%
Orca-Math	2.6%
TxT360	1.7%
FactBase	1.7%
OpenMathInstruct-2 MATH	1.7%
UltraData-Math	1.7%
SciQ (train split)	0.5%
OpenBookQA (train split)	0.09%

Third-party synthetic data. OpenMathInstruct-2 (NVIDIA), Orca-Math (Microsoft) and Cosmopedia (Hugging Face) are LLM-generated datasets.

Our synthetic arithmetic is written by our own programs: grade-school word problems and equations with exact answers. It mixes four generators in equal parts:

• v2: broad topics: money, time, lengths, patterns, comparison, arrays, shapes, fractions.
• v3: compositional gain/loss stories, carry/borrow-balanced operands, clock arithmetic.
• v4: relations ("N more/fewer than X"), differences asked many ways, sharing / grouping division, measures, minimal pairs, equation tails.
• v4.1: multi-step chains with distractor sentences, two-hop relations, perimeters.

Phrasings are composed from parts split into train and held-out sets. This split only tests new combinations of our own templates. It did not predict performance on independently written problems.

Public sample. A 5,000-problem sample of this stream, with its overlap report, is at ReckonerLabs/Pascaline-3M-synthetic-sample.

Fairness checks against all ArithMark-3 rows, enforced at generation:

• 0% exact 8-gram and 13-gram overlap;
• under 0.01% overlap of number-masked 6-grams (shared first names);
• a banned list of ArithMark sentence frames, which is never generated.

No benchmark item, benchmark generator or train split of an evaluated benchmark was used. SciQ and OpenBookQA are not evaluated here. WikiHow, the GSM8K-derived sources, Orca-Math, FactBase, SciQ and OpenBookQA were filtered with a 13-gram overlap check against all five test sets. The large web sources were scanned on a sample.

Credits

• Architecture: GPT-S3-3M by AxiomicLabs (Apache-2.0).
• Decay-phase recipe ideas: pulvis-v1/v2 (bench-labs).
• Units-first digit tokens and place features: Atom2.7m.
• Precedent for generated arithmetic data: MathBananaMind-1.1.
• Kernels used in training: flash-linear-attention.

Limitations

• English only. A 3M model: it confabulates freely and should not be relied on for facts.
• Its arithmetic is grade-school word problems and multi-digit addition and subtraction. It is not a reasoning model.
• Benchmarks score it by ranking given answers. Free-form generation of an answer is less reliable: in our spot checks 3-digit sums and differences and familiar story phrasings come out right, while unfamiliar phrasings, division and measurement problems are often off by one or more.
• Base model only: no instruction tuning, no safety tuning.