Jadson/Nanbeige4.2-3B-dlora-GGUF

Nanbeige4.2-3B dlora (GGUF, IQ4_NL)

Nanbeige/Nanbeige4.2-3B with a diffusion-generated FFN LoRA ("dlora"), in GGUF for llama.cpp. A small diffusion model, the LoRADiffuser (22 M parameters), generates a rank-16 LoRA for the gate, up, and down projections of every FFN, for every token. It's conditioned on what attention produced for that token. The backbone is frozen; only the diffuser was trained.

This GGUF needs the llama.cpp fork with dlora support: jadsongmatos/llama.cpp, branch teste3-nanbeige-dlora. Stock llama.cpp rejects the extra diffuser.* tensors.

Files

File	Size	Weights
Nanbeige4.2-3B-dlora-IQ4_NL.gguf	2.60 GB	backbone: 128 IQ4_NL, 45 F32, 27 Q5_K, 1 Q6_K; diffuser: 22 F32

SHA-256: f33e25892ccf308df659d7025990929ea070216a473cc4ab6cdd1615e983a0e2

The backbone weights are the NF4 weights the diffuser was trained against, dequantized to bf16 and quantized to IQ4_NL with llama.cpp's default IQ4_NL mix (llama-quantize --tensor-type diffuser=f32 ... IQ4_NL). The diffuser stays in F32.

Run

Build the fork and serve the model with the OpenAI-compatible API:

git clone -b teste3-nanbeige-dlora https://github.com/jadsongmatos/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # leave out -DGGML_CUDA=ON for a CPU-only build
cmake --build build -j --target llama-server
./build/bin/llama-server -m Nanbeige4.2-3B-dlora-IQ4_NL.gguf -ngl 99 -np 1 -c 36864

The model reasons before it answers (its chat template opens <think>), so give it room: MMLU-Pro answers average about 5,384 tokens. llama-server returns the reasoning in reasoning_content and the answer in content.

How it works

• The looped backbone runs its 22 layers twice, so the FFN runs 44 times per token. One diffuser serves all 44.
• At each FFN, the diffuser takes the attention output at that point (x_attn), normalizes it, and projects it to a 32-dimensional context. It starts from a projection of that context plus a fixed noise draw (σ = 1.0, seed 42, stored in the GGUF), runs 4 DDIM steps with a small MLP denoiser, and decodes the LoRA's A and B matrices for the gate, up, and down projections.
• Training: the backbone loaded in NF4 and frozen; a sample of epfml/FineWeb2-HQ in 20 languages, without English and with the same number of tokens per language; whole documents packed into 262,144-token sequences; about 80 M tokens (step 152). Each document drew 4 noise samples and trained on the best one.

Evaluation: MMLU-Pro with CoT

The official TIGER-Lab harness (evaluate_from_api.py): 5-shot chain of thought, greedy decoding, through llama-server's API. Each answer can use up to 32,768 tokens instead of the harness's 4,000; at 4,000, 44% of this model's answers were cut while still reasoning. The answer is extracted from content, after the reasoning. Official accuracy counts an unanswered question as a random guess, as the harness does; strict accuracy counts it as wrong.

bf16 weights, all 12,032 questions

Measure	Value
Official accuracy	77.98%
Strict accuracy	77.86% (standard error ±0.38 points)
No extracted answer	161 (1.3%)
Hit the 32,768-token limit	178 (1.5%)
Tokens per answer	5,384

Category	Questions	Official	Strict
math	1,351	92.30%	92.08%
biology	717	88.01%	87.87%
chemistry	1,132	87.63%	87.54%
physics	1,299	87.61%	87.45%
business	789	84.16%	84.03%
economics	844	82.82%	82.82%
computer science	410	81.22%	81.22%
psychology	798	77.69%	77.69%
health	818	71.15%	71.15%
engineering	969	69.66%	69.04%
philosophy	499	69.14%	69.14%
other	924	66.02%	66.02%
history	381	65.09%	65.09%
law	1,101	54.41%	54.31%

This IQ4_NL file

This file wasn't evaluated on MMLU-Pro; the numbers above are for the bf16 weights.

Limitations

• The file needs the fork above; stock llama.cpp can't load it.
• There's no same-protocol run of the plain Nanbeige4.2-3B yet, so this card doesn't measure how much the diffuser adds. The instruct model has no published MMLU-Pro number; 63.8 is for Nanbeige4.2-3B-Base, with the protocol not stated.
• The diffuser was trained against the NF4 backbone, and this file re-quantizes that backbone to IQ4_NL.
• The diffuser makes decoding slower: the graph has about 3.5 times as many nodes as the plain model.
• Inference uses one fixed noise draw; training drew a new one for every document.

License

Apache-2.0, the license of Nanbeige4.2-3B. This repository adds the trained diffuser and re-quantizes the backbone weights.