webmp3/Sakura-Jais-3-7B-GGUF

Sakura — Jais-3-7B (GGUF)

GGUF quantizations of Jais-3-7B (Arabic and English, dense, K2 Horizon architecture, 36 layers, 4096 hidden size, context up to 524,288 tokens by its config). The official GGUF repository IFM/Jais-3-7B-GGUF already exists; this repository differs in that every file up to 5 bits was made with an importance matrix (calibration text below) and in what we measured. The version with edited weights is Sakura-Jais-3-7B-Uncensored-GGUF.

What was done

• We converted the original safetensors checkpoint (inception42/Jais-3-7B, revision 94d69f49) to a BF16 GGUF with convert_hf_to_gguf.py of a llama.cpp checkout of 2026-10-07 (commit 9c2e0e491), then quantized with llama-quantize of llama.cpp b11594.
• The importance matrix was computed on the BF16 file with llama-imatrix (96 chunks of 512 tokens of our calib_mix_v1 text mix, which contains English and German but no Arabic). Q8_0 and Q6_K were made without it.
• No weights were edited.

Files

File	Size	What it is
Sakura-Jais-3-7B-Q8_0-8.92GiB.gguf	8.92 GiB	Q8_0 (no importance matrix)
Sakura-Jais-3-7B-Q6_K-6.89GiB.gguf	6.89 GiB	Q6_K (no importance matrix)
Sakura-Jais-3-7B-Q5_K_M-6.02GiB.gguf	6.02 GiB	Q5_K_M with importance matrix
Sakura-Jais-3-7B-Q4_K_M-5.21GiB.gguf	5.21 GiB	Q4_K_M with importance matrix
Sakura-Jais-3-7B-IQ4_XS-4.76GiB.gguf	4.76 GiB	IQ4_XS with importance matrix

Run

llama-server -m Sakura-Jais-3-7B-Q5_K_M-6.02GiB.gguf --jinja -ngl 99 -c 8192

Jais-3 is a model that always thinks: its chat template opens <ifm|think>, <ifm|think_fast> or <ifm|think_faster> for reasoning_effort high (default), medium or low. Pass the effort per request as chat_template_kwargs: {"reasoning_effort": "low"}. The architecture (K2 Horizon) needs a llama.cpp build with the support merged on 2026-10-06 (PR 29535); we used b11594, older builds do not know it. enable_thinking is not used by this template.

What we measured

Distance to Q8_0 (KL divergence of the next-token distribution of each file against our Q8_0 of the same model, llama-perplexity b11594, 20 chunks of 2048 tokens of an English (en) and a web (web) evaluation text; mean ± standard error; the reference is Q8_0, not the BF16 file). The self-check (Q8_0 against itself) gave 99.995 % same top token and KLD 0.

File	Size	KLD en	KLD web	same top token en / web
Q6_K	6.89 GiB	0.0157 ± 0.0006	0.0072 ± 0.0002	94.9 / 95.3 %
Q5_K_M	6.02 GiB	0.0138 ± 0.0008	0.0073 ± 0.0001	95.1 / 95.1 %
Q4_K_M	5.21 GiB	0.0320 ± 0.0012	0.0168 ± 0.0002	92.6 / 92.6 %
IQ4_XS	4.76 GiB	0.0601 ± 0.0022	0.0234 ± 0.0003	90.6 / 91.4 %

The perplexity of the Q8_0 file on the same texts is 8.51 ± 0.18 (en) and 12.69 ± 0.24 (web). Q5_K_M (with importance matrix) is not further from Q8_0 than Q6_K (without); the 4-bit files are roughly two to three times (Q4_K_M) and three to four times (IQ4_XS) as far from Q8_0 as Q5_K_M, so we would take Q5_K_M or larger when memory allows.

Smoke test (Q8_0 and the uncensored Q8_0 of the sister repository, llama-server b11594, reasoning_effort low, temperature 0): three short questions, one English and two Arabic; all answers were coherent and in the language of the question.

Not measured: anything in Arabic (quality of the quantizations, perplexity, benchmarks), any capability benchmark, speed, long contexts, tool calls, the high and medium reasoning efforts.

License

Apache-2.0 as the original model (LICENSE). See NOTICE.

Credits

• Model: inception42/Jais-3-7B and the official GGUF IFM/Jais-3-7B-GGUF.
• Quantization and measurement: ggml-org/llama.cpp (MIT).

---

中文版见 README_zh.md。