BoldingBuilds/Qwen3.8-Flash-Next-Reasoning-Termination-Fix-GGUF

Qwen3.8 Flash Next Reasoning Termination Fix (UD-IQ4_XS GGUF)

Fewer empty answers from Qwen3.8 Flash Next within a 4,096-token output budget. This GGUF changes only the output-layer row for </think>. On a 200-problem MATH-500 subset, empty final responses fell from 29 to 0 and correct answers rose from 150 to 168. llama.cpp's --reasoning-budget flag scored similarly (172), while the weight edit needs no runtime budget control. Occasional stray </think> tags remain.

This is the same method as Qwen3.8-27B-Reasoning-Termination-Fix-GGUF, applied to a different architecture.

200 MATH-500 problems, 4,096 output tokens	correct	empty final response
this model	168	0
stock	150	29
stock + --reasoning-budget 2048	172	0

Paired over the same problems, this model solved 25 that stock missed and missed 7 that stock solved (exact McNemar p = 0.002). All 29 of stock's empty responses were the model still thinking when the 4,096 tokens ran out. Every p-value on this card is exploratory and unadjusted (see Statistics below).

Compared with the flag. The 2,048 flag scored 172 against 168 for this model (8 vs 4 on the problems where they differ, p = 0.39). The flag is slightly ahead, but the difference is not statistically significant. The point of this build is that it needs no flag, so it should behave the same in apps and APIs that do not expose one. That is expected, not tested: only llama.cpp was tested (see Compatibility). Larger output budgets were not tested for this model; on the 27B version the flag did better at 16,384 tokens.

September 29 update: Clarified that agent/tool use and speculative decoding were not tested for this release. I have not published a medium-effort comparison for Flash-Next. The earlier medium-effort comparison on 27B is a separate development experiment and does not establish a result for Flash-Next. The released weights and results are unchanged.

Files

file	size
Qwen3.8-Flash-Next-Reasoning-Termination-Fix-UD-IQ4_XS-00001-of-00003.gguf	10.9 MB
Qwen3.8-Flash-Next-Reasoning-Termination-Fix-UD-IQ4_XS-00002-of-00003.gguf	49.84 GB
Qwen3.8-Flash-Next-Reasoning-Termination-Fix-UD-IQ4_XS-00003-of-00003.gguf	43.84 GB

Point llama.cpp at the first file; it loads the other two. SHA-256 of every shard is in evidence/hashes.txt. Only shard 2 differs from the stock files.

Language model only. The vision encoder is not included and vision was not tested.

Usage

llama-server -m Qwen3.8-Flash-Next-Reasoning-Termination-Fix-UD-IQ4_XS-00001-of-00003.gguf -ngl 99 -fa on -c 32768 --jinja

This command sets no sampler options, so llama.cpp uses the defaults stored in the file (temperature 1.0, top-k 20, top-p 0.95) plus its own min-p default of 0.05. The evals sent no sampler settings either, so they ran on these defaults. (The eval build's source applies them, and a mainline server loading these exact files reported the same values.) -c is the context window, not the output budget. The budget is the max_tokens your client sends; every result here uses 4,096. The model is about 94 GB, so most setups will need to keep part of it in system RAM (for example --n-cpu-moe).

Compatibility

• llama.cpp, tested. Results were produced with llama.cpp built from commit 035e227 on the branch of pull request #27742 (which adds this architecture; the pull request was merged later with more commits) plus local instrumentation commits that only activate through an environment variable the evals did not set. These exact files were also loaded on mainline llama.cpp (ggml-org master, commit 5262471, 2026-09-28) with the experts in system RAM (--cpu-moe) on one RTX 3090: it answered all 30 problems of a 30-problem MATH check (26 correct, no stray tags), and the server reported the file's sampler defaults.
• LM Studio, Ollama and other apps: untested. They need a llama.cpp recent enough for this architecture. The fix needs no settings, so the benefit should carry over, but I have not checked.

What changed from stock

edited	1 row of output.weight in shard 2 (the </think> token, id 248069, Q6_K): 2,084 of its 2,100 bytes changed, 0 bytes changed anywhere else (full byte comparison of the shard)
unchanged	shards 1 and 3 are byte-identical to unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS (revision 8bdc6666), and so is every other byte of shard 2

How the fix works. Only one output-layer row was fitted; no other weights were trained. The change is a fitted delta to the </think> row, intended to improve when the model ends its reasoning. I fit it on the model's own final hidden states so the </think> score rises once the model is ready to answer or is going in circles, and stays low mid-thought and inside the answer. It was fit in two rounds: the second added places where the first version closed its reasoning too late, wrote a stray </think> into its answer, or ended its turn inside its thinking with no reply.

Fitting data vs test data. The fix was fit on 469 prompts that are not in any test here: 150 MATH training problems, 80 MBPP training problems, 40 Alpaca instructions and 199 harmful requests (AdvBench, JailbreakBench), the same set as the 27B version with its escaping defect fixed. Any prompt that matched or closely resembled (word overlap of 50% or more) a prompt from any test here was removed first. Only the released round was run on the test sets, so no candidate was chosen by looking at test results. The method itself was developed on the 27B model using these same test sets.

One measurement quirk to disclose. Before fitting, I check that a standalone computation of the </think> score from saved hidden states reproduces the server. For this model it matched to a median error of 0.0001 nats, but 8 of 36 check points were off by 0.1 to 1.1 nats, apparently because this architecture's hidden states depend slightly on how requests are batched. The fit may therefore be a little noisier than on the 27B. The live test results above are unaffected; they are measured directly.

Other results

HumanEval (164 problems), pass@1, 4,096 output tokens:

build	passed	empty final response
this model	155	0
stock	151	11

This model vs stock: 6 vs 2 on the problems where they differ (p = 0.29). That leans toward this model but is not statistically significant.

Refusals: aggregate rates stayed close to stock. Same judge and prompt for every row:

	stock	this model
StrongREJECT (first 150), judged a refusal	143 (95.3%)	143 (95.3%)
SimpleSafetyTests (100), judged a refusal	85	84
XSTest safe prompts (first 100), full compliance	88	92
XSTest safe prompts, partial refusal	7	7
XSTest safe prompts, full refusal	1	1
XSTest safe prompts, empty (ran out of tokens)	4	0
empty reply that ended the turn, 250 harmful prompts	1	2
empty reply that ran out of tokens, 250 harmful prompts	1	0

These are aggregate rates. Individual prompts moved in both directions; per-prompt labels are in evidence/per_item/refusals_judged.csv.

Stray </think> tags. This model wrote a stray </think> into its final answer in 1 of 200 MATH answers and 1 of 164 HumanEval answers, and in none of the refusal answers. No stray tags were observed in the tested stock runs.

How this was tested

• Server: llama-server with -ngl 99 --flash-attn on --jinja --override-tensor per_layer_token_embd.weight=CPU, 16 requests in parallel (MATH: 5,120 tokens per slot; other sets: 6,144), one rented H200 per server. Each request sends the messages, max_tokens (4,096) and seed: 0, nothing else. MATH and refusal prompts are a single user message (MATH adds "Put your final answer in \boxed{}."); HumanEval adds a short system prompt asking for code only.
• Thinking on, at the chat template's default reasoning effort (xhigh).
• Sampler: the file's defaults as above. Sampling at temperature 1.0 with parallel slots is not bit-reproducible run to run.
• Empty final response means the reply after the reasoning was empty. It has two causes, counted separately in evidence/summary_counts.csv: the output budget ran out during reasoning (finish_reason: length), or the model ended its turn with no answer (finish_reason: stop).
• MATH-500: the first 200 problems after a seeded shuffle (seed 7), scored by exact match of the last \boxed{} answer after normalisation. The scorer is strict, so compare rows to each other, not to published numbers.
• HumanEval: all 164 problems, one sample each; the code block is run against each problem's tests.
• Refusals: StrongREJECT (first 150), SimpleSafetyTests (all 100), XSTest safe prompts (first 100), graded by an LLM judge that is not the model under test: a Q8_0 file from OBLITERATUS/Qwen3.8-27B-OBLITERATED, thinking off, temperature 0, JSON-schema output, StrongREJECT rubric for harmful sets and the XSTest three-way labels for safe prompts. Same judge, prompt and settings as the 27B card.
• Statistics: exact two-sided McNemar test on paired per-problem results. The p-values are exploratory and not adjusted for multiple comparisons. No candidate was chosen by looking at these test results (only the released round was run on them), but the method itself was developed on the 27B model using these same test sets.

Evidence

evidence/ holds per-problem results for every row (IDs, scores and flags only, no model text); summary_counts.csv with every count on the card; hashes.txt with SHA-256 of every shard, the judge, datasets and scripts; the scripts and settings as run; judge labels by prompt index; the fitting-prompt manifest (source and SHA-256); the bake and verification logs; and the rebuild log showing the release shard reproduces the tested one byte for byte.

It is enough to check every number and to rerun the evals (the refusal sets after rebuilding them from the public datasets), but not everything used is published. Not included: the text of the fitting and refusal test prompts (many are harmful requests; all come from public datasets, listed with the selection rules), the model's own generations used for fitting, and six local instrumentation commits in the eval build (inactive in the evals). The evidence README lists what the setup script expects, where each public piece comes from, and the one source patch needed.

Limitations

• All results are at a 4,096-token output budget. Larger budgets were not tested on this model.
• The fix makes the model stop thinking sooner. Overall scores went up, but not on every problem: it missed 7 MATH problems and 2 HumanEval problems that stock solved, and its safety labels differ from stock on individual prompts.
• Occasional stray </think> tags in answers: 1 of 200 MATH answers and 1 of 164 HumanEval answers.
• Tested in English, text only, single turn. Agent, tool use and multi-turn use were not tested.
• Speculative decoding with a draft model was not tested for the released file; its effect on draft acceptance and speedup is unknown.
• One quant (UD-IQ4_XS). One seed per configuration.

Related work

See the 27B card for the full list: run-time budget control, hidden-state probes for early exit (Zhang et al. 2025, LYNX), stopping rules on the </think> score (ThinkBrake), weight edits that change reasoning length (ThinkEdit, Steer2Edit), and the analysis of the </think> score in Sheng et al. 2025.

License and credits

Base model: Qwen/Qwen3.8-Flash-Next, under the Qwen Community License 1.0, included here as LICENSE. Quantization: Unsloth's UD-IQ4_XS (unsloth/Qwen3.8-Flash-Next-GGUF). Test sets: MATH (MATH-500), HumanEval, StrongREJECT, SimpleSafetyTests, XSTest. Fitting prompts: MATH (train), MBPP (train), Alpaca, AdvBench, JailbreakBench.