Qwimi-3.6-27B-Coder-MTP-BF16

A coding-focused SFT of Qwen 3.6 27B, merged to full-precision BF16. Trained on a mixed corpus of coding, tool-calling, and agentic (SWE-agent) data in a single supervised fine-tuning run, with the coding domain deliberately kept dominant (~82.5% of training tokens) to avoid the smaller tool/agentic buckets degrading coding capability.

Status: public repository. This is the merged, ready-to-use model. For a smaller adapter-only download see Qwimi-3.6-27B-LoRA; for quantized GGUF files (llama.cpp / Ollama / LM Studio) see Qwimi-3.6-27B-GGUF.

---

Contents

1. Model summary
2. Intended use
3. Training configuration
4. Dataset composition
5. Training results
6. Chat template & tool-calling format
7. How to use
8. Verification & reproducibility
9. Known limitations
10. Benchmarks

---

1. Model summary

Base model	unsloth/Qwen3.6-27B (finetune of Qwen/Qwen3.6-27B)
Architecture	Dense, 27.8B params. Hybrid: standard attention layers interleaved with GatedDeltaNet linear-attention layers. model_type: qwen3_5, class Qwen3_5ForConditionalGeneration
Modality	Multimodal (image-text-to-text) checkpoint. This SFT is text-only — the vision tower was frozen and untouched (see §9)
Native context	262,144 tokens (base). This SFT trained and validates up to 16,384 tokens
Fine-tuning method	4-bit QLoRA (r=64), merged to BF16
Trainable parameters	466,911,232 / 27,823,639,792 (1.68%) — language layers only (attention + MLP), vision layers frozen
License	Apache 2.0 (inherited from base)

2. Intended use

Coding assistance (generation, debugging, refactoring, explanation) with visible chain-of-thought reasoning, plus native function/tool calling and basic agentic (SWE-agent style: multi-turn tool use over a repository) behavior.

Not validated for: image/vision input (untouched by this SFT — see §9), non-English use, safety-critical or high-stakes deployment, or any task outside code/tool/agentic reasoning.

3. Training configuration

Setting	Value
Precision (training)	4-bit QLoRA (NF4)
Max sequence length	16,384 tokens
LoRA rank / alpha	64 / 64
LoRA dropout	0.0
rsLoRA	off
LoRA scope	language layers only (attention + MLP); vision tower frozen
Optimizer	paged_adamw_8bit
Learning rate	2e-5, cosine schedule, warmup ratio 0.03
Weight decay	0.01
Max grad norm	1.0
Per-device batch size	2
Gradient accumulation	8
Effective batch size	16 (2 × 8 × 1 GPU)
Epochs	1
Optimizer steps	1,398
Training tokens	72,529,798
Masking	assistant-turns-only (train_on_responses_only); instruction marker <|im_start|>user\n, response marker <|im_start|>assistant\n
Gradient checkpointing	Unsloth offloaded mode, verified active (measured activation delta 3.1→5.5→9.2 GB across 2k→8k→16.4k token samples)
Hardware	1× A100-SXM4-80GB (Google Colab Pro)
Software	Unsloth 2026.6.9, Transformers 5.5.0, Torch 2.10.0+cu128, CUDA 12.8, Triton 3.6.0, flash-linear-attention + causal-conv1d (required for the GatedDeltaNet fast path)
Checkpoint cadence	every 150 steps, rolling window of 4, persisted to Google Drive

4. Dataset composition

Single mixed SFT run — coding, tool-calling, and agentic examples trained together (not sequential stages), so the model learns all three without later stages overwriting earlier ones. Coding was deliberately kept dominant by token share, not just row count.

Coding (5 sources, deduplicated, content- or domain-filtered for code where the source had no reliable label; one low-quality synthetic source capped hard):

Source	Raw rows contributed	Filter
trjxter/Kimi-K2.7-CodingTraces-9000x	9,014	none — used in full
trjxter/Kimi-K2.6-Technical-Reasoning-AddOn-3300x	1,645	domain ∈ {coding, debugging}
trjxter/Kimi-K2.6-Reasoning-3300x-WandB	798	content-filtered for code
Jackrong/Kimi-K2.5-Reasoning-1M-Cleaned (General-Distillation config)	6,500	content-filtered for code, capped at target
Jackrong/Claude-opus-4.7-TraceInversion-5000x	577	content-filtered; hard cap 600 — synthetically reverse-engineered reasoning traces, lowest-quality source, deliberately minimized

Tool-calling: minpeter/xlam-function-calling-60k-parsed — 6,000 rows (APIGen-generated, verifiable function calls, native tools schema). No chain-of-thought; direct tool calls.

Agentic (SWE-agent trajectories; the dominant length constraint in this run — most raw trajectories exceed 16,384 tokens and were dropped whole rather than truncated):

Source	Contribution	Note
SWE-Factory/DeepSWE-Agent-Kimi-K2-Trajectories-2.8K	~200 rows survive the 16k length filter	p50 trajectory length ≈ 23,000 tokens — most don't fit
nebius/SWE-agent-trajectories (top-up)	788 unique successful issues	target == True only; 1 trajectory per GitHub issue (deduplicated at the issue level so the later train/eval split can't leak the same issue into both)

Final composition after dedup, 90/10 per-domain split, and whole-fit-or-drop length filtering at 16,384 tokens:

Domain	Train rows	Eval rows	Train token share	Dropped for length
Coding	16,083	1,778	82.5%	598 (3.6% — accepted as pathological-outlier policy)
Tool	5,360	595	4.8%	0
Agentic	917	102	12.7%	2,321 (multi-turn trajectories that don't fit whole at 16k)
Total	22,360†	2,475	72.5M tokens	—

† One row was dropped by the trainer itself (all labels masked after truncation-safety filtering) — 22,359 rows actually trained on.

Reasoning-token fraction (coding + agentic, which carry <think> traces) was 95.2% of all training tokens, comfortably above the 75% floor generally recommended to preserve reasoning ability when mixing in non-reasoning (tool-calling) data.

5. Training results

Eval loss reported per domain via three held-out splits, evaluated every 150 steps (subsampled during training for speed: 400/1,778 coding rows, 200/595 tool rows, all 102/102 agentic rows — full-split numbers are pending a re-run and will be added if produced).

Step	Train loss	Eval — Coding	Eval — Tool	Eval — Agentic
150	—	0.5070	0.0246	0.3574
300	—	0.4911	0.0226	0.3257
450	—	0.4853	0.0217	0.3118
600	—	0.4823	0.0214	0.3056
750	0.5004	0.4794	0.0211	0.2981
900	0.4769	0.4781	0.0209	0.2946
1050	0.4931	0.4767	0.0208	0.2915
1200	—	0.4762	0.0208	0.2899
1350	—	0.4761	0.0208	0.2893
1398 (final)	PENDING	0.4761	0.0208	0.2892

All three eval splits decreased monotonically across the entire run with no split turning up — no overfitting signal at 1 epoch. Coding, the dominant and highest-value domain, continued improving through the final checkpoint rather than being degraded by the smaller tool/agentic data — the outcome the mixed-run design targeted. Note that eval-loss magnitudes are not directly comparable across domains (different sequence structure and supervised-token counts per split); track each split against itself over time.

Single-batch training loss (0.500, 0.477, 0.493 at steps 750/900/1050) is noisy at effective batch 16 and is shown only as a secondary signal — eval loss is the trend to read.

6. Chat template & tool-calling format

Standard ChatML markers, with <think> reasoning canonically formatted by the tokenizer's chat template (verified — the template re-renders think content even if you pass it inline):

<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant
<think>
{reasoning}
</think>

{answer}<|im_end|>

Tool calls use Qwen 3.6's native XML-style format (confirmed empirically against the installed chat template — this is not the JSON-object tool_call format used by earlier Qwen versions):

<|im_start|>assistant
<think>

</think>

<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
</function>
</tool_call><|im_end|>

An empty <think></think> block is the canonical "direct response" form — tool calls in this model are trained to skip reasoning and act directly. Parallel tool calls render as sibling <tool_call> blocks. Pass your tool definitions via the standard tools= argument to apply_chat_template; the template injects a # Tools system block describing the exact call format automatically.

7. How to use

Recommended (tested path) — Unsloth:

from unsloth import FastVisionModel

model, processor = FastVisionModel.from_pretrained(
    "trjxter/Qwimi-3.6-27B-BF16",
    load_in_4bit=False,        # this repo is already full-precision BF16
    max_seq_length=16384,
)
tokenizer = processor.tokenizer
FastVisionModel.for_inference(model)

messages = [{"role": "user", "content": "Write a Python function to reverse a linked list."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024, temperature=0.7, top_p=0.95)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Alternative — plain transformers (requires transformers>=5 for qwen3_5 support; confirm the exact model class name against your installed version's docs before relying on AutoModelForImageTextToText resolving correctly):

from transformers import AutoModelForImageTextToText, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("trjxter/Qwimi-3.6-27B-BF16")
model = AutoModelForImageTextToText.from_pretrained("trjxter/Qwimi-3.6-27B-BF16", dtype="bfloat16", device_map="auto")

This model was fine-tuned and validated on text-only input; no image inputs were used during training (see §9).

8. Verification & reproducibility

• Adapter integrity: final adapter adapter_model.safetensors = 1,867.8 MB, matching 466,911,232 trainable params × 4 bytes (fp32 save) exactly — confirms the saved file contains the complete, correctly-scoped trained adapter.
• Merge integrity: max |lora_B| (the check that the adapter actually trained before merging — an all-zero adapter would merge silently into an unchanged copy of the base) — PENDING, not yet recorded from the merge run. Treat the merge as provisionally verified pending this number.
• Tokenizer/config repair: the merged repo's tokenizer_config.json originally recorded a runtime-internal tokenizer_class value (TokenizersBackend, from transformers v5's tokenizer backend) that AutoTokenizer.from_pretrained cannot resolve. This was corrected in the repo to match the base model's tokenizer class — confirm your load succeeds before reporting issues; this repair should make plain AutoTokenizer loading work correctly.
• Checkpoint recovery: this run survived two Colab VM interruptions via checkpoints persisted to Drive every 150 steps; the final adapter reflects the complete 1,398-step run, not a recovered partial checkpoint.

9. Known limitations

• Vision tower untouched. The base checkpoint is multimodal; this SFT explicitly froze the vision layers (finetune_vision_layers=False) and trained on text-only data. No claims are made about image understanding — it is whatever the base model shipped with, unvalidated here.
• Agentic data is thin and short. After length-filtering, only 917 agentic training rows survived (12.7% of tokens), and DeepSWE trajectories longer than ~16k tokens were entirely excluded rather than truncated — this model has seen comparatively little of the longest, most complex agentic trajectories.
• One low-quality source, deliberately capped. Claude-opus-4.7-TraceInversion-5000x contains reasoning traces reverse-engineered by a small model from compressed summaries, not genuine teacher traces. Capped at 577 rows (3.5% of the coding split) specifically to limit this risk.
• 3.6% of coding examples were dropped for length (598 rows exceeding 16,384 tokens) — accepted as pathological outliers rather than truncated, since truncating a reasoning trace teaches the model to stop mid-thought.
• Context validated to 16,384 tokens despite the base model's much longer native context; behavior beyond that length is unvalidated by this training run.

10. Benchmarks

Qwimi was compared against the base Qwen 3.6 27B model using equivalent Q6 GGUF builds with MTP enabled and the same evaluation harness.

Two benchmark suites were run:

1. A fully auto-scored 300-task custom benchmark covering coding, tool-calling, and agentic workflows.
2. A broader 200-task mixed benchmark containing standard coding tasks and personal Qwimi-oriented evaluations.

Important: the latency figures below measure average end-to-end wall time per task, not raw tokens per second. SFT can reduce task completion time by producing shorter, more direct, better-formatted responses and reaching the stop condition sooner. The results do not prove that the underlying Q6 inference kernels became faster.

---

10.1 Custom 300-task benchmark

This benchmark contains 300 custom-authored, fully auto-scored tasks:

Category	Tasks
Coding	150
Tool-calling	75
Agentic workflows	75
Total	300

Both models answered the same 300 tasks, producing 600 total answer rows.

Overall results

Model	Tasks	Pass rate	Average wall time	Total wall time
Base Qwen 3.6 27B Q6 MTP	300	81.33%	53.56 s	267.79 min
Qwimi 3.6 27B Coder Q6 MTP	300	82.67%	27.09 s	135.45 min

Overall change	Result
Absolute pass-rate gain	+1.33 percentage points
Relative pass-rate gain	+1.64%
Average wall-time reduction	49.4%
Effective task-level speedup	~1.98×
Total evaluation time saved	132.34 minutes

The overall result is a small accuracy improvement combined with a very large reduction in task-completion time.

Category results

Category	Base pass rate	Qwimi pass rate	Accuracy delta	Base avg time	Qwimi avg time	Wall-time reduction
Custom coding	81.33%	85.33%	+4.00 pts	59.33 s	21.16 s	64.3%
Custom tool-calling	88.00%	89.33%	+1.33 pts	31.77 s	27.13 s	14.6%
Custom agentic	74.67%	70.67%	-4.00 pts	63.80 s	38.92 s	39.0%

The strongest result was in custom coding, where Qwimi improved pass rate by four percentage points while completing tasks with 64.3% lower average wall time.

Tool-calling showed a smaller but positive accuracy gain. Agentic accuracy regressed by four points, although Qwimi still completed the agentic tasks substantially faster.

Pairwise outcomes

Pairwise result	Tasks	Share
Qwimi passed and base failed	23	7.67%
Base passed and Qwimi failed	19	6.33%
Both passed	225	75.00%
Both failed	33	11.00%
Total	300	100%

Among the 42 decisive tasks where only one model passed:

Decisive result	Count	Share
Qwimi wins	23	54.8%
Base wins	19	45.2%

Category-level pairwise outcomes

Category	Qwimi-only wins	Base-only wins	Both pass	Both fail
Coding	11	5	117	17
Tool-calling	4	3	63	5
Agentic	8	11	45	11
Total	23	19	225	33

300-task verdict: Qwimi achieved a slight overall quality win, driven by stronger coding and tool-calling performance, while also completing the suite in roughly half the time. The main remaining weakness was agentic-task accuracy.

---

10.2 Mixed 200-task benchmark

A separate 200-task evaluation compared the same Q6 MTP builds across standard coding tasks and personal Qwimi-oriented tests.

The run produced 400 raw answer rows, with 200 answers from each model. Some categories were omitted from the scored summary because their scorer/export integration was incomplete; they were not treated as zero-score failures.

Overall auto-scored results

Model	Auto-scored rows	Passed	Pass rate	Average wall time
Base Qwen 3.6 27B Q6 MTP	151	46	30.46%	92.86 s
Qwimi 3.6 27B Coder Q6 MTP	166	51	30.72%	71.30 s

Overall change	Result
Absolute pass-rate gain	+0.26 percentage points
Relative pass-rate gain	+0.85%
Average wall-time reduction	23.2%

Pairwise auto-scored results

On the 146 tasks with a valid paired auto-score:

Pairwise result	Tasks	Share
Qwimi better	7	4.79%
Base better	2	1.37%
Same result	137	93.84%
Total paired tasks	146	100%

Among the nine decisive paired tasks, Qwimi won:

[ \frac{7}{9} = 77.8% ]

This corresponds to a net pairwise edge of +3.42 percentage points across all paired auto-scored tasks.

Category highlights

Category	Base pass rate	Qwimi pass rate	Delta
HumanEval	80.00%	94.29%	+14.29 pts
Personal XML tool tasks	50.00%	100.00%	+50.00 pts
MBPP	5.77%	1.92%	-3.85 pts
Tool-calling checks	100%	100%	Tie
Architecture checks	100%	100%	Tie
Debugging checks	100%	100%	Tie

200-task verdict: the result was directionally positive for Qwimi, particularly on HumanEval, XML-style tool behavior, and wall time. However, the overall pass-rate gain was small, most paired results were ties, and MBPP performance regressed.

---

10.3 Combined interpretation

Across both benchmark suites, Qwimi demonstrated:

• Higher overall pass rate in both runs.
• Stronger performance on custom coding, tool-calling, HumanEval, and personal XML-tool tasks.
• A positive pairwise record:
  • 23–19 on the 300-task custom benchmark.
  • 7–2 on decisive paired tasks in the 200-task benchmark.
• Substantially lower task-completion time:
  • 49.4% lower on the custom 300-task suite.
  • 23.2% lower on the mixed 200-task suite.
• A remaining weakness in some areas:
  • -4.00 points on custom agentic accuracy.
  • -3.85 points on MBPP.

The fairest overall conclusion is:

Qwimi is a modest but measurable coding/tool-use improvement over the base model, with unusually large task-level latency reductions. Its largest quality gains appear in coding and structured tool usage, while agentic and MBPP performance require further work.

---

10.4 Interpreting the speed results

Both evaluated models used Q6 quantization, so the speed difference should not be interpreted as Qwimi having fundamentally faster matrix-multiplication kernels.

Autoregressive task time can be approximated as:

[ T_{\text{task}} \approx T_{\text{prefill}} + N_{\text{generated tokens}} \times T_{\text{decode per token}} ]

Because both models use the same architecture family and quantization level, their raw decode cost per token should be broadly similar. SFT can still reduce total wall time by changing generation behavior:

• reaching the correct solution with fewer tokens;
• producing less unnecessary explanation or repetition;
• following the expected output format sooner;
• avoiding failed attempts, malformed tool calls, or self-correction loops;
• emitting the stop token earlier.

Therefore, the supported claim is:

Qwimi completed the benchmark tasks faster.

A separate controlled measurement of generated-token counts and tokens per second would be required to claim that Qwimi is intrinsically faster on a per-token basis.

---

10.5 Benchmark limitations

• Only the Q6 GGUF builds were compared; these results should not be assumed to apply identically to every quant listed in §2.
• Wall time depends on hardware, llama.cpp build, prompt length, output length, sampling settings, MTP support, and stop conditions.
• The 300-task suite is custom-authored and is not a replacement for established public benchmarks.
• Some categories in the 200-task run were excluded because their official scorer/export integration was incomplete; they were not counted as zero.
• Agentic tasks used keyword/structural auto-scoring and may not capture every aspect of repository-level software-engineering quality.
• More seeds, repeated runs, and token-level throughput measurements would improve confidence in the latency conclusions.