Llama-3.2-3B-Instruct-DSpark

A DSpark draft model for speculative decoding with unsloth/Llama-3.2-3B-Instruct as the verifier, trained with speculators. The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone — a lossless speedup. Mean acceptance length is 3.41 tokens committed per verification step, up to 5.53 on math_reasoning. That gives a throughput speedup over the verifier alone of 3.62× on HumanEval and 3.48× on math_reasoning.

Training code: rasyosef/train-dspark-draft-models.

Trained on 100,000 samples.

Usage

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve rasyosef/Llama-3.2-3B-Instruct-DSpark --port 8000 --gpu-memory-utilization 0.8

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.

Details

5 Qwen3 layers (hidden size 3072, intermediate size 8192, 24 attention heads over 8 KV heads, sliding-window attention with a 2048-token window), ~0.7B params, bfloat16. Block size 8, draft vocabulary reduced to 50,000, aux hidden-state layers 2/8/14/20/26, confidence head with Markov (rank 256).

Trained for 3 epochs at lr 3e-4 (AdamW, cosine schedule with 4% warmup) on 100,000 Open PerfectBlend prompts regenerated by the verifier itself, split 96/4 into train and validation, with a {"ce": 0.1, "tv": 0.9} loss. Prompts prepared at 1024 tokens; training sequence length 8192, up to 1024 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use rather than staged to disk up front. speculators 0.8.0.dev207, vLLM 0.28.0, transformers 5.15.1, torch 2.13.0.

Evaluation

evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token — floor 1.0, ceiling 9.0 at block size 8.

subset	acceptance_length	pos_0	pos_1	pos_2	pos_3	pos_4	pos_5	pos_6	pos_7
math_reasoning	5.529	89.6%	78.4%	67.9%	58.8%	50.5%	43.3%	36.0%	28.2%
HumanEval	5.058	85.6%	72.6%	61.6%	52.3%	43.7%	36.3%	29.9%	23.8%
tool_call	3.492	73.6%	54.2%	39.5%	28.6%	20.8%	14.8%	10.5%	7.2%
question	2.952	67.7%	43.8%	28.8%	19.7%	13.8%	10.0%	6.8%	4.6%
rag	2.917	67.6%	45.8%	30.8%	20.4%	12.6%	7.6%	4.4%	2.4%
writing	2.908	67.5%	43.3%	28.0%	18.7%	13.0%	9.3%	6.4%	4.5%
translation	2.653	67.0%	42.7%	25.2%	14.2%	7.9%	4.4%	2.6%	1.3%
summarization	2.597	65.6%	40.5%	23.9%	13.9%	8.0%	4.3%	2.2%	1.2%
qa	2.241	55.0%	31.6%	17.9%	9.4%	5.0%	2.7%	1.6%	0.8%

Weighted across all subsets: 3.412 over 69,061 verification steps.

Acceptance is still highest where the verifier's next token is most predictable — math, code, structured tool calls. math_reasoning leads HumanEval by about half a token and holds a margin at every position in the block; math_reasoning's pos_5 (43.3%) is above summarization's pos_1 (40.5%), and both subsets still commit better than one token in four at pos_7. tool_call sits on its own at 3.49, between the two groups.

Five of the six prose-like subsets start in a tight band at pos_0 (65.6–67.7%) and separate only deeper into the block; qa is the exception, starting well below the rest at 55.0% and staying last throughout. question and writing decay gently and are still above 4% at pos_7. rag has the strongest pos_1 of the group (45.8%) but falls away after pos_3, and summarization, translation, and qa are all under 1.5% at pos_7 — so the tail of the block is worth something on open-ended generation but essentially nothing on summarization, translation, and qa.

Throughput (tokens/s)

Output throughput (tokens/s) on a single A100 at max concurrency 1, across the same nine subsets. Baseline is Llama-3.2-3B-Instruct with no speculative decoding. Speedup is the mean throughput relative to baseline.

subset	baseline (no drafter)	DSpark (this model)	speedup
math_reasoning	150.8	524.9	3.48×
HumanEval	150.6	545.2	3.62×
tool_call	147.8	337.7	2.28×
question	149.9	335.9	2.24×
rag	142.6	272.6	1.91×
writing	150.8	273.5	1.81×
translation	150.0	289.3	1.93×
summarization	147.7	279.5	1.89×
qa	150.3	259.1	1.72×
average speedup	1.00×	—	2.32×

DSpark averages a 2.32× speedup (unweighted mean of per-subset speedups). The biggest gains are on HumanEval at 3.62× (150.6 → 545.2 tokens/s) and math_reasoning at 3.48×. HumanEval edges out math_reasoning here even though math_reasoning accepts about half a token more per step. math_reasoning's median throughput (550.5 tokens/s) is above HumanEval's, so a few slower requests pull its mean down. tool_call and question are next at around 2.25×. The remaining prose-like subsets sit between 1.7× and 1.95×, with qa lowest at 1.72×. question's mean (335.9 tokens/s) is well above its median (272.5), so a handful of highly predictable responses lift its average. On a typical question request the speedup is closer to 1.8×.

Limitations

Works only with Llama-3.2-3B-Instruct and is not usable as a standalone model. Acceptance falls off steeply past the first few positions on prose-like traffic (summarization, translation, qa), so a block size of 8 is mostly wasted there — the gains concentrate in math, code, and tool calls. Real-world speedup depends on your traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.

License

Llama 3.2 Community License, inherited from the verifier. The speculators training code is Apache-2.0.