Qwen3-1.7B-DSpark

A DSpark draft model for speculative decoding with Qwen/Qwen3-1.7B as the verifier, trained with speculators. The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone — a lossless speedup. Mean acceptance length is 3.75 tokens committed per verification step, up to 5.01 on math_reasoning. That gives a throughput speedup over the verifier alone of 3.30× on math_reasoning and 2.67× on HumanEval, and 2.35× averaged across nine task types.

Training code: rasyosef/train-dspark-draft-models.

Trained on 100,000 samples.

Usage

vLLM

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve rasyosef/Qwen3-1.7B-DSpark --port 8000 --gpu-memory-utilization 0.8

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.

llama.cpp

llama-server -hf unsloth/Qwen3-1.7B-GGUF:Q8_0 \
  -hfd rasyosef/Qwen3-1.7B-DSpark-GGUF:Q8_0 \
  --spec-type draft-dspark --spec-draft-n-max 8 --spec-draft-n-min 0 \
  -fa on -ngl 99

The block size is read from the sidecar metadata (n-max is clamped to it). Speculative decoding is exact: the target verifies every proposed token, so greedy output equals the target alone; per-response timings report draft_n / draft_n_accepted.

Details

4 Qwen3 layers (hidden size 2048, intermediate size 6144, 16 attention heads over 8 KV heads, head dim 128). The first three layers use sliding-window attention with a 2048-token window and the last layer uses full attention. Block size 8, aux hidden-state layers 2/10/18/26.

Trained for 3 epochs at lr 3e-4 (cosine schedule with 4% warmup) on 100,000 Open PerfectBlend prompts regenerated by the verifier itself, split 96/4 into train and validation, with a {"ce": 0.1, "tv": 0.9} loss. Prompts prepared at 2048 tokens; training sequence length 8192, up to 1024 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use rather than staged to disk up front.

Evaluation

evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token — floor 1.0, ceiling 9.0 at block size 8.

subset	acceptance_length	pos_0	pos_1	pos_2	pos_3	pos_4	pos_5	pos_6	pos_7
math_reasoning	5.010	85.8%	72.5%	61.4%	51.4%	42.9%	35.6%	28.7%	22.8%
HumanEval	4.201	82.2%	65.7%	51.5%	39.7%	30.2%	22.5%	16.5%	11.7%
translation	3.468	76.3%	57.2%	41.1%	27.8%	18.8%	12.5%	7.9%	5.0%
rag	3.439	74.8%	54.8%	39.7%	28.1%	19.5%	13.0%	8.6%	5.4%
question	3.429	73.6%	52.8%	38.1%	27.4%	19.7%	14.2%	10.1%	7.0%
writing	3.426	73.8%	52.8%	37.8%	27.3%	19.7%	14.1%	10.0%	7.1%
qa	3.377	72.3%	51.9%	36.8%	27.0%	19.3%	13.6%	9.9%	7.0%
tool_call	3.247	72.8%	51.4%	35.7%	24.6%	16.8%	11.2%	7.5%	4.8%
summarization	2.641	66.5%	41.4%	24.9%	14.2%	8.3%	4.8%	2.5%	1.4%

Weighted across all subsets: 3.749 over 357,424 verification steps.

Acceptance is highest where the verifier's next token is most predictable — math and code. math_reasoning leads HumanEval by about 0.8 tokens and holds a margin at every position in the block; math_reasoning's pos_7 (22.8%) is nearly double HumanEval's (11.7%), and its pos_4 (42.9%) is above summarization's pos_1 (41.4%). translation, rag, question, and writing cluster between 3.43 and 3.47, with qa just behind at 3.38. Unlike many drafters, tool_call does not stand apart here: at 3.25 it sits at the bottom of the prose-like group rather than above it.

Throughput (tokens/s)

Mean output throughput (tokens/s) across the same nine subsets, measured on a single A100 at concurrency 1. Baseline is Qwen3-1.7B with no speculative decoding, measured on every subset; without a drafter, throughput barely depends on the task (218.7–223.3 tokens/s). Speedup is the mean throughput relative to that subset's baseline.

subset	baseline (no drafter)	DSpark (this model)	speedup
math_reasoning	220.1	725.8	3.30×
HumanEval	219.7	586.5	2.67×
translation	223.3	555.9	2.49×
rag	218.7	506.0	2.31×
question	221.3	476.0	2.15×
writing	220.9	483.8	2.19×
qa	222.5	477.3	2.15×
tool_call	219.9	471.7	2.15×
summarization	219.4	384.5	1.75×
average speedup	1.00×	—	2.35×

DSpark averages a 2.35× speedup (unweighted mean of per-subset speedups). The biggest gains are on math_reasoning at 3.30× (220.1 → 725.8 tokens/s) and HumanEval at 2.67×. translation is next at 2.49× and rag at 2.31×; writing, question, qa, and tool_call sit in a tight band between 2.15× and 2.19×. summarization is lowest at 1.75×.

Means and medians agree to within about 5% on every subset. The gap is widest on writing, question, and tool_call, where the mean runs 4–5% above the median, so a typical request on those subsets sees closer to 2.05×.

Limitations

Works only with Qwen3-1.7B and is not usable as a standalone model. Acceptance falls off steeply past the first few positions on prose-like traffic, and on summarization in particular a block size of 8 is mostly wasted — the gains concentrate in math and code. Tool-call traffic sees only prose-level acceptance with this drafter. Throughput was measured at concurrency 1 on one A100; real-world speedup depends on your traffic mix, hardware, and load, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.