patrickbdevaney/deepseek-v4-flash-0731-REAP-calibrated-draft-head

DSpark MTP draft head s3 for DeepSeek-V4-Flash-0731-REAP

This is a fine-tuned set of DSpark MTP draft-head tensors for 0xSero/DeepSeek-V4-Flash-0731-REAP. It is not a standalone language model. It replaces the checkpoint's own draft head for speculative decoding. It is the session-3 head s3 (packaged as dspark-mtp-draft-head-0731reap-v1), measured at draft width 6 on a Jetson AGX Thor at 25.53 tok/s, against 13.8 tok/s plain autoregressive decode in the same run.

It has since been superseded. For new use, take dspark-mtp-draft-head-s3recap-p25-b0.1 or the ready-to-run checkpoint DeepSeek-V4-Flash-0731-REAP-DSpark.

Overview

Target model	0xSero/DeepSeek-V4-Flash-0731-REAP, a REAP prune (160 of 256 routed experts) of deepseek-ai/DeepSeek-V4-Flash-0731 @ 9e165c30
What the file holds	The 72 non-expert tensors of the three chained DSpark MTP blocks (mtp.0/1/2): attention, router gate and bias, shared expert, hyper-connection parameters, main_proj, Markov head and confidence head
Parameters	517,318,215, all BF16 (training master). Routed experts are not included; they come unchanged from the target checkpoint.
Weight file	mtp_trained.safetensors, 1,034,643,326 bytes, SHA-256 5dd3ae629d3d…
Measured draft width	6. The checkpoint's own dspark_block_size is 5.
Engine revision at measurement	85dbea6cf of deepseek-v4-flash-0731-cuda
Licence	MIT

Which version should I use

Repo	What it is
DeepSeek-V4-Flash-0731-REAP-DSpark	Recommended. Full 48-shard checkpoint with the later s3recap-p25-b0.1 head installed; 28.38 tok/s at width 5
dspark-mtp-draft-head-s3recap-p25-b0.1	The later head alone, plus an archive of every trained head, including a copy of this one at arms/s3/
This repo	The s3 head, kept as a historical point

Using it

The loadable head is the target checkpoint with these tensors substituted into shards 46–48, re-quantised to the checkpoint's formats. tools/build_trained_head.py in the engine repo does this deterministically. It computes a fresh power-of-two E8M0 block scale for each FP8 tensor and refuses any tensor whose round-trip error exceeds 0.10.

git clone https://github.com/patrickbdevaney/deepseek-v4-flash-0731-cuda && cd deepseek-v4-flash-0731-cuda
python3 tools/build_trained_head.py \
    --base /path/to/DeepSeek-V4-Flash-0731-REAP \
    --trained /path/to/mtp_trained.safetensors \
    --out /path/to/head_out

This is the engine's documented rebuild path. The author used it on Thor. The head has only been run with that CUDA engine (Jetson AGX Thor, sm_110a). No other runtime has been tested.

Evaluation

The protocol is the engine's frozen 8-prompt suite, one prompt per category, with at least 200 generated tokens each, greedy decoding, draft width 6 and adaptK 1.50. τ is the number of tokens committed per target forward; the maximum is 7 at width 6. The log checks the first 8 speculative tokens against plain AR (LOSSLESS GATE ... PASS). Source: eval.log and head_card.json in this repo.

	τ	tok/s
Plain AR decode (same run)	—	13.80
This head, width 6	3.8438	25.53 (1.85x)

Paired re-measurement against the stock head. The two heads were measured back to back on engine 93699e6: stock τ 3.5362 at 22.14 tok/s, s3 τ 3.8438 at 24.25 tok/s, which is +8.70% τ and +9.52% tok/s (docs/BEST_SETUP.md). The often-quoted stock figure of 22.66 tok/s came from an earlier engine revision. Use the paired numbers for a head-to-head comparison.

By category (τ at width 6):

Category	Stock head	s3
long_context	5.54	3.94
agentic_format	5.15	3.64
multi_turn	4.46	6.09
code_edit	4.43	4.20
short_factual	3.27	3.48
reasoning	1.85	3.85
code_gen	1.84	2.49
explanation	1.75	3.06
Suite mean	3.5362	3.8438

The stock column comes from the run-0 measurement in the engine's dev/archive/LOOP_LOG.md (Finding 117). The s3 column comes from evidence/s3_verdict.log.

Held-out drafting. On the 64-sequence hold-out, which is disjoint from training, this head's median τ was 4.622 against 6.000 for the untrained head (−1.378). The session's secondary gate therefore read SATURATED (evidence/s3_verdict.log). The two earlier sessions showed the same pattern against a paired control: −0.404 and −0.648 τ.

At width 5, the checkpoint's native width, the same weights measure τ 3.6888 (docs/HEAD_REGISTRY.md).

Not measured: the head's effect on output quality. Under greedy verification there is none by construction, because the target model decides every token.

How it was made

• Data. 1,536 prompts were balanced 8 ways (192 per category); 1,472 were used for training and 64 held out. Continuations were generated by the target checkpoint itself, and the head was trained on the target's hidden states captured over those sequences. The agentic_format prompts came from BFCL.
• Training. One pass of 1,472 steps over three chunks, 16 positions per sequence. Peak learning rate 5e-5 with 73 warmup steps, then cosine decay. Loss went from 0.9736 to 0.0405 (tail mean 0.4446). The confidence-loss weight was 0. Source: train_metrics.json.
• Selection. The head was promoted over s1 (τ 3.5762) on the frozen suite and served from 2026-08-20 until s3recap-p25-b0.1 replaced it.

Limitations

• Measured on one Jetson AGX Thor with one custom engine.
• Measured at width 6, not the checkpoint's native width 5. At width 5 it falls below the later head.
• Training helped the categories the stock head was weak at and hurt the ones it was strong at (long_context, agentic_format).
• The head is specific to this 160-expert REAP checkpoint and is not expected to transfer to another pruning, quantisation or engine.

Changelog and corrections

• 2026-08-12: first upload.
• 2026-11-06: card rewritten and verified against repo files and build artifacts; a 2026-10-26 automated rewrite contained errors.

License and attribution

MIT, following the target checkpoint. Upstream model by DeepSeek (deepseek-ai/DeepSeek-V4-Flash-0731, MIT). REAP pruning by 0xSero (0xSero/DeepSeek-V4-Flash-0731-REAP, MIT).

Citation

@misc{devaney2026dsv4s3head,
  title  = {DSpark MTP draft head s3 for DeepSeek-V4-Flash-0731-REAP},
  author = {Devaney, Patrick},
  year   = {2026},
  url    = {https://huggingface.co/patrickbdevaney/deepseek-v4-flash-0731-REAP-calibrated-draft-head}
}