appmana/deepseek-v4-int4-int8

DeepSeek-V4-Flash-0731 INT4/INT8 for RTX 30xx and DGX Spark

DeepSeek-V4-Flash-0731 converted for the AppMana/forks-vllm-consumer-nvidia-platforms runtime:

• routed experts: INT4 W4A16 Marlin;
• dense, shared-expert and attention linears: INT8 W8A16 AllSpark;
• sparse-MLA KV cache and indexer: packed INT8 using int8_ds_mla;
• three DSpark MTP draft stages.

Stock vLLM cannot load quant_method=dsv4_int or the int8_ds_mla cache.

Platform support

Platform
RTX 30xx
DGX Spark GB10

Both platforms use the same weights and the same kernel configuration. The runtime image must contain the native extension code for the actual GPU.

Kernel map

This table describes what executes for this INT4/INT8 checkpoint. It does not describe the separate NVFP4/FP8 SparkInfer checkpoint.

Function	RTX 30xx	DGX Spark GB10	Source
Routed-expert MoE GEMM	Marlin W4A16	Marlin W4A16	community (Marlin)
Dense, shared-expert and attention linear GEMMs	AllSpark W8A16	AllSpark W8A16	vLLM
Sparse-MLA decode	flash_mla.sparse_mla_decode_int8	flash_mla.sparse_mla_decode_int8	AppMana (forks-flash-mla-int)
Sparse-MLA prefill	flash_mla.sparse_mla_prefill_int8	flash_mla.sparse_mla_prefill_int8	AppMana (forks-flash-mla-int)
Indexer K cache write	indexer_k_quant_and_cache_int8	indexer_k_quant_and_cache_int8	vLLM
Indexer Q RoPE/quantization	fused Triton	fused Triton	vLLM
Indexer logits	Triton IMMA	Triton IMMA	vLLM
Indexer top-k	native CUDA row/persistent top-k	native CUDA row/persistent top-k	vLLM
KV cache	528-byte int8_ds_mla slots	528-byte int8_ds_mla slots	vLLM
mHC mixing and HCHead	torch/Triton	torch/Triton	vLLM
Speculative decoding	three-stage DSpark	three-stage DSpark	vLLM

Kernel configuration

We added a vllm block to our checkpoint configs so each checkpoint can state which cache and kernels it needs:

"vllm": {
  "kernels": [
    "flash_mla.sparse_mla_decode_int8",
    "flash_mla.sparse_mla_prefill_int8",
    "vllm._custom_ops.indexer_k_quant_and_cache_int8",
    "vllm.models.deepseek_v4.common.ops.fused_indexer_q.fused_indexer_q_rope_quant_int8",
    "vllm.model_executor.layers.quantization.utils.marlin_utils.marlin_act_int8_process_scales",
    "vllm.model_executor.layers.sparse_attn_indexer.streaming_prefill_topk"
  ],
  "cache_type": "int8_ds_mla"
}

vLLM reads this block at startup. --hf-overrides can replace it for testing. Streaming top-k is enabled by default to keep prefill memory bounded at long context.

Deployment

See the public Kubernetes manifests in the examples/deployment directory.

How the checkpoint was built

Converted from deepseek-ai/DeepSeek-V4-Flash-0731@9e165c30e2704aec5d9d593cce3eebd58bbef1cb with tools/ampere/dsv4_requant_checkpoint.py in the AppMana vLLM fork. The published config.json supplies the kernel names and int8_ds_mla selection shown above.

The three DSpark MTP stages are native to 0731 and are converted in place; earlier revisions of this repository grafted them from deepseek-ai/DeepSeek-V4-Flash-DSpark, whose stages were not co-trained with the backbone they were spliced onto. 0731 declares num_nextn_predict_layers: 1 while shipping mtp.0, mtp.1 and mtp.2; the published config states the three stages actually present, since vLLM sizes the draft model from that field.

Quantization

The source routed experts use packed MXFP4 E2M1 values and E8M0 group scales. tools/ampere/dsv4_requant_checkpoint.py in the vLLM fork:

1. unpacks the E2M1 values;
2. applies the E8M0 scale to recover FP32 group values;
3. groups values by 32 along the final dimension;
4. searches BF16-rounded candidate scales;
5. requantizes with round(group / scale).clamp(-8, 7);
6. selects the lowest round-trip squared-error scale (scale_mode: mse, recorded in quantization_config), worth +4.2 to +8.7 dB reconstruction SNR over dividing by the group max;
7. stores two signed INT4 values per byte as uint4b8 (code + 8).

Stored expert weights therefore use signed INT4 nibbles, BF16 per-group scales and group size 32. Dense/shared/attention weights use biased UINT8 plus channel scales for AllSpark W8A16, except attn.wo_a, which is dequantized straight to BF16 with no scale companion: it is consumed by an inverse-RoPE BF16 einsum rather than a GEMM, so quantizing it costs accuracy and buys nothing.

Weights stay in the per-expert, separate-projection checkpoint layout and are fused while loading. Do not pre-fuse the checkpoint.

Benchmarks

Measured on the previous revision of this repository (ace78a6e9b, converted from deepseek-ai/DeepSeek-V4-Flash with absmax7 expert scales and grafted MTP stages). Same kernels, same on-disk convention, same deployment.

RTX 30xx

PP=12, C=1, async scheduling:

Input tokens	Output tokens	DSpark	Decode tok/s	Prefill tok/s
4,000	1,000	off	33.3	—
4,000	1,000	on	56.1	—
620,000	—	off	—	3,230
620,000	—	on	—	3,390
950,000	—	on	66.7	3,020

DGX Spark GB10

Two GB10 systems, TP=2, checkpoint revision ace78a6e9b, DSpark num_speculative_tokens=5, torch.compile with CUDA graphs (capture sizes 1,2,4,8,16), 500,000-token admission limit, int8_ds_mla KV cache pinned at 8 GiB per rank. llm_bench.py, 96 output tokens per request.

C=1:

Input tokens	TTFT s	Prefill tok/s	Decode tok/s	ITL ms p50
1,104	1.2	928.9	70.56	85.2
8,823	10.5	841.0	63.68	88.3
68,924	77.7	886.7	49.71	121.2
137,847	182.2	756.5	44.98	127.6
215,386	279.6	770.4	65.31	91.8
430,771	581.8	740.4	64.45	98.7

C=2 (per stream; prefill aggregate across streams):

Input tokens	TTFT s p50	Prefill tok/s aggregate	Decode tok/s per stream
1,108	2.0	501.4	43.89
8,827	8.2	1,423.1	35.68
68,928	66.7	1,560.7	33.14
137,851	140.1	1,468.1	33.51

The admission limit trades against decode latency: the FULL decode CUDA graphs carry logits scratch sized to the limit, so a 1,000,000-token limit on the same hardware decodes at 13–19 tok/s per stream over the same ladder. 500,000 is the largest measured limit that holds the decode envelope above.

Tool calling

dsv4_tool_bench.py, 20 cases against the same deployment: forced tool calls (calculator, document search), restraint cases that must answer without calling a tool, and multi-turn follow-ups that reuse an earlier tool result. Revision ace78a6e9b: 20/20, 34.9 s wall.

Re-measured 2026-08-07

Same two GB10 systems and the same ace78a6e9b weights, on a rebuilt runtime image (sm86-sm121-8349e7be1c-arm64), to confirm the envelope above is reproducible rather than a one-off. C=1, 256 output tokens:

Input tokens	TTFT s	Prefill tok/s	Decode tok/s	ITL ms p50	Accepted tokens per step
8,841	5.7	1,557.5	69.54	85.7	6.0
68,942	43.8	1,575.3	69.11	86.9	6.0

Decode holds ~69 tok/s at both context lengths. Tool calling on the same run: 20/20, 41.6 s wall.