docs/FIDELITY.md

Fidelity

Back to the model card.

KL to the original DeepSeek-V4.1-Flash-1017, measured two ways (recipe/kl/):

• AtomicChat's corpora, our 1017 reference. AtomicChat's V4.1-Flash protocol: three corpora (neutral text, code, agentic transcripts), 24 windows of 4,096 tokens each, the second half of each window scored (49,152 positions per corpus), top-512 logprobs. There is no published 1017 reference for it, so we ran the original 1017 checkpoint ourselves (vLLM, exact FP4 expert weights) twice: once as the reference, once as the noise floor. The neutral corpus is multilingual (about half Latin script, the rest spread over some 20 others); the code and agentic corpora are mostly English.
• kyuz0's short100 panel. An independent reference: full-vocabulary FP32 logits of the original 1017 from DeepSeek's reference code (kyuz0/deepseek-v4.1-flash-1017-qa), 100 cases, 6,159 scored positions.

Each cell shows the mean KL lower bound in nats, then top-1 agreement with the original. Lower KL is better.

Model	Neutral text	Code	Agentic	short100
Original 1017, second run (noise floor)	0.0082 · 97.1%	0.0050 · 98.2%	0.0047 · 98.8%	—
This pack, as served (default)	0.0522 · 92.8%	0.0306 · 96.0%	0.0111 · 98.2%	0.0338 · 95.4%
This pack, as served with the CPU runtime (CPU_EXPERTS=1), mean of 3 runs	0.0520 · 92.9%	0.0309 · 96.0%	0.0112 · 98.2%	0.0354 · 95.4%
The first 1017 release (bdc01239), as served	0.0522 · 92.8%	0.0308 · 96.0%	0.0112 · 98.3%	0.0346 · 95.1%
The first 1017 release, CPU runtime	0.0521 · 92.9%	0.0310 · 95.9%	0.0107 · 98.2%	0.0339 · 95.2%
This pack on the previous release's serving stack	0.0518 · 92.9%	0.0309 · 95.9%	0.0109 · 98.3%	0.0336 · 95.5%

"As served" means recipe/serve/serve.sh with its defaults (GPU-only host experts, the 8-bit prefill all-reduce, the int8 decode-once kernel, FP8 lm_head, the VRAM cache, CUDA graphs and every VRAM trim), with speculative decoding off, an 8K context window and no prefix cache for the logprob capture. Our gate for a serving change is a same-day KL pair on our development build, with every corpus within +3%. Each new change passed it:

Change	Neutral	Code	Agentic	short100
8-bit prefill all-reduce	0.0517 → 0.0519	0.0309 → 0.0309	0.0110 → 0.0109	0.0345 → 0.0341
Int8 decode-once kernel	0.0513 → 0.0519	0.0309 → 0.0311	0.0111 → 0.0110	0.0348 → 0.0344
All four changes	0.0516 → 0.0520	0.0307 → 0.0309	0.0111 → 0.0112	0.0346 → 0.0339

The KL capture sends token IDs and asks for prompt logprobs, so it does not go through the prompt encoder. Requests with prompt logprobs also keep every replay row. The replay trim was therefore checked separately, for identical outputs and teacher-forced agreement (recipe/results/RESULTS.md). An FP8 (e4m3) version of the int8 kernel failed the gate (+5.1% on neutral text) and is not shipped. The previous release's pack scored 0.0545 / 0.0321 / 0.0101 against the original V4.1-Flash on AtomicChat's own reference. Those numbers are not directly comparable: a different model, and a reference with a higher noise floor (0.0158 / 0.0100 / 0.0055), measured on a different engine.

Quality probes with thinking off and greedy decoding:

	GSM8K-200	HumanEval
This pack, default configuration	98.5%	97.0%
The first 1017 release (default; same with the CPU runtime)	98.5%	97.0%
Previous release (V4.1-Flash)	98.0%	95.7%

Text and image input both work. The model's own vision encoder is loaded by default, and images go through the OpenAI image_url content type (VISION=0 serves text only). Prompts are rendered exactly like DeepSeek's 1017 reference encoder: vLLM PR #62196 gave identical prompts, or the same error for invalid input, on all 40 cases we checked. They are also tokenized the way the official API tokenizes them (see CHANGELOG.md, 2026-10-21). A needle in a fresh 1.04M-token prompt is found at the default 1M context window (110.4 s of prefill).

Lossy changes, prompt encoding and API compatibility

• Image input is checked with a small probe (recipe/bench/test-image.py), not an image benchmark suite.
• The quality evidence is the KL table, GSM8K-200, HumanEval and needle tests, not a full benchmark suite.
• Two prefill changes are lossy, and KL to the original model stayed within run-to-run noise for each:
  • the 8-bit prefill all-reduce: about 0.6% relative RMS error per all-reduce against the exact one (DS41_FP8AR=0 serves with the exact NCCL all-reduce);
  • the int8 decode-once kernel: about 1% relative error per expert GEMM, against 0.02% for the fp16 kernel (DO_INT8=0 serves with the fp16 kernel).
• Prompt encoding changed for these requests (the VLLM_DS41_* switches restore the previous behavior one by one):
  • message text that spells a control token;
  • developer messages;
  • response_format;
  • tool_choice: "none";
  • tools with extra or null fields;
  • conversations whose first message is not a system message.
• Some official-API behaviors are not ported:
  • tool_choice required or a named function (vLLM uses guided decoding instead);
  • prefix continuation of the last assistant message;
  • the API's strict-schema normalization.