chiruan/DeepSeek-R1-Distill-Qwen-1.5B-QuaRot-W4A8KV4

DeepSeek-R1-Distill-Qwen-1.5B · QuaRot · W4A8KV4

Complete 28-layer quantized checkpoint from run 20260915T034854Z, exported on 2026-09-15. Base model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B.

This is a BF16 fake-quant checkpoint. W4A8KV4 denotes 4-bit weights, 8-bit activations and 4-bit keys/values in the quantize-dequantize numerical simulation. Weights are stored as BF16 values in safetensors, approximately 3.55 GB. This repository does not provide packed INT4 weights or INT4 kernels.

中文：这是 W4A8KV4 伪量化模型，权重以 BF16 保存。请使用随附的 vLLM 自定义加载代码，以启用激活和 KV 量化及所需变换。

Calibration and format

Hadamard rotations (seed 0), GPTQ, WikiText-2, 128 samples × 2048 tokens; activation clip 0.9 and K/V clip 0.95. The checkpoint selects the local Qwen2 legacy QuaRot implementation.

Weights use per-channel symmetric quantization; activations use per-token asymmetric quantization. K/V use asymmetric quantization with group size 128. Tensor parallelism is 1. Full details are in config.json.

Online Hadamard rotations and A/K/V quantization are provided by the bundled QuaRot runtime.

Load and generate

Use Python 3.11, PyTorch 2.7.0, Transformers 4.51.3 and vLLM 0.9.0. Install a vLLM/PyTorch build that supports your GPU. The environment used to evaluate this checkpoint has CUDA 12.8 and a local vLLM 0.9.0 build for RTX 5090 (SM 12.0). An incompatible vLLM wheel cannot run that GPU.

hf download chiruan/DeepSeek-R1-Distill-Qwen-1.5B-QuaRot-W4A8KV4 --local-dir ./DeepSeek-R1-Distill-Qwen-1.5B-QuaRot-W4A8KV4
cd DeepSeek-R1-Distill-Qwen-1.5B-QuaRot-W4A8KV4
python -m pip install -r requirements.txt
python generate.py --model . --prompt "What is 7 * 8?" --max-new-tokens 128

The included script registers Qwen2QuaRotForCausalLM and runs the bundled adapter. Stock Transformers or vLLM loading alone does not install this checkpoint's custom numerical operations. Use a local downloaded directory; all model files and auxiliary state must remain together.

For QuaRot, the optional fast_hadamard_transform CUDA extension accelerates online rotations; the bundled helper also has a PyTorch fallback.

Validation and provenance

The completed source run passed full checkpoint export and short prefill / cached-decode checks with a 32768-token context configuration. That check does not establish quality on a full 32K-token input. Benchmark evaluation was still in progress when the upload was prepared; no unfinished score is claimed here.

The bundled vLLM adapter comes from the frozen source used in that run. Imports were relocated into a standalone package. For FlatQuant, unrelated optional KV backends were replaced by explicit unsupported-operation guards; the scalar KV path used by this checkpoint is retained. File provenance and changes are recorded in runtime-provenance.json; file integrity hashes are in SHA256SUMS.

License

The base model uses MIT, included in LICENSE. Quantized-Reasoning-Models utilities retain their MIT license. The vLLM/Qwen-derived adapters and QuaRot-related code retain Apache 2.0 licensing. Both license texts and attribution are included; see NOTICE.