LibertAIDAI/GLM-5.3-Flash-NVFP4

GLM-5.3-Flash · NVFP4

320B total · 18B active · natively multimodal · 1M context

598.5 GiB → 181 GiB  ·  −70%  ·  round-trip cosine 0.99665

Weight-only NVFP4 quantization of Z.ai's GLM-5.3-Flash — the first GLM to combine sparse and linear attention.

Quantized by LibertAI · not affiliated with Z.ai / Zhipu

---

✨ What this is

A weight-only NVFP4 (NVFP4-A16) checkpoint. The routed-expert FFN tensors — 97% of the model's parameters — are quantized to NVFP4 (E2M1, with FP8-E4M3 per-16-block scales and an FP32 per-tensor global scale). Everything outlier-sensitive stays in BF16:

both attention flavours (all 34 KDA linear-attention layers and the 11 DeepSeek-sparse layers, including the sparse indexer) · the entire vision tower · shared experts · MoE routers · dense-MLP layers · the MTP head · the Manifold-Constrained Hyper-Connection (mHC) tensors · token embeddings · lm_head · all norms

Activations stay BF16 — there is no activation quantization.

Because the experts dominate the footprint, quantizing only them buys the full 70% while protecting quality where it matters. Leaving the vision tower untouched keeps multimodal behaviour bit-identical to the source.

Produced with NVIDIA ModelOpt 0.45.0 through a memory-frugal shard-streaming pass — CPU-only, never more than one shard resident, no calibration data (weight-only NVFP4 derives its scales from the weights themselves).

---

🚦 Engine support — read this first

glm5_next is a brand-new architecture (released 2026-08-26) and is not yet in vLLM main (vllm#53906, sglang#36507). Support ships in per-model images.

Hardware	Engine	Status
H100 / B200 / GB200	vLLM	✅ vendor-verified
GB10 / DGX Spark (sm_121)	SGLang	✅ verified by us — recipe below
GB10 / DGX Spark (sm_121)	vLLM	❌ see caveat below

vllm/vllm-openai:glm53-flash-x86_64-cu130     # x86_64, CUDA 13
vllm/vllm-openai:glm53-flash-arm64-cu130      # arm64 (GH200/GB200/GB10)
lmsysorg/sglang:glm-5.3-flash-arm64           # SGLang, arm64

⚠️ On sm_121 (GB10), vLLM currently produces wrong output — it echoes the prompt back with a clean finish_reason: stop. We traced this to vLLM's own glm5_next path: its sm_121 MLA kernels assert pe_dim == 64, and GLM-5.3-Flash is NoPE (qk_rope_head_dim = 0). The checkpoint is not at fault — the same files generate correct text on the same GPUs under SGLang, whose TileLang DSA path has a dedicated tail_dim == 0 kernel.

llama.cpp has no glm5_next support, so there is no GGUF.

---

🚀 Usage

SGLang — verified on 2× GB10 (sm_121), TP=2

python3 -m sglang.launch_server \
  --model-path LibertAIDAI/GLM-5.3-Flash-NVFP4 \
  --trust-remote-code --tp-size 2 \
  --attention-backend dsa \
  --dsa-prefill-backend tilelang --dsa-decode-backend tilelang \
  --moe-runner-backend flashinfer_cutlass \
  --kv-cache-dtype bfloat16 \
  --disable-shared-experts-fusion \
  --reasoning-parser glm45 --tool-call-parser glm47 \
  --mem-fraction-static 0.84 \
  --context-length 65536 --max-running-requests 2

Every flag above is load-bearing on sm_121:

Flag	Why it is required
--disable-shared-experts-fusion	The shared expert is BF16 in this checkpoint (only routed experts are quantized). Shared-expert fusion would pack it into the NVFP4 buffer and the load fails on a shape mismatch.
--dsa-*-backend tilelang	The only DSA backend with a NoPE (tail_dim == 0) kernel. flashinfer_sparse_mla hardcodes a rope-bearing 448+64 page layout; all four flashmla_* are excluded by index_kpool=4.
--kv-cache-dtype bfloat16	TileLang on CUDA is bf16-KV-only. Merely omitting an fp8 flag is not enough — the DSA default re-selects fp8.
--moe-runner-backend flashinfer_cutlass	The NVFP4 MoE runner auto-selects a datacenter-Blackwell backend on (12, 1). marlin also works but its repack costs ~19 GiB/rank.
--reasoning-parser glm45	Without it reasoning_parser is None and the entire thinking trace is emitted inside content, ending in a bare </think> with no opening tag, while reasoning_content stays null. See below.
--tool-call-parser glm47	Enables tool calling. ⚠️ Not glm — see the warning below. Note the two parser flags legitimately take different spellings: glm45 for reasoning, glm47 for tools.

🧠 Thinking / reasoning output

This model starts inside its reasoning block without emitting an opening <think> — the chat template opens it — and closes with </think>. With no reasoning parser configured you therefore get:

// ✗ no --reasoning-parser
"content": "The user is asking ... so the answer is 391.</think>391",
"reasoning_content": null

With --reasoning-parser glm45 it splits correctly:

// ✓
"content": "391",
"reasoning_content": "17 × 23 = 17 × 20 + 17 × 3 = 340 + 51 = 391",
"usage": { "reasoning_tokens": 43 }

deepseek-r1 also splits this shape (it is force_reasoning=True on </think>), but prefer glm45: it additionally excludes <tool_call>, </tool_call>, <eop> and <|user|> from the reasoning span, which matters once tool calling is enabled.

🛠️ Tool calling — use glm47, and beware the silent failure

⚠️ --tool-call-parser glm fails silently on this model. It is the GLM-4.5 format; GLM-5.x emits a different one. The symptom is not an error — the request succeeds and you get:

// ✗ --tool-call-parser glm
"finish_reason": "stop",
"tool_calls": null,
"content": ""          // <- the parser consumed the tool call and dropped it

An empty content alongside tool_calls: null is the tell: the parser matched enough to swallow the output but not enough to emit a call. Use glm47, which is also what the vendor vLLM recipe specifies for GLM-5.x.

📐 Sizing: the KDA state cache is the real constraint, not KV

GLM-5.3-Flash has 34 KDA linear-attention layers, each needing a per-request recurrent state. That cost scales with concurrency, and on a memory-tight box it — not the KV cache — is what limits you:

RuntimeError: Hybrid (mamba/linear-attention) state cache is too small to serve any requests.
max_mamba_cache_size=1, mamba_ratio=5, resulting max_num_reqs=0.

So context length and concurrency compete directly. Measured on 2× GB10 (TP=2) at --mem-fraction-static 0.84:

--max-running-requests	--context-length	result
8	131072	❌ mamba cache → max_num_reqs=0
2	65536	✅ KV 212,864 tok (2.52 GB), mamba cache 31 slots (2.19 GB)

If you hit that error, lower --max-running-requests before lowering context — the state cache is per-request, so concurrency is usually the cheaper thing to give up.

--enable-radix-cache (prefix reuse) and CUDA graphs both work; SGLang reports cuda graph: True in its decode lines and initializes a UnifiedTreeCore radix cache with a MAMBA component.

⚠️ GB10 also needs a small TileLang shared-memory patch

SGLang's stock TileLang tile requests 169,984 B of dynamic shared memory. Consumer/workstation Blackwell allows far less — GB10 measures:

shared_memory_per_block_optin = 101,376 B

so the kernel fails to launch. Working tile on GB10: block_I=32, num_stages=1, threads=128. Measured coupling worth knowing: threads must drop to 128 before block_I can drop to 32, or the m_i/alpha fragment layouts become unsatisfiable; block_I=16 hits an MMA assert.

Retuned kernel verified against a float32 gathered-attention reference: rel err 2.4e-3 (bf16 rounding noise) — correct, not merely non-crashing.

Datacenter parts (H100/B200/GB200) have the shared-memory headroom and need no patch.

vLLM — H100 / B200 / GB200

docker run --gpus all --ipc=host -p 8000:8000 \
  vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  --model LibertAIDAI/GLM-5.3-Flash-NVFP4 \
  --tensor-parallel-size 4 \
  --tool-call-parser glm47 --enable-auto-tool-choice \
  --reasoning-parser glm45

The MTP layer is kept in BF16, so speculative decoding works as in the vendor recipe:

  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'

💡 Engine init on a 320B MoE is slow. Set VLLM_ENGINE_READY_TIMEOUT_S=3600 or the server is killed mid-warmup.

---

📦 What's quantized

Tensor group	Precision	Count
Routed-expert FFN (language_model.layers.*.mlp.experts.*.{gate,up,down}_proj)	NVFP4 (g16, weight-only)	37,152
KDA linear attention (34 layers) · DeepSeek sparse attention + indexer (11 layers)	BF16	—
Vision tower (model.visual.*) · shared experts · routers · dense/MTP MLP · mHC · embeddings · lm_head · norms	BF16	1,618

38,770 tensors total. Every non-expert Linear is listed in config.json's ignore.

📝 That ignore list names modules as they appear in the checkpoint (unfused, e.g. *.self_attn.q_proj). Engines fuse at load time and each picks its own name for the fused module, so the list also carries the fused spellings (qkv_proj, fused_qkvbfg_a_proj, fused_fg_b_proj, qkv_conv1d, fused_qkv_a_proj_with_mqa). Without them SGLang treats BF16 attention as quantized and the load asserts.

💾 Size

	BF16 source	This checkpoint
Routed experts (311.65B params)	623 GB	175 GB (NVFP4)
Everything else (9.67B params)	19 GB	19 GB (BF16)
Total	598.5 GiB	≈181 GiB

That is the difference between "needs a GB200 tray" and "fits on two GB10 desktops."

---

🔬 Provenance & verification

• Base: zai-org/GLM-5.3-Flash (BF16, 120 shards, 598.5 GiB)
• Method: shard-streaming ModelOpt NVFP4QTensor weight-only pass, CPU-only. Expert FFN → NVFP4; everything else copied through in BF16.
• Partition check: every Linear weight is provably quantized XOR ignored — no tensor both, none uncovered.
• Round-trip: per-expert cosine ≈ 0.99665, relative error ≈ 0.0925 vs the BF16 source.
• End-to-end: generates correct, coherent text on 2× GB10 under SGLang (TP=2).

⬜ No throughput numbers are published. We have not benchmarked this checkpoint on any hardware, and we would rather publish nothing than publish a number we did not measure.

---

🧠 Why NVFP4?

On NVIDIA Blackwell GPUs (RTX 50-series, B100/B200, GB10/DGX Spark) NVFP4 weights run on native FP4 tensor-core kernels. For an MoE, whose decode is dominated by memory bandwidth, 4-bit expert weights cut both VRAM and the bytes moved per token. This gives format parity with the NVFP4 path used by vLLM, SGLang and TensorRT-LLM.

🌐 About LibertAI

LibertAI is a decentralized AI platform — private inference, an OpenAI-compatible API, and a chat UI, all running on community GPUs over Aleph Cloud instead of a single company's servers. No accounts required to chat, no logs sent home, and the same models you'd self-host are available behind a sovereign endpoint.

Want to put this model to work as an autonomous agent without running your own infrastructure? LiberClaw hosts Hermes-style agents on Aleph Cloud with LibertAI inference. Free tier: 2 agents, no credit card, 5 minutes to deploy. Open source.

---

MIT licensed, as is the base model · Quantized by LibertAI