Nemotron-3-Super-120B-A12B-tq3

TurboQuant 3-bit MLX quantization of nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 — produced with TurboQuant-MLX.

This is the standard 3-bit variant. For 48 GB-RAM Macs, prefer the hybrid tq3a-tq2e-g32 variant (manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32) which keeps attention at 3-bit but pushes experts to 2-bit and fits in ~40.8 GB peak memory.

Model Details

• Base Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 (hybrid Mamba + Sparse Attention + MoE, 120 B params total, ~12 B active per token)
• Architecture: 88 layers, hybrid override pattern MEMEMEM*EMEMEMEM*… (M = Mamba, E = MoE, * = Attention)
• Experts: 512 routed experts + 1 shared expert, latent MoE with moe_latent_size = 1024
• Quantization: TurboQuant 3-bit (Hadamard rotation + Lloyd-Max codebook), group_size=64
• Calibration data: none — TurboQuant is data-free
• Size: ~50 GB on disk
• Peak memory at decode: ~55 GB — needs iogpu.wired_limit_mb raised on a 64 GB Mac
• Runs on: Apple Silicon (M1/M2/M3/M4) with 64 GB or more unified memory (and iogpu.wired_limit_mb raised), or 96/128 GB Macs comfortably

Requirements

pip install "turboquant-mlx-full>=0.2.0" "mlx-lm>=0.31.3"

If you're on a 64 GB Mac, raise the wired-memory limit before generating with this model:

sudo sysctl iogpu.wired_limit_mb=57344

Quick Start

Download the model

hf download manjunathshiva/Nemotron-3-Super-120B-A12B-tq3 \
    --local-dir ~/models/nemotron-3-super-120b-tq3

Generate text — recommended config

The Nemotron-3 family emits a <think> reasoning trace before its final answer, and the chat template primes EOS as the top-1 logit at the start of the assistant turn. Use --min-tokens 50 to ensure generation actually starts. The small repetition penalty prevents long-form generation from collapsing into degenerate tail loops past ~1500 tokens.

turboquant-generate \
    --model ~/models/nemotron-3-super-120b-tq3 \
    --prompt "Why is the sky blue? Explain in detail." \
    --max-tokens 4096 --min-tokens 50 \
    --temp 0.7 --rep-penalty 1.04 --rep-ctx 256

Generate with TurboQuant KV cache (v0.2+) — additional headroom for long context

For long-context generation, layer the v0.2 KV-cache compression on top. K8/V3 mixed precision is required when stacking on TurboQuant-quantized weights — symmetric K3 compounds the noise and breaks long-form output. The 128-token fp16 sink protects attention sinks; the cache builder is hybrid-aware and leaves Mamba/conv-layer caches untouched.

turboquant-generate \
    --model ~/models/nemotron-3-super-120b-tq3 \
    --prompt "Why is the sky blue? Explain in detail." \
    --max-tokens 4096 --min-tokens 50 \
    --temp 0.7 --rep-penalty 1.04 --rep-ctx 256 \
    --kv-k-bits 8 --kv-v-bits 3 --kv-min-tokens 128

Math/numeric prompts — omit rep-penalty

Step-by-step arithmetic on this hybrid is degraded under any non-zero --rep-penalty. For numeric/math prompts, omit it:

turboquant-generate \
    --model ~/models/nemotron-3-super-120b-tq3 \
    --prompt "A train leaves Boston at 9:00 AM going 60 mph..." \
    --max-tokens 2048 --min-tokens 50 \
    --temp 0.7

License

This model inherits the NVIDIA Nemotron Open Model License from the base model.

Citation & Project

Built with TurboQuant-MLX. For the science (Hadamard rotation + Lloyd-Max codebooks for data-free quantization), see Zandieh et al., 2025 — TurboQuant: Online Vector Quantization with Optimal Distortion-Rate Trade-off.