Apertus-v1.5-70B-FP8-KV8

FP8 (E4M3) quantisation of swiss-ai/Apertus-v1.5-70B with static fp8 KV-cache scales calibrated on German text and written into the checkpoint.

Read this before choosing. The weights here are quantised data-free: FP8_DYNAMIC derives per-channel weight scales from weight min/max alone, so the German calibration reaches only the KV scales, never the weights.

Measured against the BF16 reference, this checkpoint is the stronger of the pair on English and roughly six times weaker on German than the W4A16 sibling, at 71 GiB against 40 GiB.

	German ppl	English ppl	Size
this checkpoint	+4.64%	+3.03%	71 GiB
W4A16 sibling	+0.78%	+9.98%	40 GiB

For German-language work, take the W4A16. Take this one if English matters more, if you want higher weight fidelity, or as the data-free control that makes the W4A16 result interpretable.

---

Files and size

Weights	71 GiB (from ~134 GiB BF16)
Scheme	compressed-tensors / float-quantized, FP8 E4M3, per-channel weight scales
Activations	FP8 E4M3, per-token, dynamic (computed at runtime)
KV cache	static fp8 E4M3, per-tensor, calibrated
Quantised modules	480 Linears (80 decoder layers × 6)
Left in original precision	lm_head, embed_tokens, vision tokenizer (fp32), WavTokenizer (fp32)

The vision and audio tokenizers are not quantised. Swiss AI note they are precision-sensitive and stay in float32 on half-precision loads; the WavTokenizer additionally uses weight-norm parametrised convolutions. recipe.yaml in this repository records the exact recipe used.

---

Requirements

This checkpoint does not load with upstream transformers. Apertus 1.5 uses Apertus1p5ForConditionalGeneration, whose config class ships only in Swiss AI's transformers fork:

pip install "transformers[torch,vision,audio] @ \
  git+https://github.com/swiss-ai/transformers.git@3797303dda74844e3d1f8977ff5518bb91f818b4"

You also need a vLLM with Apertus 1.5 support — Swiss AI's fork, upstream PR #50496, or the prebuilt image onpremai/vllm-apertus-1p5. Verify both before downloading 71 GiB:

from transformers.models.auto.configuration_auto import CONFIG_MAPPING_NAMES
from vllm.model_executor.models.registry import ModelRegistry

assert "apertus1p5" in CONFIG_MAPPING_NAMES, "wrong transformers"
assert "Apertus1p5ForConditionalGeneration" in ModelRegistry.get_supported_archs()

apertus alone in CONFIG_MAPPING_NAMES is Apertus 1.0 support and is not sufficient.

---

Usage

vllm serve <path-or-repo> \
  --dtype auto \
  --enforce-eager \
  --chat-template <path>/chat_template.jinja \
  --chat-template-content-format string \
  --limit-mm-per-prompt '{"image":0,"audio":0}' \
  --enable-auto-tool-choice --tool-call-parser apertus \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92

At 71 GiB this needs roughly 36 GiB per GPU at TP=2.

KV cache precision

--kv-cache-dtype auto resolves to fp8 because this checkpoint carries KV scales, which roughly doubles the KV budget. Pass --kv-cache-dtype bfloat16 explicitly for full-precision KV. One checkpoint covers both cases.

Known serving issue

On some builds vLLM's xIELU activation falls back to forward_native under torch.compile and then emits a log line dynamo cannot trace:

torch._dynamo.exc.Unsupported: logging.Logger method not supported

Workaround: --enforce-eager. This is a vLLM/xIELU interaction, not a property of the checkpoint.

---

Evaluation

Everything below was measured. Perplexity and top-1 agreement on 64 held-out samples × 2048 tokens per language from wikimedia/wikipedia (20231101.de / 20231101.en), 131,008 scored positions each. Generations greedy (temperature=0, top_p=1, seed=0). Measured on one NVIDIA H200 (SM90) under vLLM with --enforce-eager. Eval samples use a different shuffle seed from the calibration set (1234 vs 42).

Perplexity

config	German	Δ	English	Δ
BF16 reference	6.1042	—	3.8545	—
FP8, bf16 KV	6.3876	+4.64%	3.9712	+3.03%
FP8, fp8 KV	6.3867	+4.63%	4.0033	+3.86%

Top-1 agreement with BF16

Fraction of positions where the argmax token is unchanged. More sensitive than perplexity to damage that shows up as malformed morphology, and independent of the perplexity baseline.

config	German	English
FP8, bf16 KV	0.9168	0.9220

fp8 KV cache is close to free on German

Same weights, KV dtype the only difference:

	German	English
Perplexity cost	−0.014%	+0.807%

Zero cost on German — noise-level, slightly negative — and 0.8% on English. The KV scales are the only thing that differs between those two runs, and they were computed from German activations. That asymmetry is the cleanest single piece of evidence in this family that calibration language matters: there is no other candidate explanation.

This is also the payoff from calibrating them offline. vLLM's runtime --calculate-kv-scales, which calibrates against dummy tokens during startup profiling, produced unusable output on the same architecture.

Generation probes

Two five-sentence German prompts, each requiring five specific terms, greedy. Fluent, grammatical German with well-formed nominal compounds; zero non-Latin token bleed.

Where this sits among the alternatives

All built from the same base model and the same German calibration corpus, all measured against the same BF16 reference on the same eval:

Build	Weight scales	Activations	German	English	Size
W4A16 GPTQ, 8192 calib	INT4, group-128	16-bit, unquantised	+0.78%	+9.98%	40 GiB
W4A16 AutoRound, 8192 calib	INT4, group-128	16-bit, unquantised	+0.79%	+7.76%	40 GiB
W4A16 GPTQ, 4096 calib	INT4, group-128	16-bit, unquantised	+0.93%	+10.04%	40 GiB
FP8 dynamic (this)	FP8, per-channel	FP8, per-token dynamic	+4.64%	+3.03%	71 GiB
FP8 static (discarded)	FP8, per-tensor	FP8, static per-tensor	+13.73%	+9.87%	71 GiB

The ranking is monotonic in scale granularity, not bit width. Group-128 INT4 beats per-channel FP8, which beats per-tensor FP8, even though the bit widths run the other way. The last row also quantises activations statically, which compounds the problem: a fixed per-tensor activation scale must accommodate the largest outlier seen during calibration, crushing the resolution available to everything else.

Calibration length matters too. The two GPTQ rows differ only in sequence length — 4096 against 8192 — and that is worth 0.15 points of German perplexity. It also accounts for the apparent gap between AutoRound and GPTQ: at matched calibration depth the two are indistinguishable.

---

Limitations

Weights are quantised data-free. FP8_DYNAMIC derives weight scales from min/max alone. The German calibration reaches only the KV scales. That is why German degrades more than English here (152% of the English damage in absolute nats) while the W4A16 builds show the reverse (8–11%).

Not benchmarked on a standard suite. No MMLU, HellaSwag or similar. Perplexity, top-1 agreement and targeted German probes only.

Calibration corpus is general German, not domain text: 512 samples × 4096 tokens from German Wikipedia (wikimedia/wikipedia, 20231101.de), filtered to samples that fill the full window. Domain-matched calibration would plausibly do better on that domain and is untested here.

fp8 KV tested to ~2k context in the perplexity runs, and to ~6k on the sibling checkpoint. Not validated at the model's full 262,144-token window. KV quantisation error accumulates with depth; test at your own context lengths.

Single-GPU measurement. All numbers from one NVIDIA H200 (SM90). Not tested under tensor parallelism, and not measured on Ada (SM89) despite that being the intended serving target.

---

Reproducing this checkpoint

The quantisation script is in this repository as quantize_apertus.py. Its APERTUS_* environment variables have since been renamed to QUANT_* in the generalised quantize_model.py published alongside the W4A16 sibling; the old names still work and print a deprecation note.

# isolated venv: the swiss-ai transformers fork is OLDER than current upstream
# releases and will replace whatever you have
python3 -m venv venv && source venv/bin/activate

pip install "transformers[torch,vision,audio] @ \
  git+https://github.com/swiss-ai/transformers.git@3797303dda74844e3d1f8977ff5518bb91f818b4"
pip install llmcompressor datasets accelerate

hf auth login                       # swiss-ai repos are gated

# verify the environment first -- full code path, token calibration set, minutes
APERTUS_TESTING=1 APERTUS_SIZE=8B APERTUS_SCHEME=fp8dyn APERTUS_KV_FP8=1 \
  python quantize_apertus.py

# this checkpoint
APERTUS_SIZE=70B APERTUS_SCHEME=fp8dyn APERTUS_KV_FP8=1 \
  CUDA_VISIBLE_DEVICES=0 python quantize_apertus.py

APERTUS_SCHEME=fp8dyn is FP8_DYNAMIC — the data-free path that produced this checkpoint. APERTUS_SCHEME=fp8 is a different thing: GPTQ with the FP8 preset, which uses per-tensor weight scales and static activations and measured substantially worse (see the table above). APERTUS_SCHEME=w4a16 builds the sibling.

The script writes to Apertus-v1.5-70B-FP8dyn-KV8-de; this repository renames it, since -de would imply the weights were German-calibrated and they were not.

Notes for this architecture

Quantised with llm-compressor 0.13.0. These cost real time to find:

• Apertus1p5ForConditionalGeneration and Apertus1p5Model both lack their own forward(), so llm-compressor can trace neither. Pass model.model.language_model (Apertus1p5TextModel) to oneshot(), then save the parent so the wrapper config, lm_head and encoders are written too.
• Quantising the wrapper matches more Linears than intended: the audio tokenizer contributes 25 (24 Vocos backbone + 1 ISTFT head). llm-compressor matches module paths, and the path is audio_tokenizer even though the class is WavTokenizerModel, so a regex on wavtokenizer silently matches nothing. Targeting language_model avoids this structurally.
• A submodule has no name_or_path, so llm-compressor cannot auto-load a processor. Pass processor=tok explicitly.
• save_pretrained must be patched on the parent via modify_save_pretrained(), or the weights are written as fake-quantised BF16 — all of the quality loss, none of the size saving. Check the output size, not just that it saved. The patch uses functools.wraps, so __qualname__ still reports PreTrainedModel.save_pretrained even when active; test the _overridden attribute instead.
• _retie_embeddings() compares input and output embeddings with torch.equal() without aligning devices, and compression onloads modules to the GPU before it runs. On an untied model it returns early anyway, so the script patches it to skip.
• llmcompressor and compressed-tensors must be a matched pair, and vLLM pins compressed-tensors exactly. Installing vLLM into the same venv drags the version back and breaks the import.

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head", "re:.*vision_tokenizer.*", "re:.*audio_tokenizer.*"],
    kv_cache_scheme={"num_bits": 8, "type": "float", "strategy": "tensor",
                     "symmetric": True, "dynamic": False},
)

---

License and provenance

Inherits the licence and usage policy of swiss-ai/Apertus-v1.5-70B. The base model ships EU AI Act documentation and a public training-data summary; consult the upstream repository for both.

Quantisation only — no fine-tuning, no additional training data, no modification of model behaviour beyond numerical precision.