cortexelus commited on
Commit
6736003
·
verified ·
1 Parent(s): 0905190

cpu-amx README: document the bf16 DiT tier (--dit-precision)

Browse files
Files changed (1) hide show
  1. cpu-amx/README.md +8 -4
cpu-amx/README.md CHANGED
@@ -15,7 +15,7 @@ Torch-free **C++ / Intel-AMX** inference engines for **Stable Audio 3 medium**,
15
  pipeline on CPU with no PyTorch, MLX, TFLite, or stable-audio-tools at runtime — just the `.so`s + numpy.
16
 
17
  ```
18
- prompt ─▶ T5Gemma (C++ AMX) ─▶ DiT pingpong (C++ AMX int8) ─▶ SAME-S/L decoder (C++ AMX) ─▶ WAV
19
  audio ─▶ SAME-S/L encoder (C++ AMX) ─▶ latent ─▶ … (audio-to-audio / inpainting)
20
  ```
21
 
@@ -35,7 +35,8 @@ than fp32** — rebuild from source (`optimized/cpu-amx/build/`) for other ISAs.
35
  | file(s) | model | precision / method | notes |
36
  |---|---|---|---|
37
  | `t5gemma_bf16.*` | T5Gemma-b-b-ul2 encoder (text conditioner) | **bf16** AMX GEMMs, fp32 islands (RMSNorm/softmax/softcap/RoPE) | standard softmax attention → bf16-safe |
38
- | `dit_medium_int8.*` + `dit_medium_int8_kernels.tar.gz` | medium DiT (24-block rectified-flow) | **int8** — fully-fused all-integer, AOT-compiled Triton kernels + oneDNN | naive RTN (weight-PTQ *hurts* the DiT — see below) |
 
39
  | `same_{s,l}_decoder_bf16.*` | SAME-S (50M) / SAME-L (426M) decoder | **bf16** AMX GEMMs, fp32 differential-attention islands | highest-fidelity CPU decoders |
40
  | `same_{s,l}_decoder_int8.*` | SAME-S / SAME-L decoder | **int8 (w8a8, fused)** via **SmoothQuant α0.9 → GPTQ** ("improved" grid) | ½ size; SAME-S +1.2 dB over naive int8 on real music |
41
  | `same_{s,l}_encoder_bf16.*` | SAME-S / SAME-L encoder (audio→latent) | **bf16** AMX GEMMs, fp32 differential-attention islands | for audio-to-audio / inpainting |
@@ -43,8 +44,11 @@ than fp32** — rebuild from source (`optimized/cpu-amx/build/`) for other ISAs.
43
  | `same_l_encoder_bf16_weights_f32.bin` | SAME-L encoder (optional) | **fp32** refinement mode | max-fidelity; bf16 is the default |
44
 
45
  **Quantization findings baked into these choices** (full write-up in the repo's `LESSONS.md`):
46
- - The medium **DiT stays naive-int8** — GPTQ/SmoothQuant *lower* its accuracy (its adaLN-modulated qkv +
47
- the chaotic 8-step sampler make a calibration-averaged Hessian overfit and generalize worse than RTN).
 
 
 
48
  - The **decoders are activation-limited**: SmoothQuant (activation-outlier migration) does the work,
49
  GPTQ just lets α go higher. The gain **transfers to real music for SAME-S (+1.2 dB), not SAME-L**.
50
  - **bf16** is used where attention is either standard-softmax (T5Gemma) or where the fp32 differential-
 
15
  pipeline on CPU with no PyTorch, MLX, TFLite, or stable-audio-tools at runtime — just the `.so`s + numpy.
16
 
17
  ```
18
+ prompt ─▶ T5Gemma (C++ AMX) ─▶ DiT pingpong (C++ AMX int8 | bf16) ─▶ SAME-S/L decoder (C++ AMX) ─▶ WAV
19
  audio ─▶ SAME-S/L encoder (C++ AMX) ─▶ latent ─▶ … (audio-to-audio / inpainting)
20
  ```
21
 
 
35
  | file(s) | model | precision / method | notes |
36
  |---|---|---|---|
37
  | `t5gemma_bf16.*` | T5Gemma-b-b-ul2 encoder (text conditioner) | **bf16** AMX GEMMs, fp32 islands (RMSNorm/softmax/softcap/RoPE) | standard softmax attention → bf16-safe |
38
+ | `dit_medium_int8.*` + `dit_medium_int8_kernels.tar.gz` | medium DiT (24-block rectified-flow) | **int8** — fully-fused all-integer, AOT-compiled Triton kernels + oneDNN | naive RTN (weight-PTQ *hurts* the DiT — see below); default, 1-thread core |
39
+ | `dit_medium_bf16.*` (reuses the int8 kernels tarball) | medium DiT | **bf16** AMX GEMMs (oneDNN, packed weights) + bf16 flash attention, **fp32 RoPE/RMSNorm islands** | **near-lossless** (~59/54 dB @ L1292/L4096, cos 0.9997+) — `--dit-precision bf16`; ~1.24× the int8 latency, runs at `--threads` |
40
  | `same_{s,l}_decoder_bf16.*` | SAME-S (50M) / SAME-L (426M) decoder | **bf16** AMX GEMMs, fp32 differential-attention islands | highest-fidelity CPU decoders |
41
  | `same_{s,l}_decoder_int8.*` | SAME-S / SAME-L decoder | **int8 (w8a8, fused)** via **SmoothQuant α0.9 → GPTQ** ("improved" grid) | ½ size; SAME-S +1.2 dB over naive int8 on real music |
42
  | `same_{s,l}_encoder_bf16.*` | SAME-S / SAME-L encoder (audio→latent) | **bf16** AMX GEMMs, fp32 differential-attention islands | for audio-to-audio / inpainting |
 
44
  | `same_l_encoder_bf16_weights_f32.bin` | SAME-L encoder (optional) | **fp32** refinement mode | max-fidelity; bf16 is the default |
45
 
46
  **Quantization findings baked into these choices** (full write-up in the repo's `LESSONS.md`):
47
+ - The medium **DiT's quantized tier stays naive-int8** — GPTQ/SmoothQuant *lower* its accuracy (its
48
+ adaLN-modulated qkv + the chaotic 8-step sampler make a calibration-averaged Hessian overfit and
49
+ generalize worse than RTN). For fidelity-over-speed there's a **near-lossless `bf16` DiT** tier
50
+ (`--dit-precision bf16`): bf16 only on the AMX matmuls (GEMMs + flash attention), RoPE/RMSNorm kept in
51
+ fp32 islands (bf16-RoPE angle breaks long renders). ~59/54 dB vs the int8 tier's ~40/36 dB, at ~1.24× the latency.
52
  - The **decoders are activation-limited**: SmoothQuant (activation-outlier migration) does the work,
53
  GPTQ just lets α go higher. The gain **transfers to real music for SAME-S (+1.2 dB), not SAME-L**.
54
  - **bf16** is used where attention is either standard-softmax (T5Gemma) or where the fp32 differential-