Instructions to use L-Alchemyst/GLM-5.3-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use L-Alchemyst/GLM-5.3-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf L-Alchemyst/GLM-5.3-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf L-Alchemyst/GLM-5.3-GGUF:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf L-Alchemyst/GLM-5.3-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf L-Alchemyst/GLM-5.3-GGUF:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf L-Alchemyst/GLM-5.3-GGUF:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf L-Alchemyst/GLM-5.3-GGUF:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf L-Alchemyst/GLM-5.3-GGUF:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf L-Alchemyst/GLM-5.3-GGUF:Q2_K

Use Docker

docker model run hf.co/L-Alchemyst/GLM-5.3-GGUF:Q2_K

LM Studio
Jan
Ollama
How to use L-Alchemyst/GLM-5.3-GGUF with Ollama:
```
ollama run hf.co/L-Alchemyst/GLM-5.3-GGUF:Q2_K
```
Unsloth Desktop

How to use L-Alchemyst/GLM-5.3-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf L-Alchemyst/GLM-5.3-GGUF:Q2_K

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "L-Alchemyst/GLM-5.3-GGUF:Q2_K"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use L-Alchemyst/GLM-5.3-GGUF with Docker Model Runner:
```
docker model run hf.co/L-Alchemyst/GLM-5.3-GGUF:Q2_K
```

Lemonade

How to use L-Alchemyst/GLM-5.3-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull L-Alchemyst/GLM-5.3-GGUF:Q2_K

Run and chat with the model

lemonade run user.GLM-5.3-GGUF-Q2_K

List all available models

lemonade list

Hermes Agent

How to use L-Alchemyst/GLM-5.3-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf L-Alchemyst/GLM-5.3-GGUF:Q2_K

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default L-Alchemyst/GLM-5.3-GGUF:Q2_K

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use L-Alchemyst/GLM-5.3-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf L-Alchemyst/GLM-5.3-GGUF:Q2_K

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "L-Alchemyst/GLM-5.3-GGUF:Q2_K" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

GLM-5.3

Imatrix quants best-in-class for ik_llama.cpp

NOTE: ik_llama.cpp is highly versatile and runs mainline llama.cpp quants such as existing GGUFs from bartowski, unsloth, mradermacher, Aes Sedai etc

IQ2_K done! ✅ IQ3_K done! ✅ IQ4_K done! ✅. All quants done unless someone really wants a Q1. See the collection below.

General Specs

	Value
Architecture	GLM-DSA (MoE, 256 experts × 22B, 8 active)
Params	744B total / 40B active
Context	1,048,576
Base	zai-org/GLM-5.3-BF16
Imatrix	Aes Sedai — converted for ik_llama.cpp
Calibration	calibration_datav5.txt

PPL Analysis

Quant Collection

✅ IQ4_K 4.581 BPW · 401.77 GiB

Evaluated on wiki.test.raw, 512 ctx, 565 chunks, against BF16 reference logits.

Why the specific layer targeting? (Analysis notes)

Rather than relying solely on conventional depth-based heuristics, the quantization targets were selected using imatrix activation-energy analysis and residuals after detrending for the model's natural increase in activation energy with depth.

The analysis identified blk.3–11 and blk.76–77 as unusually low-energy relative to their expected depth, making them candidates for more aggressive iq4_ks quantization. This is intentionally non-conventional. Since the measured sensitivity profile for these layers was weaker than expected.

In contrast, the mid-depth MoE layers, particularly around blk.29–40, show substantially higher residual energy and were retained at iq4_k. The expert down projections were also treated more conservatively because they feed directly back into the residual stream.

Detrended layer ranking (down output energy):
Trend slope: d(log E)/d(layer) = 0.1545. Residual = actual/trend; lower residual ⇒ weaker than depth predicts.

blk.3 resid=0.016 up_flow=2.331e+06 blk.4 resid=0.075 up_flow=3.914e+06 blk.5 resid=0.110 up_flow=4.457e+06 blk.6 resid=0.209 up_flow=7.723e+06 blk.7 resid=0.289 up_flow=1.349e+07 blk.76 resid=0.292 up_flow=5.089e+09 blk.10 resid=0.325 up_flow=2.200e+07 blk.9 resid=0.349 up_flow=2.073e+07 blk.77 resid=0.350 up_flow=2.786e+09 blk.11 resid=0.355 up_flow=2.457e+07 blk.75 resid=0.364 up_flow=6.132e+09 blk.67 resid=0.377 up_flow=4.028e+09 ... blk.36 resid=4.166 up_flow=9.199e+08 blk.32 resid=4.225 up_flow=6.227e+08 blk.34 resid=4.276 up_flow=7.851e+08 blk.33 resid=4.401 up_flow=7.199e+08 blk.35 resid=4.430 up_flow=8.652e+08

The resulting recipe is therefore sensitivity-guided rather than position-guided: quantize the measured low-sensitivity outliers slightly more, while preserving precision through the high-energy middle of the network.

Metric	Value
PPL (quant)	2.707673 ± 0.014047
PPL (BF16 ref)	2.674315 ± 0.013811
(PPL(Q)/PPL(base)) - 1	+1.2473% ± 0.0963%
KLD	0.041196 ± 0.000334
Same top-p	93.941 ± 0.063 %
Δp RMS	7.839 ± 0.052 %

Quantization recipe

# Attention blk\..*\.attn_k_b\.weight=q8_0 blk\..*\.attn_v_b\.weight=q8_0 blk\..*\.attn_kv_a_mqa\.weight=q8_0 blk\..*\.attn_q_a\.weight=q8_0 blk\..*\.attn_q_b\.weight=q8_0 blk\..*\.attn_output\.weight=q8_0

# Router blk\..*\.ffn_gate_inp\.weight=f32

# First 3 Dense Layers blk\..*\.ffn_down\.weight=iq6_k blk\..*\.ffn_(gate|up)\.weight=iq5_ks

# Shared Expert Layers blk\..*\.ffn_down_shexp\.weight=iq6_k blk\..*\.ffn_(gate|up)_shexp\.weight=iq6_k

# Routed Experts blk\.(78)\.ffn_down_exps\.weight=q6_K blk\.(78)\.ffn_(gate|up)_exps\.weight=q6_K blk\.(3|4|5|6|7|8|9|10|11|76|77)\.ffn_down_exps\.weight=iq4_ks blk\.(3|4|5|6|7|8|9|10|11|76|77)\.ffn_(gate|up)_exps\.weight=iq4_ks blk\..*\.ffn_down_exps\.weight=iq4_k blk\..*\.ffn_(gate|up)_exps\.weight=iq4_k

# Indexer blk\..*\.indexer\.attn_q_b\.weight=q6_K blk\..*\.indexer\.attn_k\.weight=q8_0 blk\..*\.indexer\.proj\.weight=q8_0 blk\..*\.indexer\.k_norm\.(weight|bias)=f32

# MTP block - eh_proj is the throughput-critical tensor per ik's finding blk\..*\.nextn\.eh_proj\.weight=q4_0 blk\..*\.nextn\.enorm\.weight=f32 blk\..*\.nextn\.hnorm\.weight=f32 blk\..*\.nextn\.shared_head_norm\.weight=f32

token_embd\.weight=iq6_k output\.weight=iq6_k

✅ IQ3_K 3.670 BPW · 321.83 GiB

Evaluated on wiki.test.raw, 512 ctx, 565 chunks, against BF16 reference logits.

Metric	Value
PPL (quant)	2.820603 ± 0.014842
PPL (BF16 ref)	2.674315 ± 0.013811
(PPL(Q)/PPL(base)) - 1	+5.4701% ± 0.1627%
KLD	0.108800 ± 0.000809
Same top-p	90.352 ± 0.078 %
Δp RMS	12.780 ± 0.070 %

Quantization recipe

# Attention blk\..*\.attn_k_b\.weight=q8_0 blk\..*\.attn_v_b\.weight=q8_0 blk\..*\.attn_kv_a_mqa\.weight=q8_0 blk\..*\.attn_q_a\.weight=iq6_k blk\..*\.attn_q_b\.weight=iq6_k blk\..*\.attn_output\.weight=iq6_k

# Router blk\..*\.ffn_gate_inp\.weight=f32

# First 3 Dense Layers blk\..*\.ffn_down\.weight=iq6_k blk\..*\.ffn_(gate|up)\.weight=iq5_ks

# Shared Expert Layers blk\..*\.ffn_down_shexp\.weight=iq6_k blk\..*\.ffn_(gate|up)_shexp\.weight=iq6_k

# Routed Experts blk\.(78)\.ffn_down_exps\.weight=q6_K blk\.(78)\.ffn_(gate|up)_exps\.weight=q6_K blk\..*\.ffn_down_exps\.weight=iq4_ks blk\..*\.ffn_(gate|up)_exps\.weight=iq3_ks

# Indexer blk\..*\.indexer\.k_norm\.(weight|bias)=f32 blk\..*\.indexer\.proj\.weight=q8_0 blk\..*\.indexer\.attn_k\.weight=q8_0 blk\..*\.indexer\.attn_q_b\.weight=q6_K

token_embd\.weight=iq5_k output\.weight=iq6_k

✅ IQ2_K 2.695 BPW · 236.39 GiB

The Q2 to run on 256 GiB systems. Tight 256 GiB-friendly footprint at 2.695 BPW. GGUF upload in progress — see status above.

Evaluated on wiki.test.raw, 512 ctx, 565 chunks, against BF16 reference logits.

Metric	Value
PPL (quant)	3.436789 ± 0.018731
PPL (BF16 ref)	2.674315 ± 0.013811
(PPL(Q)/PPL(base)) - 1	+28.511% ± 0.3730%
KLD	0.381602 ± 0.002102
Same top-p	80.498 ± 0.104 %
Δp RMS	24.399 ± 0.087 %

Quantization recipe

# Router blk\..*\.ffn_gate_inp\.weight=f32

# First 3 Dense Layers blk\..*\.ffn_down\.weight=iq6_k blk\..*\.ffn_(gate|up)\.weight=iq5_ks

# Shared Expert Layers blk\..*\.ffn_down_shexp\.weight=iq6_k blk\..*\.ffn_(gate|up)_shexp\.weight=iq6_k

# Routed Experts blk\.(78)\.ffn_down_exps\.weight=q6_K blk\.(78)\.ffn_(gate|up)_exps\.weight=q6_K blk\..*\.ffn_down_exps\.weight=iq3_ks blk\..*\.ffn_(gate|up)_exps\.weight=iq2_ks

# Indexer blk\..*\.indexer\.attn_q_b\.weight=q6_K blk\..*\.indexer\.attn_k\.weight=q8_0 blk\..*\.indexer\.proj\.weight=q8_0 blk\..*\.indexer\.k_norm\.(weight|bias)=f32

token_embd\.weight=iq6_k output\.weight=iq6_k

Quick Start

Requires ik_llama.cpp. Standard llama.cpp will not run these quants.

# Clone and build git clone https://github.com/ikawrakow/ik_llama.cpp cd ik_llama.cpp cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON cmake --build build --config Release -j $(nproc)

# Download pip install huggingface_hub hf download L-Alchemyst/GLM-5.3-GGUF --repo-type model --include "IQ3_K/*" --local-dir ./glm5.3-iq3k

# Hybrid CPU+GPU — adjust --n-cpu-moe and -cram to your RAM ./build/bin/llama-server
--model GLM-5.3-IQ3_K.gguf
-muge --merge-qkv
--ctx-size 131072
-ctk f16 -mla 3 -amb 1536
-ngl 999
--n-cpu-moe 50
--parallel 1
--threads 96 --threads-batch 128
--host 127.0.0.1 --port 8080
--no-mmap
-cram 8192
--jinja

CPU-only: use -ctk q8_0 and prefix with numactl -N ${SOCKET} -m ${SOCKET}.

Credits
Imatrix & reference logits: Aes Sedai
Base model: zai-org/GLM-5.3-BF16
Runtime: ik_llama.cpp

Downloads last month: 21,616

GGUF

Model size

753B params

Architecture

glm-dsa

Hardware compatibility

2-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for L-Alchemyst/GLM-5.3-GGUF

Base model

zai-org/GLM-5.3-BF16

Quantized

(23)

this model