ubergarm

/

GLM-5.1-GGUF

Like 26

how much does this quant need in total for example? IQ2_KL 261.988 GiB (2.985 BPW)

I recommend looking closely at the startup logs when running llama-server which will break it down and tell you exactly how much MiB is used for what e.g. kv-cache, VRAM, CPU buffers etc depending on how you are running it.

As I mention in the model card next to each one, the size printed is not actual size used in memory due to unimplemented tensors for lightning/DSA etc. So for the IQ2_KL, when I run it it says:

numactl -N "$SOCKET" -m "$SOCKET" \
./build/bin/llama-perplexity \
    -m "$model" \
    -f wiki.test.raw \
    --seed 1337 \
    --ctx-size 512 \
    -mla 3 \
    -muge \
    --merge-qkv \
    -ub 4096 -b 4096 \
    --no-mmap \
    --numa numactl \
    --threads 96 \
    --threads-batch 128 \
    --validate-quants

llm_load_print_meta: model size       = 261.988 GiB (2.985 BPW)
...
Free memory 0 MiB on device 0 is less the required compute buffer size 672 MiB
Memory required for model tensors + cache: 267541 MiB
Memory available on all devices - compute: 0 MiB
llm_load_tensors: ggml ctx size =    0.77 MiB
...
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/80 layers to GPU
llm_load_tensors:        CPU buffer size = 261984.39 MiB
llama_kv_cache_init:        CPU KV buffer size =   351.00 MiB
llama_init_from_model: KV self size  =  351.00 MiB, c^KV (f16):  351.00 MiB, kv^T: not used
llama_init_from_model:        CPU  output buffer size =     4.73 MiB
llama_init_from_model:        CPU compute buffer size =  2516.00 MiB

This example is only 512 context, but easy to adjust and see the changes.

Because this model uses deep seek style MLA multihead latent attention, you only need to specify -ctk f16 or -ctk q8_0 . When you do this it will print out the updated totals, so you can see the amount of overhead due to kv-cache. Using -ub 4096 -b 4096 will increase compute buffers as well. The kv-cache and compute buffer overhead are the same across all the models. Each model only changes the weights size.

It's a good excercise to make a graph with x-axis the amount of context you specify, and the y axis the kv-cache buffer size in MiB. I believe it grows linearly for this style model, and going from f16 to q8_0 would require about half the VRAM for same context length. Some other models it grows exponentially depending on exact head dimensions etc.

With 4x RTX Pros the IQ3_KS will be a great size leaving plenty of extra VRAM for full context at full -ctk f16 (default) as it is quite efficient. You have a great local vibe-coding setup! Cheers!

Look at this post for some more examples of logs: https://huggingface.co/ubergarm/GLM-5.1-GGUF/discussions/3

Thank you, this is very helpful. Spent some time today testing variations.

Long shot, but perhaps you have some ideas about this - trying to fit IQ2_KL into 3 RTX 6000 Pros with ik_llama.
Long story short - can't fit much more than 60k context, some cards have VRAM, 1 is a bottleneck.
This is real VRAM usage across 2 gpus after ~10 simple prompts: 95.14, 90.72,88.95
In theory, 2nd and 3rd ones could just have 1 extra weights layer (3464.74 fits).

I wonder if there is any way to make a custom mapping for that configuration to make everything fit and use remainder of the VRAM for more context?

llm_load_tensors: offloaded 80/80 layers to GPU
llm_load_tensors:  CUDA_Host buffer size =   510.47 MiB
llm_load_tensors:      CUDA0 buffer size = 90935.14 MiB
llm_load_tensors:      CUDA1 buffer size = 86618.56 MiB
llm_load_tensors:      CUDA2 buffer size = 83920.24 MiB

...

llama_kv_cache_init:      CUDA0 KV buffer size =  1018.27 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =   877.82 MiB
llama_kv_cache_init:      CUDA2 KV buffer size =   842.71 MiB
llama_init_from_model: KV self size  = 2738.76 MiB, c^KV (q8_0): 2738.76 MiB, kv^T: not used
llama_init_from_model:  CUDA_Host  output buffer size =     0.59 MiB
llama_init_from_model: pipeline parallelism enabled (n_copies=1)
llama_init_from_model:      CUDA0 compute buffer size =  3264.70 MiB
llama_init_from_model:      CUDA1 compute buffer size =  3264.70 MiB
llama_init_from_model:      CUDA2 compute buffer size =  3264.71 MiB
llama_init_from_model:  CUDA_Host compute buffer size =   283.02 MiB
llama_init_from_model: graph nodes  = 6818
llama_init_from_model: graph splits = 4

Tightened batch sizes a bit which helped. Meanwhile will try to further reduce them further. Current config

      /app/run-server.sh
      --model /models/models--ubergarm--GLM-5.1-GGUF/snapshots/697c5ea8b228fdb2af1342ea3cd7be73694b99ab/IQ2_KL/GLM-5.1-IQ2_KL-00001-of-00007.gguf
      --port 8088
      --alias glm-5.1-q2
      --jinja
      -c 60000
      --threads 1
      --n-gpu-layers 99
      --merge-qkv
      -ctk q8_0
      -mla 3 -amb 2048
      -smgs -sas -muge
      --grouped-expert-routing
      --batch-size 4096
      --ubatch-size 2048
      --no-mmap
      --temp 1.0
      --top-p 0.95
      --host 0.0.0.0
      -cram 50000

Found it.
Tuned layers offloading specifically for IQ2_XL quant on 3 RTX 6000 Pro (96gb)

      -ot "blk\.([0-9]|1[0-9]|2[0-7])\.ffn=CUDA0"
      -ot "blk\.(2[8-9]|3[0-9]|4[0-9]|5[0-2])\.ffn=CUDA1"
      -ot "blk\.(5[3-9]|6[0-9]|7[0-9])\.ffn=CUDA2"

This allows full context and higher batches. Which has noticeable performance impact on Blackwell. Tested with those. But would probably sacrifice some KV for even higher batches.
--batch-size 4096
--ubatch-size 2048

@curiouspp8

nice job tuning up your rig! yes increasing -ub 2048 -b 2048 etc will likely improve PP speeds but costs some more VRAM for larger compute buffers.

Since you are on full GPU offload you might benefit from playing with speculative decoding stuff too e.g.: --spec-type ngram-map-k4v --spec-ngram-size-n 8 --draft-min 1 --draft-max 4 --draft-p-min 0.2 but honestly i don't know how to tune that myself or what to expect from it.

https://github.com/ikawrakow/ik_llama.cpp/pull/1261