Instructions to use unsloth/Qwen3.8-Flash-Next-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use unsloth/Qwen3.8-Flash-Next-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL # Run inference directly in the terminal: ./build/bin/llama cli -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
Use Docker
docker model run hf.co/unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
- Llama new
- LM Studio
- Jan
- vLLM
How to use unsloth/Qwen3.8-Flash-Next-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "unsloth/Qwen3.8-Flash-Next-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "unsloth/Qwen3.8-Flash-Next-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
- Ollama
How to use unsloth/Qwen3.8-Flash-Next-GGUF with Ollama:
ollama run hf.co/unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
- Unsloth Desktop
- Pi
How to use unsloth/Qwen3.8-Flash-Next-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use unsloth/Qwen3.8-Flash-Next-GGUF with Docker Model Runner:
docker model run hf.co/unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
- Lemonade
How to use unsloth/Qwen3.8-Flash-Next-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
Run and chat with the model
lemonade run user.Qwen3.8-Flash-Next-GGUF-UD-Q4_K_XL
List all available models
lemonade list
- Hermes Agent
How to use unsloth/Qwen3.8-Flash-Next-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use unsloth/Qwen3.8-Flash-Next-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Qwen3.8-Flash-Next (qwen4exp) garbage output on Strix Halo (gfx1151) over ROCm 7.x - root cause and verified fix (build from PR #27941)
Qwen3.8-Flash-Next (qwen4exp) produces garbage on Strix Halo / gfx1151 over ROCm 7.x - root + verified fix
TL;DR
Every released/prebuilt llama.cpp build (as of 2026-08-31) generates corrupted output for Qwen3.8-Flash-Next (qwen4exp arch) on a Strix Halo APU (AMD Ryzen AI MAX+ 395, gfx1151, ROCm 7.14) over a unified-memory / --kv-unified setup. The model quickly degenerates to a wall of /, or a mixed-language token soup, and the server throws peg-native 500 errors. This is not a quantization problem, not a flag problem, and not a client/Studio problem - it is an unfixed bug in the qwen4exp KV/QSA indexer in released llama.cpp. Building llama.cpp from PR #27941 (868e2f52 on the qwen4exp follow-up-fixes branch) fixes it completely.
Symptom
With any released build (bundled Unsloth Studio b10687, lemonade-sdk/llamacpp-rocm b1318/3173a56), loading unsloth/Qwen3.8-Flash-Next-GGUF and generating:
- First ~150 tokens are often plausible, then the decode collapses into a repetition of a single garbage token stream:
/,syn, mixed-CJK/Russian/English, or token soup likeglobal/world/global_global. - Multi-turn / multi-segment prompts degenerate on the very first generated token.
- The chat endpoint intermittently returns
500: "The model produced output that does not match the expected peg-native format". - Raw
/completion,/v1/chat/completions, reasoning on/off,--skip-chat-parsing, CPU-only, GPU offload - all reproduce it.
Environment where reproduced
- AMD Strix Halo APU: Ryzen AI MAX+ 395, Radeon 8060S, gfx1151
- ROCm 7.14, HIP, unified memory
- Kernel 7.x / Debian 13
- Builds tested: bundled b10687 (unsloth release blobs), lemonade b1318 (
3173a56), upstream master todaef7b687 - Quantizations tested: UD-Q2_K_XL, UD-Q4_K_XL (the published "Q3" shards in the hub are git-lfs pointer stubs on HuggingFace, not weights)
What this note is NOT claiming
This is specific to the qwen4exp architecture on this hardware/config. Other models (plain dense, MoE variants) served perfectly on the same binary and env. So the build itself, ROCm stack, GPU kernels, and hardware are all sound - the bug is qwen4exp-specific.
Investigation trail (what was ruled out)
To save others the hours we spent, here is the systematic elimination. In every case the garbage was reproducible.
Not a quantization issue. Q2 and the recommended Q4 (
UD-Q4_K_XL, 111.3 GB) produce byte-identical garbage on the broken build. Override-tensor of PLE/ngram tensors and metadata inspection showed the n-gram tensors preserved at high precision regardless. Quantization is exonerated.Not Unsloth Studio specifically. The exact same garbage reproduces running llama.cpp standalone (no Studio at all) with the same model and flag set.
Not the vision/multimodal module. Launch config with
--mmprojand identical config without--mmprojproduced the same garbage.--mmprojis not the trigger.Not a flag / template / parser problem. Minimal config, Studio-heavy config, recommended config,
--reasoning onvs--chat-template-kwargs enable_thinking,--spec-defaulton/off,--parallel 1/4, context 8K vs 32K vs 262K,--skip-chat-parsing, sampling overrides - none changed the outcome.Not a CPU/GPU kernel issue. CPU-only decode (
-ngl 0) also degenerated (after a transient clean start). The GPU path is not the sole cause.Not the
--kv-unifiedunified-memory flag by itself. The collapse also happens in plain KV mode.--kv-unifiedmerely makes it deterministic/earlier.
Root cause (as identified in upstream PRs)
The qwen4exp implementation has an incorrect KV-cache / QSA (Quadratic Sparse Attention) block selector and PLE n-gram indexer in llama-memory-hybrid-idx. Under long-context, unified-cache, multi-sequence, or multimodal inference it pools history blocks from the wrong positions/sequences, causing "silent logit drift / quality loss" - the model loses the ability to predict anything not already in its degenerate loop. There is also a separate --kv-unified NaN-collapse bug. Relevant upstream discussion:
- https://github.com/ggml-org/llama.cpp/issues/27797 - deterministic multi-segment
//////repro on gfx1151 - https://github.com/ggml-org/llama.cpp/issues/27763 - garbage output on SM110 when >8 layers offloaded
- https://github.com/ggml-org/llama.cpp/pull/27941 - qwen4exp follow-up fixes, includes the
-kvu NaN collapse fix - https://github.com/ggml-org/llama.cpp/pull/27879 - qwen4exp correctness fixes (draft/reference)
The verified fix
Build llama.cpp from PR #27941 head (868e2f52). Do not merge the draft PR #27879 on top (it conflicts with current master; #27941 already carries the equivalent correctness work and is self-consistent).
git clone https://github.com/ggml-org/llama.cpp.git llama.cpp-fixed
cd llama.cpp-fixed
git config user.email you@example.com && git config user.name you
git fetch origin pull/27941/head:pr27941
git checkout -B fix-qwen4exp pr27941
mkdir -p build && cd build
cmake .. -G Ninja -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 \
-DCMAKE_BUILD_TYPE=Release -DGGML_HIP_ROCWMMA=OFF -DCMAKE_PREFIX_PATH=/opt/rocm
ninja llama-server llama-cli
Run (test flags + vision):
HSA_OVERRIDE_GFX_VERSION=11.5.1 LD_LIBRARY_PATH="<build>/bin:/opt/rocm/lib" \
./bin/llama-server -m <UD-Q4_K_XL shard 1> \
--mmproj <mmproj-F16.gguf> --host 127.0.0.1 --port 18399 \
--jinja --reasoning on --reasoning-budget 256 --flash-attn on \
--no-context-shift -c 32768 --fit off --kv-unified -ngl -1 --n-predict 1024
(--reasoning-budget is important for the hybrid-thinking model; without it a longer task spends the whole budget on thinking and content comes back empty with finish_reason: length.)
Result after the fix
Verified clean, all with finish_reason: stop and no junk:
- "What is the capital of France?" -> "The capital of France is Paris."
- "What is 2+2? Answer with just the number." -> "4"
- 5 repeated identical prompts with no degradation (the broken build collapses on the 2nd or 3rd)
- A long cat-story generation in fluent English
- Multi-turn conversation tracked correctly ("My name is Bob..." -> "Your name is Bob, and you are in Poland.")
Throughput ~21 t/s on the Q4_K_XL at -c 32768 on this APU.
Status / caveat
As of 2026-08-31, PR #27941 (and #27879) are open and not merged into master, and no released/prebuilt llama.cpp contains the fix yet. The workaround is to build from #27941 yourself. Once these land upstream and a release/prebuilt is published, a stock build should work.
Hope this saves the next person the multi-day hunt. Questions welcome.
CLARIFICATION / CORRECTION (2026-08-31, later same day)
Further testing has narrowed the root cause down further, and it overturns part of the original post. The real trigger is GGML_CUDA_ENABLE_UNIFIED_MEMORY=1, not the qwen4exp indexer/build being broken in released llama.cpp.
Controlled evidence
Controlled A/B, identical build + model + flags, only the env var differs:
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1-> garbage (token soup,finish=length,////)- without it (or
ROCR_VISIBLE_DEVICES=0, no unified memory) -> clean
And the unmodified released build (lemonade-sdk b1318, commit 3173a56 - a stock prebuilt, NOT the PR build), WITHOUT the env var, serves all three quants correctly:
| Quant | Result | Gen t/s |
|---|---|---|
| Q2_K_XL | "The capital of France is Paris." (clean) | ~15.8 |
| Q3_K_XL | "The capital of France is Paris." (clean) | ~22.1 |
| Q4_K_XL | "The capital of France is Paris." (clean) | ~19.5 |
Also clean with Studio's FULL heavy flag set (262144 ctx, --parallel 4, --spec-default, --mmproj, --kv-unified) and multi-turn conversations (the "degenerates to ////" scenario).
Why Unsloth Studio was always broken
Studio's backend forces GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 in the child process on AMD unified-memory APUs. That is why Studio produced garbage with EVERY build pointed at it - including the PR #27941 build. It was Studio's forced env, not the build.
ggml checks the env var's presence, not value - so even GGML_CUDA_ENABLE_UNIFIED_MEMORY=0 still enables it. It must be ABSENT.
The real fix (simpler than the original post)
- No compile needed. Any llama.cpp build works.
- Ensure
GGML_CUDA_ENABLE_UNIFIED_MEMORYis NOT in the environment when serving qwen4exp. - In Unsloth Studio: set
UNSLOTH_DISABLE_UNIFIED_MEMORY=1(a supported opt-out) so Studio does not set the var.
Correction to the original post
- The "build from PR #27941 to fix it" advice is overstated / not required for Strix Halo at 32K context - old released builds work without the env var.
- The root cause is the env var, not a missing llama.cpp fix. I earlier conflated correlation with causation: the fixed build appeared to work because my clean tests happened to omit the env var; under Studio (which forces it) even the fixed build failed.
- The original post's exoneration of quant, vision-module, and the upstream bug references remain valid context, but the operative fix is the env var, not a compile.
This correction is the takeaway for a Strix Halo user.
Just swith to Vulkan, I run Q6_K_XL on the same hardware with the same speed (Ubuntu 26.04).
command:
LLAMA_MMAP_RANDOM=1 GGML_OP_OFFLOAD_MIN_BATCH=512 ./llama-server --model ".../models/unsloth/Qwen3.8-Flash-Next-GGUF-Q6/Qwen3.8-Flash-Next-UD-Q6_K_XL-00001-of-00006.gguf" --host 192.168.1.xx --port 8080 --tools get_info --api-key xxx --jinja --n-gpu-layers all --n-cpu-moe 2 --parallel 1 --no-kv-unified --kv-offload --load-mode mmap --spec-type none --fit off --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --ctx-size 196608 --batch-size 4096 --ubatch-size 2048 --no-op-offload --override-tensor "per_layer_token_embd=CPU" --lazy-mode on --override-kv qwen4exp.attention.indexer.top_k=int:4096 --temperature 0.8 --top-k 20 --repeat-penalty 1.05 --presence-penalty 0.05 --top-p 0.95 --min-p 0.05 --reasoning on --chat-template-kwargs '{"reasoning_effort":"xhigh"}' --reasoning-preserve
output:
2.10.851.434 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 1702, progress = 0.45, t = 8.68 s / 195.97 tokens per second
2.20.936.369 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 3742, progress = 1.00, t = 18.77 s / 199.35 tokens per second
2.21.523.782 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 3750, progress = 1.00, t = 19.36 s / 193.71 tokens per second
...
2.35.014.583 I slot launch_slot_: id 0 | task 140 | processing task, is_child = 0
2.40.954.126 I slot print_timing: id 0 | task 140 | n_gen = 100, tg = 20.16 t/s, tg_3s = 20.36 t/s
2.43.992.482 I slot print_timing: id 0 | task 140 | n_gen = 161, tg = 20.13 t/s, tg_3s = 20.08 t/s
2.47.011.149 I slot print_timing: id 0 | task 140 | n_gen = 222, tg = 20.15 t/s, tg_3s = 20.21 t/s
2.50.016.912 I slot print_timing: id 0 | task 140 | n_gen = 283, tg = 20.18 t/s, tg_3s = 20.29 t/s
2.53.032.881 I slot print_timing: id 0 | task 140 | n_gen = 344, tg = 20.19 t/s, tg_3s = 20.23 t/s
Q5_K_XL a little bit faster in tg, about 21t/s, and about 300t/s of pp (because it fully fits in memory while Q6_K_XL requires 2 layers to be offloaded)
p.s. no chance to load Q5 or Q6 on ROCm, I tried. Only Vulkan can do this with surprisenly good speed. Especially Q5 - it doesn't differ in pp/tg from Q4.
k-mktr report is gold. This PR #27941 turned the model from worse than Qwen3.5-122B into one of the best local models you can run! Very satisfied with the result.
I came to the same conclusion some time ago when using DeepSeek V4 Flash, entirely removing the GGML_CUDA_ENABLE_UNIFIED_MEMORY stopped the garbled generation.
However, I have one concern with this when running Qwen3.8-Flash-Next with GGML_CUDA_ENABLE_UNIFIED_MEMORY unset: the thermals of the Strix Halo reach ~98°C... When the flag is set, it generates garbage, but stays around 80-85°C, which is what I have when running DeepSeek-v4-Flash-Vision-Exp.
So:
| DeepSeek V4 Flash | Qwen3.8-Flash-Next | |
|---|---|---|
| GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON | Garbage output, 80°C | Garbage output, 80°C |
| No GGML_CUDA_ENABLE_UNIFIED_MEMORY set | Works, 80°C | Works, 100°C |
Anyone else seeing this?
Because it uses CPU to do some work now, so your temps go up.