How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf geoffmunn/Qwen3Guard-Gen-8B-GGUF:
# Run inference directly in the terminal:
llama cli -hf geoffmunn/Qwen3Guard-Gen-8B-GGUF:
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf geoffmunn/Qwen3Guard-Gen-8B-GGUF:
# Run inference directly in the terminal:
llama cli -hf geoffmunn/Qwen3Guard-Gen-8B-GGUF:
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf geoffmunn/Qwen3Guard-Gen-8B-GGUF:
# Run inference directly in the terminal:
./build/bin/llama cli -hf geoffmunn/Qwen3Guard-Gen-8B-GGUF:
Use Docker
docker model run hf.co/geoffmunn/Qwen3Guard-Gen-8B-GGUF:
Quick Links

Qwen3Guard-Gen-8B-GGUF

This is a GGUF-quantized version of Qwen3Guard-Gen-8B, a safety-aligned generative model from Alibaba's Qwen team, built on the 8-billion-parameter Qwen3 architecture.

Unlike standard LLMs, this model is fine-tuned to refuse harmful requests by design, making it ideal for applications where content safety and advanced reasoning are both critical.

โš ๏ธ This is a generative model with built-in safety constraints, not a classifier like Qwen3Guard-Stream-4B.

๐Ÿ›ก What Is Qwen3Guard-Gen-8B?

Itโ€™s a helpful yet harmless assistant trained to:

  • Respond helpfully to complex queries (math, code, logic)
  • Politely decline unsafe ones (e.g., illegal acts, self-harm)
  • Avoid generating toxic, violent, or deceptive content
  • Maintain factual consistency while being cautious

Perfect for:

  • Educational assistants with deep reasoning
  • Customer service agents handling sensitive topics
  • Mental wellness chatbots with nuanced understanding
  • Moderated community bots requiring high-quality output

๐Ÿ”— Relationship to Other Safety Models

This model complements other Qwen3 safety tools:

Model Role Best For
Qwen3Guard-Stream-4B โšก Input filter Real-time moderation of user input
Qwen3Guard-Gen-4B ๐Ÿง  Safe generator (smaller) Lightweight safe generation
Qwen3Guard-Gen-8B ๐Ÿ’ช Stronger safe generator High-quality, safe responses with deep reasoning
Qwen3-4B-SafeRL ๐Ÿ›ก๏ธ Fully aligned agent Ethical multi-turn dialogue

Recommended Architecture

User Input
    โ†“
[Optional: Qwen3Guard-Stream-4B] โ† optional pre-filter
    โ†“
[Qwen3Guard-Gen-8B]
    โ†“
Safe, High-Quality Response

You can run this model standalone or behind a streaming guard for defense-in-depth.

Available Quantizations

These variants were built from a f16 base model to ensure consistency across quant levels.

Level Quality Speed Size Recommendation
Q2_K Very Low โšก Fastest ~3.7 GB Only on severely memory-constrained systems (<6GB RAM). Avoid for reasoning.
Q3_K_S Low โšก Fast ~4.3 GB Minimal viability; basic completion only. Not recommended.
Q3_K_M Low-Medium โšก Fast ~4.9 GB Acceptable for simple chat on older systems. No complex logic.
Q4_K_S Medium ๐Ÿš€ Fast ~5.6 GB Good balance for low-end laptops or embedded platforms.
Q4_K_M โœ… Balanced ๐Ÿš€ Fast ~6.2 GB Best overall for general use on average hardware. Great speed/quality trade-off.
Q5_K_S High ๐Ÿข Medium ~6.1 GB Better reasoning; slightly faster than Q5_K_M. Ideal for coding.
Q5_K_M โœ…โœ… High ๐Ÿข Medium ~6.2 GB Top pick for deep interactions, logic, and tool use. Recommended for desktops.
Q6_K ๐Ÿ”ฅ Near-FP16 ๐ŸŒ Slow ~7.2 GB Excellent fidelity; ideal for RAG, retrieval, and accuracy-critical tasks.
Q8_0 ๐Ÿ† Lossless* ๐ŸŒ Slow ~9.8 GB Maximum accuracy; best for research, benchmarking, or archival.

๐Ÿ’ก Recommendations by Use Case

  • ๐Ÿ’ป Low-end CPU / Old Laptop: Q4_K_M (best balance under pressure)
  • ๐Ÿ–ฅ๏ธ Standard/Mid-tier Laptop (i5/i7/M1/M2): Q5_K_M (optimal quality)
  • ๐Ÿง  Reasoning, Coding, Math: Q5_K_M or Q6_K (use thinking mode!)
  • ๐Ÿค– Agent & Tool Integration: Q5_K_M โ€” handles JSON, function calls well
  • ๐Ÿ” RAG, Retrieval, Precision Tasks: Q6_K or Q8_0
  • ๐Ÿ“ฆ Storage-Constrained Devices: Q4_K_S or Q4_K_M
  • ๐Ÿ› ๏ธ Development & Testing: Test from Q4_K_M up to Q8_0 to assess trade-offs

Tools That Support It

  • LM Studio โ€“ load and test locally
  • OpenWebUI โ€“ deploy with RAG and tools
  • GPT4All โ€“ private, offline AI
  • Directly via llama.cpp, Ollama, or TGI

Author

๐Ÿ‘ค Geoff Munn (@geoffmunn)
๐Ÿ”— Hugging Face Profile

Disclaimer

Community conversion for local inference. Not affiliated with Alibaba Cloud.

Downloads last month
110
GGUF
Model size
8B params
Architecture
qwen3
Hardware compatibility
Log In to add your hardware

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for geoffmunn/Qwen3Guard-Gen-8B-GGUF

Finetuned
Qwen/Qwen3-8B
Quantized
(14)
this model