Instructions to use quimmedes/Agnes-3.0-Flash-XYZ-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use quimmedes/Agnes-3.0-Flash-XYZ-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M

Use Docker

docker model run hf.co/quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use quimmedes/Agnes-3.0-Flash-XYZ-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "quimmedes/Agnes-3.0-Flash-XYZ-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "quimmedes/Agnes-3.0-Flash-XYZ-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M

Ollama
How to use quimmedes/Agnes-3.0-Flash-XYZ-GGUF with Ollama:
```
ollama run hf.co/quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M
```
Unsloth Desktop

How to use quimmedes/Agnes-3.0-Flash-XYZ-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use quimmedes/Agnes-3.0-Flash-XYZ-GGUF with Docker Model Runner:
```
docker model run hf.co/quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M
```

Lemonade

How to use quimmedes/Agnes-3.0-Flash-XYZ-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Agnes-3.0-Flash-XYZ-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use quimmedes/Agnes-3.0-Flash-XYZ-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use quimmedes/Agnes-3.0-Flash-XYZ-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "quimmedes/Agnes-3.0-Flash-XYZ-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Agnes-3.0-Flash Preview - GGUF quantizations

GGUF quantizations of Agnes-AI/Agnes-3.0-Flash (the open-weight Preview checkpoint, Apache-2.0), quantized with imatrix-weighted llama-quantize.

These files need a patched build. Agnes has an extra parallel SwiGLU branch in every layer (ffn_gate_par / ffn_up_par / ffn_down_par) that stock llama.cpp does not know about. The fork cafe-llama.cpp adds those tensors to the qwen35 architecture (plus the recurrent/full attention layer map and the delta-attention V-head reorder). Loading these GGUFs on an unpatched build will fail or mis-load.

Main ladder

Oficial llama.cpp is not supporting this yet, you have to use cafe-llama.cpp https://github.com/quimmedes/cafe-llama.cpp

file	size on disk	answer when served	uploaded
`Agnes-3.0-Flash-Q3-v4-XYZ.gguf`	12.12 GiB	The capital of France is Paris.	yes
`Agnes-3.0-Flash-Q3.5-v4-XYZ.gguf`	15.37 GiB	The capital of France is Paris.	yes
`Agnes-3.0-Flash-Q4-v4-XYZ.gguf`	16.57 GiB	The capital of France is Paris.	yes
`Agnes-3.0-Flash-Q4.5-v4-XYZ.gguf`	19.59 GiB	The capital of France is Paris.	yes
`Agnes-3.0-Flash-Q5-v4-XYZ.gguf`	20.15 GiB	The capital of France is Paris.	yes
`Agnes-3.0-Flash-Q5.5-v4-XYZ.gguf`	20.59 GiB	The capital of France is Paris.	yes
`Agnes-3.0-Flash-Q6-v4-XYZ.gguf`	24.32 GiB	The capital of France is Paris.	yes
`Agnes-3.0-Flash-Q7-v4-XYZ.gguf`	24.60 GiB	The capital of France is Paris.	yes
`Agnes-3.0-Flash-Q8-v4-XYZ.gguf`	27.32 GiB	The capital of France is Paris.	yes
`Agnes-3.0-Flash-Q9-v4-XYZ.gguf`	32.28 GiB	The capital of France is Paris.	yes

Total: 212.9 GiB - all of it uploaded.

Other quantizations (`extra/`)

file	size on disk	answer when served
`extra/Agnes-3.0-Flash-Q3-XYZ.gguf`	18.00 GiB	The capital of France is Paris.
`extra/Agnes-3.0-Flash-Q4.5-XYZ.gguf`	17.37 GiB	The capital of France is Paris.
`extra/Agnes-3.0-Flash-Q5-XYZ.gguf`	19.52 GiB	The capital of France is Paris.
`extra/Agnes-3.0-Flash-Q8-XYZ.gguf`	31.21 GiB	The capital of France is Paris.
`extra/Agnes-3.0-Flash-Q4_K_M-control.gguf`	19.07 GiB	The capital of France is Paris.

Total: 105.2 GiB. Same model and same runtime as the ladder above; different tensor-type allocation (more precision kept in attention, more compression in the FFN).

How to run

# 24 GB card, long context, MTP speculative decoding
llama-server -m Agnes-3.0-Flash-Q4.5-v4-XYZ.gguf \
  --host 127.0.0.1 --port 8080 -ngl 20 -nr -c 32768 -np 1 \
  -ctk q8_0 -ctv q8_0 -fa on --spec-type draft-mtp

-nr (no repack) keeps weights memory-mapped, which matters at 33B.
On a 24 GB card -ngl around 20 fits; a full offload only leaves room for a small context.
The MTP block is included (nextn, 1 layer), so --spec-type draft-mtp works.
Base card's recommended sampling: temp 1.0, top_p 0.95, top_k 20.

What this checkpoint actually is (findings, with the method)

While converting this model I compared it against Qwen/Qwen3.8-27B, because both report the same architecture (qwen35). The bytes say the two are the same blueprint. Method: sha256 over HTTPS for the small files, and byte-comparison of tensor ranges (safetensors data_offsets header, Range: requests) for the weights.

Byte-identical:

item	result
`vocab.json`	sha256 `ce99b4cb2983` == Qwen3.8-27B
`merges.txt`	sha256 `a9d356d7bdf1` == Qwen3.8-27B
`chat_template.jinja`	identical (same tool-call format, same `reasoning_effort` levels)
`generation_config.json`	identical
architecture hyper-parameters	hidden 5120, 24 q / 4 kv heads, head_dim 256, FFN 17408, delta-rule 16 key / 48 value heads, conv kernel 4, `partial_rotary_factor` 0.25, mrope `[11,11,10]` interleaved, theta 1e7, vocab 248320, MTP 1 layer, swish output gate, `tie_word_embeddings=false`
vision tower	same config (only the `model_type` label differs: `agnes_vision` vs `qwen3_5`), same weights (compared `attn.proj` of block 0: 2,654,208 bytes equal), same image/video token ids
`embed_tokens.weight`	identical (248320 x 5120)
norms	`input_layernorm`, `post_attention_layernorm`, `q_norm`, `k_norm`, final `norm` identical
SSM constants	`A_log`, `dt_bias`, `conv1d`, `ssm_norm` identical

Different:

item	Agnes	Qwen3.8-27B
layers	72 (54 delta-rule + 18 global, 3:1)	64 (48 linear + 16 full)
parallel FFN	SwiGLU 2048 in every layer	absent
all projection matrices	retrained	-

The parameter count agrees: 27B x 72/64 = 30.4B, plus the parallel branch (3 x 2048 x 5120 x 72 ~ 2.3B) = ~32.6B, and this GGUF measures 32.63B. Nothing is unaccounted for.

So: Agnes-3.0-Flash Preview carries the Qwen3.8-27B blueprint - same tokenizer, same chat template, same vision tower (identical weights), same embeddings/norms/SSM constants - with 8 extra delta layers, a new parallel-FFN branch, and retrained projections. Both models are Apache-2.0, so this is entirely legitimate; the reason to write it down is that people know what they are downloading.

How these were made

Safetensors -> GGUF with the fork's converter. Changes needed for this model: register the parallel-FFN tensors, record the delta-attention layers as recurrent (otherwise every layer looks like full attention and attn_qkv fails), and map .delta_attn./.global_attn. onto the standard qwen35 names.
Imatrix built from 50 chunks of the wikitext-2 raw test split; the bf16 GGUF scores PPL 6.5737 +/- 0.14183 with it.
Quantized with llama-quantize --imatrix, per-tensor types: attention and SSM tensors stay at higher precision, the FFN carries the compression. The parallel FFN follows the FFN of its block.

Practical consequences

It is dense, not a MoE. No experts: 0 of the 1521 checkpoint tensors contain "expert", and the config has no MoE keys (intermediate_size 17408 plus a parallel 2048 SwiGLU). All ~33B parameters are read for every token, so expect dense-model speed at equal quantization, not "3B active" speed. The base card's benchmark table compares against models labelled "X active"; Agnes is not one.
Long context is comparatively cheap. Only 18 of the 72 layers hold a KV cache that grows with context (the 54 delta-rule layers keep a fixed-size recurrent state). At 128k with q8_0 KV that is about 5 GB (18 x 2 x 4 KV heads x 256 dim x 131072 x 1.06 B), which is why a 24 GB card can hold the Q4.5 weights and a long context.
Vision is not in these files. The text tower and the MTP layer are; the projector is separate. Because the vision tower weights are byte-identical to Qwen3.8-27B's, the mmproj-Qwen3.8-27B-bf16.gguf from quimmedes/Qwen3.8-27B-XYZ is expected to work - but a full image encode could not be run on this machine (the vision graph needed more RAM than was free), so treat that as unverified.
These are quantizations of a preview checkpoint; the production Agnes model has a different configuration and its benchmarks do not apply here.

Credits

Base model: Agnes-AI/Agnes-3.0-Flash (Apache-2.0).
Runtime: quimmedes/cafe-llama.cpp - required for the parallel-FFN tensors.

Downloads last month: 5,101

GGUF

Model size

33B params

Architecture

qwen35

Hardware compatibility

4-bit

View +13 variants

Model tree for quimmedes/Agnes-3.0-Flash-XYZ-GGUF

Base model

Agnes-AI/Agnes-3.0-Flash

Quantized

(20)

this model

Agnes-3.0-Flash Preview - GGUF quantizations

Main ladder

Other quantizations (extra/)

How to run

What this checkpoint actually is (findings, with the method)

How these were made

Practical consequences

Credits

Model tree for quimmedes/Agnes-3.0-Flash-XYZ-GGUF

Other quantizations (`extra/`)