Instructions to use Anjielon/ODINO-397B-v34a-TQ1_0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Anjielon/ODINO-397B-v34a-TQ1_0 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Anjielon/ODINO-397B-v34a-TQ1_0 # Run inference directly in the terminal: llama cli -hf Anjielon/ODINO-397B-v34a-TQ1_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Anjielon/ODINO-397B-v34a-TQ1_0 # Run inference directly in the terminal: llama cli -hf Anjielon/ODINO-397B-v34a-TQ1_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Anjielon/ODINO-397B-v34a-TQ1_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf Anjielon/ODINO-397B-v34a-TQ1_0
Use Docker
docker model run hf.co/Anjielon/ODINO-397B-v34a-TQ1_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use Anjielon/ODINO-397B-v34a-TQ1_0 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Anjielon/ODINO-397B-v34a-TQ1_0" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Anjielon/ODINO-397B-v34a-TQ1_0", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Anjielon/ODINO-397B-v34a-TQ1_0
- Ollama
How to use Anjielon/ODINO-397B-v34a-TQ1_0 with Ollama:
ollama run hf.co/Anjielon/ODINO-397B-v34a-TQ1_0
- Unsloth Desktop
- Pi
How to use Anjielon/ODINO-397B-v34a-TQ1_0 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Anjielon/ODINO-397B-v34a-TQ1_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Anjielon/ODINO-397B-v34a-TQ1_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use Anjielon/ODINO-397B-v34a-TQ1_0 with Docker Model Runner:
docker model run hf.co/Anjielon/ODINO-397B-v34a-TQ1_0
- Lemonade
How to use Anjielon/ODINO-397B-v34a-TQ1_0 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Anjielon/ODINO-397B-v34a-TQ1_0
Run and chat with the model
lemonade run user.ODINO-397B-v34a-TQ1_0-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use Anjielon/ODINO-397B-v34a-TQ1_0 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Anjielon/ODINO-397B-v34a-TQ1_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default Anjielon/ODINO-397B-v34a-TQ1_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Anjielon/ODINO-397B-v34a-TQ1_0 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Anjielon/ODINO-397B-v34a-TQ1_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "Anjielon/ODINO-397B-v34a-TQ1_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
ODINO 397B v3.4a
Two-plane ternary TQ1_0 ยท 88.2 GiB ยท 1.911 bits/weight
a 397B-parameter, 512-expert MoE forged on a single desktop machine
llama.cpp Vulkan TQ1_0 kernels (PR #27765) ยท the forge (paper + tools)
The headline
| ODINO v3.4a | IQ1_M (llama.cpp) | |
|---|---|---|
| bits/weight | 1.911 | 1.844 |
| size | 88.2 GiB | 85.1 GiB |
| tasks solved (143 paired trials) | 112 โ 78.3% | 79 โ 55.2% |
+23.1 points on real coding-and-agentic work, discordant pairs 39โ6, exact McNemar p = 5.42e-07, paired-bootstrap CIโโ of the difference [+14.7, +31.5] points.
โ ๏ธ The sizes are not identical and we do not round them away: the IQ1_M reference is 3.5% smaller (the ternary file is 3.6% larger), so the gap above comes with a small size handicap against us, not in our favour.
โ ๏ธ How this file was served in the two evaluations (found while preparing
the revised paper, by reading the file's tensor table against the engine's
default). The file carries second-plane (_exps2) tensors on all 60
layers, and our llama.cpp fork applies a second plane wherever those tensors
exist unless ODINO_NO_P2=1 or ODINO_P2_LAYERS=LO-HI restricts it. The
public-benchmark table below was run with no environment variable, i.e.
with the second plane applied on all 60 layers. The task-suite numbers above
were served by our launcher script, which sets ODINO_NO_P2=1 (no second
plane at all); the per-run environment was not recorded, so this is what the
launcher sets, not a logged fact. Neither configuration is the tail-only
(layers 44-59) recipe of the paper's Section 5, whose evidence rests on
engine-switch measurements, not on these tables. The first plane is RTN on
layers 0-43 and GPTQ on layers 44-59 (transplanted), per the forge record.
The finding: perplexity cannot tell this pair apart โ the public benchmarks can
Nine public benchmark corpora run whole plus a declared 1,000-item PIQA subset, same engine (our llama.cpp fork, Vulkan), same GPU (NVIDIA RTX PRO 6000 Blackwell, 96 GiB), one model per GPU, reasoning off. Accuracy is llama.cpp's per-token-normalised score with a Wilson 95% interval; bold marks the three benchmarks whose intervals are disjoint. Because both files were scored on identical items in identical order, the paper also reports the paired per-item exact McNemar test from the raw logs: at the Bonferroni-corrected level (0.05/10) it separates four benchmarks โ the three in bold plus ARC-Easy (207-148 discordant, p = 0.002); Winogrande (247-189, p = 0.006) is nominally significant and just misses the corrected threshold; ARC-Challenge, OpenBookQA, PIQA and TruthfulQA MC1 are not separated at these sample sizes.
| benchmark | n | ODINO v3.4a (88.17 GiB) | IQ1_M (85.09 GiB) | ฮ |
|---|---|---|---|---|
| HellaSwag | 10,042 | 76.40 [75.56, 77.22] | 74.07 [73.20, 74.92] | +2.33 |
| MMLU | 14,042 | 82.71 [82.07, 83.33] | 80.58 [79.92, 81.23] | +2.13 |
| BoolQ | 3,270 | 89.27 [88.16, 90.28] | 81.28 [79.91, 82.58] | +7.99 |
| ARC-Easy | 2,376 | 79.00 [77.31, 80.59] | 76.52 [74.77, 78.18] | +2.48 |
| ARC-Challenge | 1,172 | 57.85 [55.00, 60.65] | 55.63 [52.77, 58.45] | +2.22 |
| PIQA (1,000-item subset of the 1,838-item validation split) | 1,000 | 80.90 [78.35, 83.22] | 80.90 [78.35, 83.22] | 0.00 |
| OpenBookQA | 500 | 44.40 [40.10, 48.78] | 43.60 [39.32, 47.98] | +0.80 |
| Winogrande | 1,267 | 66.22 [63.57, 68.77] | 61.64 [58.93, 64.28] | +4.58 |
| TruthfulQA MC1 | 817 | 38.43 [35.16, 41.82] | 36.72 [33.48, 40.08] | +1.71 |
| perplexity (WikiText-2, 72 chunks, ctx 4096) | 295k tok | 7.5180 ยฑ 0.0519 | 7.5173 ยฑ 0.0510 | +0.0007 |
ฮ = ours โ rival; for perplexity lower is better, so the +0.0007 there is nominally in the rival's favour (one-seventieth of its standard error, and a paired per-chunk difference of 0.01 standard errors). No corpus shows a negative difference beyond its interval; the ternary file wins three of them outright (BoolQ by eight points), and the two files are indistinguishable on perplexity โ the cheapest and most widely reported of these measurements is the only one that cannot see the difference that the private task suite above puts at +23.1 points.
The earlier home measurements, and a correction
The figure and the table below are the measurements first published with this
model, taken on the AMD desktop APU (gfx1151, Vulkan/RADV); the raw logs are
in logs/.
| ODINO v3.4a | IQ1_M | |
|---|---|---|
| perplexity (WikiText-2, 72 chunks, ctx 4096) | 7.5903 ยฑ 0.0525 | 7.5438 ยฑ 0.0512 |
| HellaSwag (same 2,108-item random subset, seed 1, drawn without replacement) | 75.85% | 73.86% |
| Winogrande (1,267 items, ยฑ one standard error) | 65.67 ยฑ 1.33 | 61.09 ยฑ 1.37 |
โ ๏ธ Correction. This page previously described the HellaSwag row as the
"same 2,108-item prefix". That was wrong: llama.cpp's HellaSwag scorer always
draws tasks at random without replacement (fixed seed 1), so the 2,108 items
are a random subset in random order, and the ODINO run had been launched for
4,000 tasks and stopped at 2,108 (see the selecting ... randomized tasks
lines in the logs). Both files were scored on the same 2,108 items, so the
comparison was paired โ but the sample is superseded by the whole-corpus
figures above. The perplexity row was also over-read: 7.5903 vs 7.5438 is
inside its own error bars, and on the second GPU the two files come out
identical (7.5180 vs 7.5173). The same items on the two GPUs agree within one
combined standard error (Winogrande 65.67 โ 66.22 and 61.09 โ 61.64;
perplexity 7.5903 โ 7.5180 and 7.5438 โ 7.5173). Paired on the same items the
picture is finer, and the paper reports it: 14.6% of Winogrande items change
outcome between the two GPUs on this file (balanced, McNemar p = 0.66), and
the perplexity offset on this file is systematic once paired per chunk (+0.96%
at home, t = 3.9 over 72 chunks) โ a declared backend difference that changes
no comparison, since both files are measured on the same GPU in the table
above.
Task results by family
Reasoning off, two fixed seeds (7, 42), identical server and judge for both
files. Frozen data: bench_v2_CONGELATO_20260829-1959.csv,
md5 66c0d923c2818bb306b578a5e10cedd0.
Files
| file | size | what it is |
|---|---|---|
ODINO-397B-v34a.gguf |
88.2 GiB | the model โ two-plane ternary TQ1_0, 1.911 bits/weight |
logs/*.log |
โ | the raw home runs (perplexity, HellaSwag 2,108-item subset, Winogrande) for this file and the IQ1_M baseline |
assets/*.png |
โ | the figures on this page |
hf download Anjielon/ODINO-397B-v34a-TQ1_0 ODINO-397B-v34a.gguf --local-dir .
Running it
The single-plane TQ1_0 Vulkan path is upstream: llama.cpp PR #27765 (merged 2026-09-07). The same shaders have been exercised on AMD gfx1151 (Radeon 8060S, where this model is served daily), on an NVIDIA RTX 4060 and on NVIDIA RTX PRO 6000 Blackwell (the benchmark table above) โ no vendor-specific changes. The quantizer itself is bit-exact across CPU and CUDA backends (identical packed payloads, |ฮ| = 0).
llama-server -m ODINO-397B-v34a.gguf -ngl 999 --ctx-size 4096 # ~88 GiB weights
On a 128 GB unified-memory APU the model runs fully resident. โ ๏ธ The
second ternary plane (the _exps2 tensors and their application) is not
upstream: on stock llama.cpp the file is rejected at load
(done_getting_tensors: wrong number of tensors; expected 1278, got 1098 โ
the 180 unrecognised _exps2 tensors; verified 2026-09-20 on an unmodified
upstream checkout). Only the single-plane TQ1_0 format is upstream; this
two-plane file needs the fork. The forge scripts are in
the forge repository; the two-plane
loader is not yet in any public repository โ the engine used for every
run is released with the revised paper's supplementary material as two diffs
with their SHA-256. The fork's git history was not preserved (the binary
reports commit unknown), so the upstream commit it was branched from was
identified afterwards by matching every file the fork does not touch against
upstream history: all of them are blob-identical to the tree of
521a64cd01979bb5b1a466152c576a9d809b068d (2026-08-20), and the fork changes
19 files relative to it (the TQ1_0 Vulkan kernels and shaders, the two-plane
loader in src/, an activation-dump facility, one test). The second diff is
against 8fe90e1fbfc065f17a0b233c9df239423cd24a75, the commit that merged
PR #27765. To rebuild: check out the base commit, apply the first diff,
cmake -B build -DGGML_VULKAN=ON. Runtime switches: ODINO_P2_LAYERS=LO-HI
restricts the second plane to a band, ODINO_NO_P2=1 disables it; with
neither set it is applied on every layer whose _exps2 tensors exist (all 60
in this file).
Reproducing the numbers on this page
Sources: HellaSwag klosax/hellaswag_text_data (hellaswag_val_full.txt,
llama.cpp get-hellaswag.sh); MMLU cais/mmlu config all, test split;
ARC allenai/ai2_arc, test splits; TruthfulQA MC1 truthfulqa/truthful_qa
config multiple_choice, validation split (817 items, lm-eval's fixed
six-shot preamble, up to 13 options per question); PIQA gimmaru/piqa validation โ a
1,000-item subset of the official 1,838-item ybisk/piqa validation split
(verified: every row and label is in the official split, no duplicates, not a
prefix; declared as a subset, not a whole corpus); OpenBookQA
allenai/openbookqa main, test;
BoolQ google/boolq, validation; Winogrande winogrande-debiased-eval.csv
(llama.cpp get-winogrande.sh);
WikiText-2 raw wiki.test.raw (llama.cpp get-wikitext-2.sh). The
multiple-choice sets are converted to llama.cpp's binary --multiple-choice
format by converti_multiple_choice.py (released with the paper's
supplementary material, with the SHA-256 of every converted file), zero-shot
except TruthfulQA MC1, with these prompt templates: MMLU The following are multiple choice questions (with answers) about {subject}.\n\n{question}\nA. โฆ\nB. โฆ\nC. โฆ\nD. โฆ\nAnswer:
scored over the letters A/B/C/D; ARC and PIQA Question: {q}\nAnswer: over
the option texts; OpenBookQA the bare stem; BoolQ {passage}\nQuestion: {q}?\nAnswer: over no/yes; TruthfulQA MC1 lm-eval's fixed six-example
QA preamble (tasks/truthfulqa/truthfulqa_mc1.yaml, copied verbatim),
prepended to every question, over the mc1_targets option strings. The
template decides the number (an earlier
BoolQ template without the scaffold put this file below chance).
# perplexity (the metric that cannot tell this pair apart)
llama-perplexity -m ODINO-397B-v34a.gguf -ngl 999 -c 4096 -b 2048 -f wiki.test.raw
# HellaSwag: every task of the file (random order, seed 1, without replacement)
llama-perplexity -m ODINO-397B-v34a.gguf -ngl 999 -c 4096 \
--hellaswag --hellaswag-tasks 100000 -f hellaswag_val_full.txt
# MMLU / ARC / PIQA / OpenBookQA / BoolQ: every task, in order.
# -np 8 is REQUIRED: ARC has 5-option items and with the default -np 4
# llama-perplexity stops there while reporting a result.
llama-perplexity -m ODINO-397B-v34a.gguf -ngl 999 -c 8192 -np 8 \
--multiple-choice --multiple-choice-tasks 100000 -f FILE.bin
# Winogrande: all 1,267 items
llama-perplexity -m ODINO-397B-v34a.gguf -ngl 999 -c 4096 \
--winogrande --winogrande-tasks 100000 -f winogrande-debiased-eval.csv
# TruthfulQA MC1: up to 13 options per question, six-shot preamble baked
# into FILE.bin; batch reduced to fit this file's compute buffers
llama-perplexity -m ODINO-397B-v34a.gguf -ngl 999 -c 3328 -np 13 -b 256 -ub 256 \
--multiple-choice --multiple-choice-tasks 100000 -f FILE.bin
The raw output of the original home runs (perplexity, HellaSwag 2,108 random,
Winogrande), for this file and for the IQ1_M baseline, is in
logs/. Numbers are llama.cpp's per-token-normalised
scores: comparable between these two files, not with lm-eval leaderboards.
Honesty of the measurement
- The task suite is private: aggregates are published, fixtures are not. Families built on the author's own home automation are excluded from every number on this page.
- A number without its raw log does not enter this page. The raw
perplexity / HellaSwag / Winogrande logs of the home runs for both files
are in
logs/in this repository; the whole-corpus runs were scored chunk by chunk (HellaSwag in 21 chunks of 500 tasks) and every raw log, including discarded ones, is kept and released with the revised paper's supplementary material. - Generative benchmarks are not on this page. Everything here is log-likelihood scoring of given choices; GSM8K, HumanEval and the like need a harness against a server and are deferred.
- Runs whose harness logged a transport fault (5xx, dropped connection, malformed response) are never scored as model failures: the run is discarded and re-executed.
- Four figures from an earlier night had been transcribed by hand while their logs sat in a temporary folder the system cleaned. They were re-measured from scratch rather than copied from memory โ the transcribed Winogrande said 64.80; the true value is 65.67.
What is NOT yet demonstrated
The forge has one measured but not-yet-spendable lever: a Hadamard rotation (incoherence processing in the sense of QuIP#/QuaRot/SpinQuant) over the full 256-weight TQ1_0 block that removes ~5% of reconstruction error at the same bit budget on this model's tensors (0/40 tensors worse), waiting on the decode-side kernel that undoes it. It is stated here so that the current file is not mistaken for the ceiling.
Citation
The accompanying paper is under review; its title will be added here when the review concludes.
@misc{odino2026,
title = {ODINO 397B v3.4a: a two-plane ternary TQ1\_0 quantization of
Ornith-1.5-397B (paper under review)},
author = {Anjielon},
year = {2026},
note = {Model card and forge: github.com/Anjielon/fucina}
}
- Downloads last month
- 50
We're not able to determine the quantization variants.
Model tree for Anjielon/ODINO-397B-v34a-TQ1_0
Base model
ornith-ai/Ornith-1.5-397B
