NPO-SAM-WMDP-llama3-8b-instruct-GGUF

GGUF quantizations of OPTML-Group/NPO-SAM-WMDP-llama3-8b-instruct.

About the original model

• Architecture: LlamaForCausalLM (Llama 3 8B Instruct fine-tune)
• Produced by OPTML-Group using machine unlearning on the WMDP benchmark
• Method: NPO (Negative Preference Optimization) combined with SAM (Sharpness-Aware Minimization), aimed at making unlearning more robust to relearning attacks
• See the original model page for full details, training setup, and evaluation results

Quantizations

File	Bits	Approx. size	Use case
model_f16.gguf	16	~16.1 GB	Full precision, reference / requantization
model_q8_0.gguf	8	~8.5 GB	Near-lossless
model_q6_k.gguf	6	~6.6 GB	Very high quality
model_q5_k_m.gguf	5	~5.7 GB	High quality, recommended
model_q5_k_s.gguf	5	~5.6 GB	High quality, slightly smaller
model_q4_k_m.gguf	4	~4.9 GB	Best balance of size and quality
model_q4_k_s.gguf	4	~4.7 GB	Balanced, smaller
model_q3_k_l.gguf	3	~4.3 GB	Low memory, noticeable quality loss
model_q3_k_m.gguf	3	~4.0 GB	Low memory
model_q3_k_s.gguf	3	~3.7 GB	Very low memory
model_q2_k.gguf	2	~3.2 GB	Smallest, significant quality loss

Memory requirements (approximate, 4K context)

Quant	RAM / VRAM
F16	~18 GB
Q8_0	~10 GB
Q6_K	~8 GB
Q5_K_M / Q5_K_S	~7 GB
Q4_K_M / Q4_K_S	~6.5 GB
Q3_K_L / Q3_K_M / Q3_K_S	~5.5–6 GB
Q2_K	~5 GB

Usage

llama.cpp

./llama-cli -m model_q4_k_m.gguf -p "Hello" -n 256 -c 4096

With GPU offload:

./llama-cli -m model_q4_k_m.gguf -p "Hello" -n 256 -c 4096 -ngl 99

llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="tinyopsec/NPO-SAM-WMDP-llama3-8b-instruct-GGUF",
    filename="model_q4_k_m.gguf",
    n_ctx=4096,
)

out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello"}]
)
print(out["choices"][0]["message"]["content"])

LM Studio

1. Open the Discover tab and search for tinyopsec/NPO-SAM-WMDP-llama3-8b-instruct-GGUF.
2. Pick a quantization (Q4_K_M recommended) and download it.
3. Load the model in the Chat tab.

Ollama

huggingface-cli download tinyopsec/NPO-SAM-WMDP-llama3-8b-instruct-GGUF model_q4_k_m.gguf --local-dir .
echo "FROM ./model_q4_k_m.gguf" > Modelfile
ollama create npo-sam-wmdp-llama3-8b -f Modelfile
ollama run npo-sam-wmdp-llama3-8b

Links

• Original model: https://huggingface.co/OPTML-Group/NPO-SAM-WMDP-llama3-8b-instruct
• llama.cpp: https://github.com/ggerganov/llama.cpp

License

Inherited from the original model and the Llama 3 Community License. See the original model page for terms.