sizzlebop/Llama-3.2-3B-TechWriter-Instruct-GGUF

Llama-3.2-3B TechWriter Instruct GGUF

This repository provides GGUF quantizations for Shankarblr/Llama-3.2-3B-TechWriter-Instruct.

Llama-3.2-3B TechWriter is a fine-tune of meta-llama/Llama-3.2-3B-Instruct specialized for semiconductor and data-center interconnect technical documentation, product briefs, datasheets, application notes, and CLI user guides.

All GGUF files were converted from the original safetensors weights using llama.cpp at native F16 precision, followed by quantization into standard k-quant variants.

---

Available Files and Quantizations

File	Quant Type	Size	Description / Recommendation
Llama-3.2-3B-TechWriter-Instruct-F16.gguf	F16	5.99 GB	Full precision base conversion. Highest fidelity reference weights.
Llama-3.2-3B-TechWriter-Instruct-Q8_0.gguf	Q8_0	3.19 GB	Near-lossless 8-bit quantization. Recommended for best quality.
Llama-3.2-3B-TechWriter-Instruct-Q6_K.gguf	Q6_K	2.46 GB	High quality retention with minimal degradation. Excellent balance.
Llama-3.2-3B-TechWriter-Instruct-Q5_K_M.gguf	Q5_K_M	2.16 GB	Solid balance between memory usage and generation accuracy.
Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf	Q4_K_M	1.88 GB	Fast, lightweight 4-bit quant. Recommended default for mobile and edge setups.
Llama-3.2-3B-TechWriter-Instruct-Q3_K_M.gguf	Q3_K_M	1.57 GB	Compact footprint when memory headroom is strictly limited.
Llama-3.2-3B-TechWriter-Instruct-Q2_K.gguf	Q2_K	1.27 GB	Maximum compression for ultra-constrained environments.

---

Domain Capabilities

This model is fine-tuned to produce structured, consistent technical documentation:

• Product Briefs & Datasheets: Feature summaries, pinout descriptions, electrical specs, and register descriptions.
• Hardware & CLI Documentation: Host-adapter (HBA/NIC), switch, and DPU command-line user manuals.
• Architecture Notes: Process node, throughput, bus width, and protocol stack consistency.
• Spec Extraction: Grounded extraction and QA from raw hardware documentation excerpts.

---

Prompt Format

This model uses the standard Llama 3.2 Instruct template format:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a technical marketing and documentation writer for semiconductor and data-center interconnect products. Write clear, structured content. Match the requested document type. Keep specifications internally consistent: one process node, one primary throughput, and one form factor unless the source explicitly lists options.<|eot_id|><|start_header_id|>user<|end_header_id|>

Write a short feature overview for a 400G PCIe Gen5 NIC.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

---

How to Use

1. With llama.cpp

Run interactive or single-turn generation with llama-cli:

llama-cli -m ./Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf \
    -p "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful technical writer.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWrite a short paragraph explaining what an API endpoint is.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n" \
    -n 256 --temp 0.2

Launch a local OpenAI-compatible HTTP server:

llama-server -m ./Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf \
    --host 127.0.0.1 --port 8080 -c 4096

2. With Ollama

Create a Modelfile:

FROM ./Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""

PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER temperature 0.3

Then create and run the model:

ollama create techwriter-3b -f Modelfile
ollama run techwriter-3b

3. With LM Studio

1. Copy the desired .gguf file (e.g., Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf) into your LM Studio models directory.
2. Select Llama 3 as the chat preset.
3. Load the model and begin prompting.