Instructions to use batiai/gemma-4-E2B-it-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use batiai/gemma-4-E2B-it-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf batiai/gemma-4-E2B-it-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf batiai/gemma-4-E2B-it-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf batiai/gemma-4-E2B-it-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf batiai/gemma-4-E2B-it-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf batiai/gemma-4-E2B-it-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf batiai/gemma-4-E2B-it-GGUF:Q4_K_M
Use Docker
docker model run hf.co/batiai/gemma-4-E2B-it-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use batiai/gemma-4-E2B-it-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "batiai/gemma-4-E2B-it-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "batiai/gemma-4-E2B-it-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/batiai/gemma-4-E2B-it-GGUF:Q4_K_M
- Ollama
How to use batiai/gemma-4-E2B-it-GGUF with Ollama:
ollama run hf.co/batiai/gemma-4-E2B-it-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use batiai/gemma-4-E2B-it-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf batiai/gemma-4-E2B-it-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "batiai/gemma-4-E2B-it-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use batiai/gemma-4-E2B-it-GGUF with Docker Model Runner:
docker model run hf.co/batiai/gemma-4-E2B-it-GGUF:Q4_K_M
- Lemonade
How to use batiai/gemma-4-E2B-it-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull batiai/gemma-4-E2B-it-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.gemma-4-E2B-it-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use batiai/gemma-4-E2B-it-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf batiai/gemma-4-E2B-it-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default batiai/gemma-4-E2B-it-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use batiai/gemma-4-E2B-it-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf batiai/gemma-4-E2B-it-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "batiai/gemma-4-E2B-it-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Gemma 4 E2B-it GGUF β Quantized by BatiAI
Optimized GGUF quantizations of google/gemma-4-E2B-it β Google's tiniest fully-multimodal Gemma 4 (text + image + audio) for on-device AI on Mac. Built directly from official Google BF16 weights by BatiAI for BatiFlow.
The "E" stands for Edge. Tiny enough to run on basically any Mac (8 GB+) while still understanding speech and images alongside text β the only Gemma 4 size with audio support shared with E4B.
Quick Start
# Recommended
ollama pull batiai/gemma4-e2b:q4
# Higher quality
ollama pull batiai/gemma4-e2b:q6
Available Quantizations
| Tag | Quant | Size | Recommended For |
|---|---|---|---|
:q4 |
Q4_K_M | ~2.1 GB | balanced (recommended default) |
:q6 |
Q6_K | ~2.8 GB | higher quality, near-lossless |
Sub-3 B models stay sharp at Q4 β quantization noise is tiny relative to model capacity.
Two modes β text-only by default, multimodal opt-in
Upstream Gemma 4 E2B-it is fully multimodal β text + image + audio in one model. In the GGUF ecosystem this is delivered as two files: a main model.gguf (text tower) and a separate mmproj.gguf that holds both vision and audio encoders together (a single 1411-tensor projector covering image and speech input).
| Text-only (default) | Multimodal (opt-in) | |
|---|---|---|
| Files needed | main GGUF only | main GGUF + mmproj-BF16.gguf |
| Capabilities | Q&A, coding, tool calling, agents | + image (OCR, captioning, visual reasoning) + audio (speech understanding) |
ollama pull |
β single command | β Ollama mmproj integration is still rough β use llama.cpp directly |
| Disk / RAM | smaller (no projector weights) | larger (+ ~940 MB) |
| Recommended for | most users (chat, code, agents) | OCR, image / speech understanding |
Multimodal usage (llama.cpp)
# Pick a main model (text tower)
wget https://huggingface.co/batiai/Gemma-4-E2B-it-GGUF/resolve/main/google-gemma-4-E2B-it-Q4_K_M.gguf
# Get the multimodal projector (vision + audio in one file)
wget https://huggingface.co/batiai/Gemma-4-E2B-it-GGUF/resolve/main/mmproj-BF16.gguf
Server mode (image input via OpenAI-compatible Vision API):
llama-server \
-m google-gemma-4-E2B-it-Q4_K_M.gguf \
--mmproj mmproj-BF16.gguf \
-c 32768 --host 127.0.0.1 --port 8080
curl http://127.0.0.1:8080/v1/chat/completions -d '{
"messages": [{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
{"type": "text", "text": "What does this show?"}
]
}]
}'
One-shot CLI with image:
llama-mtmd-cli \
-m google-gemma-4-E2B-it-Q4_K_M.gguf \
--mmproj mmproj-BF16.gguf \
--image ~/Desktop/photo.jpg \
-p "describe this image"
One-shot CLI with audio:
llama-mtmd-cli \
-m google-gemma-4-E2B-it-Q4_K_M.gguf \
--mmproj mmproj-BF16.gguf \
--audio ~/Downloads/voice.wav \
-p "transcribe and summarize"
mmproj quantization
| File | Quant | Size | Why this only? |
|---|---|---|---|
mmproj-BF16.gguf |
BF16 | ~940 MB | The vision + audio projector tensors don't satisfy K-quant block-alignment requirements (108+ tensors require fallback), so Q6_K reliably aborts on this combined projector. BF16 is the only safe choice today β applies to every quantizer of this model. The main text GGUF is unaffected. |
Why E series (E2B / E4B) vs 26B / 31B?
| E2B / E4B | 26B-A4B / 31B | |
|---|---|---|
| Audio support | β | β (vision only) |
| Min RAM | 8 GB+ | 24 GB+ |
| Speed | very fast (small) | slower (larger) |
| Reasoning depth | lower | higher |
| Use case | edge / mobile-class Mac / voice + image | desktop chat + agents |
If you need voice + image + text in one model on a small Mac, E series is the only Gemma 4 option.
Why BatiAI?
| BatiAI | Third-party (unsloth, etc.) | |
|---|---|---|
| Source | Quantized directly from official Google weights | Often re-quantized from other GGUFs |
| Compatibility | β Verified on Ollama 0.20+ | β Known issues with Ollama 0.20+ |
| Tested on | Real Mac mini M4 (16 GB) + MacBook Pro M4 Max | Often untested |
| Tool calling | β Verified with BatiFlow's 57 tool functions | Often broken |
| Korean | β Validated | Not tested |
| Multimodal | β Vision + audio mmproj available | Often missing |
| Signing | general.author: BatiAI for provenance |
Varies |
About BatiFlow
BatiFlow is a macOS-native AI desktop automation app β just 5 MB, built with Swift.
- Free & Unlimited β On-device AI via Ollama, no API costs
- 100 % Private β All data stays on your Mac
- Ultra Lightweight β Native macOS app, only 5 MB
- 57 built-in tools β calendar, notes, reminders, files, email, browser, messaging, and more
Related models in the BatiAI Gemma 4 lineup
| Model | Modalities | Min RAM | Repo |
|---|---|---|---|
| Gemma 4 E2B-it | text + image + audio | 8 GB | this repo |
| Gemma 4 E4B-it | text + image + audio | 8 GB | batiai/Gemma-4-E4B-it-GGUF |
| Gemma 4 26B-A4B-it | text + image / video (no audio) | 24 GB | batiai/Gemma-4-26B-A4B-it-GGUF |
| Gemma 4 31B-it | text + image / video (no audio) | 24 GB | batiai/Gemma-4-31B-it-GGUF |
Technical Details
- Original Model: google/gemma-4-E2B-it
- Architecture: Gemma 4 (Edge variant)
- Modalities: Text (primary) + Image + Audio (via opt-in mmproj)
- License: Gemma (commercial use permitted under terms)
- Quantized with: llama.cpp
- Quantized by: BatiAI
- GGUF metadata:
general.author: BatiAI,general.url: https://flow.bati.ai
License
Mirrors the upstream Gemma license. Commercial use permitted per Google's Gemma terms.
BatiAI's quantization pipeline is provided under MIT.
- Downloads last month
- 391
4-bit
6-bit