Instructions to use arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2:IQ4_XS_G # Run inference directly in the terminal: llama cli -hf arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2:IQ4_XS_G
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2:IQ4_XS_G # Run inference directly in the terminal: llama cli -hf arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2:IQ4_XS_G
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2:IQ4_XS_G # Run inference directly in the terminal: ./build/bin/llama cli -hf arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2:IQ4_XS_G
Use Docker
docker model run hf.co/arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2:IQ4_XS_G
- Llama new
- LM Studio
- Jan
- vLLM
How to use arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2:IQ4_XS_G
- Ollama
How to use arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2 with Ollama:
ollama run hf.co/arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2:IQ4_XS_G
- Unsloth Desktop
- Docker Model Runner
How to use arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2 with Docker Model Runner:
docker model run hf.co/arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2:IQ4_XS_G
- Lemonade
How to use arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull arnabsanyal/LazyWeights-Mistral-7B-Instruct-v0.2:IQ4_XS_G
Run and chat with the model
lemonade run user.LazyWeights-Mistral-7B-Instruct-v0.2-IQ4_XS_G
List all available models
lemonade list
- Atomic Chat
Add project page link and library name
This PR adds a link to the project page in the model card header and sets library_name: llama.cpp, since the repository contains GGUF files and the GitHub README shows usage with llama-cli. The existing arXiv link is kept.
refs/pr/1 ref Thanks @nielsr , merged! I've also made the same change in the script that generates these cards, so the project page link now sits between Paper and Code on all three model cards and the collection card, and library_name: llama.cpp is set on all three models.