Instructions to use mattshumer/ref_70_e3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mattshumer/ref_70_e3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mattshumer/ref_70_e3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mattshumer/ref_70_e3")
model = AutoModelForCausalLM.from_pretrained("mattshumer/ref_70_e3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use mattshumer/ref_70_e3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mattshumer/ref_70_e3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/ref_70_e3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mattshumer/ref_70_e3

SGLang

How to use mattshumer/ref_70_e3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mattshumer/ref_70_e3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/ref_70_e3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mattshumer/ref_70_e3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/ref_70_e3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mattshumer/ref_70_e3 with Docker Model Runner:
```
docker model run hf.co/mattshumer/ref_70_e3
```

🚩 Report: Legal issue(s)

by RDson - opened Sep 9, 2024

Discussion

RDson

Sep 9, 2024

Advertises the models performance as its own when they used another model to perform the evaluation. Multiple sources have stated that the model never perform close to advertised. This is a legal issue because they are using other models, such as Claude and ChatGPT, without their right.

nisten

Sep 9, 2024

This is not true, I could not reproduce anything you said on that twitter thread.

Also a legal issue of open weights from a 3rd party routing api is ridiculous.

RDson

Sep 9, 2024

•

edited Sep 9, 2024

That is not the issue. I am talking specifically about these scores which are NOT made by the same models uploaded on Huggingface. See this post: https://x.com/ArtificialAnlys/status/1832965630472995220

Either they are lying about the scores and did not use their own weights , or they are lying about the uploaded weight and have some "secret" weights they are not releasing. Distribution or promoting another business's work as your own to promote a business that you are invested in (https://www.linkedin.com/posts/mattshumer_build-and-improve-custom-models-for-your-activity-7211717630703865856-wZ9a) is probably not what Huggingface want on their platform.

jukofyork

Sep 9, 2024

This is not true, I could not reproduce anything you said on that twitter thread.

Also a legal issue of open weights from a 3rd party routing api is ridiculous.

They changed it as soon as it was detected.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment