Instructions to use Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved") model = AutoModelForCausalLM.from_pretrained("Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved
- SGLang
How to use Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved with Docker Model Runner:
docker model run hf.co/Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved
Download README.md from Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved: direct link, hf CLI and curl.
- Browser
- Download file 7.9 kB
-
https://huggingface.co/Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved/resolve/main/README.md
- Command line
-
hf download hf://Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved/README.md
-
curl -L -o README.md https://huggingface.co/Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved/resolve/main/README.md
license: apache-2.0
library_name: transformers
tags:
- mergekit
- merge
base_model:
- openchat/openchat-3.5-0106
model-index:
- name: OpenChat-3.5-0106_8.99B_40Layers-Interleaved
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 59.76
name: strict accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 24.08
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 6.8
name: exact match
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 7.27
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 11.51
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 25.54
name: accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Interleaved
name: Open LLM Leaderboard
Buy me a Ko-Fi • Support my work using Patreon
OpenChat-3.5-0106_8.99B_40Layers-Interleaved
This is NOT your usual frankenmerge created using mergekit.
Merge Details
Merge Method
This model was merged using the passthrough merge method, but employing the Block Expansion method described in the paper LLaMA Pro: Progressive LLaMA with Block Expansion.
The authors of the paper added new layers interleaved in between the original layers of the model, setting the parameters of the o_proj and down_proj layers to zero. This effectively adds layers that will just output their input (as if they were "transparent") allowing the model to remain functional even without further training. These new layers can then be targeted during training or fine-tuning without risking catastrophic forgetting, if you follow the author's training method to freeze the original layers and only train the new layers.
This model has not yet received additional training, so it should perform close to the original model.
Models Merged
The following models were included in the merge:
Configuration
The following YAML configuration was used to produce this model:
slices:
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [0, 4]
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [3, 4]
parameters:
scale:
- filter: o_proj
value: 0.0
- filter: down_proj
value: 0.0
- value: 1.0
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [4, 8]
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [7, 8]
parameters:
scale:
- filter: o_proj
value: 0.0
- filter: down_proj
value: 0.0
- value: 1.0
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [8, 12]
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [11, 12]
parameters:
scale:
- filter: o_proj
value: 0.0
- filter: down_proj
value: 0.0
- value: 1.0
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [12, 16]
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [15, 16]
parameters:
scale:
- filter: o_proj
value: 0.0
- filter: down_proj
value: 0.0
- value: 1.0
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [16, 20]
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [19, 20]
parameters:
scale:
- filter: o_proj
value: 0.0
- filter: down_proj
value: 0.0
- value: 1.0
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [20, 24]
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [23, 24]
parameters:
scale:
- filter: o_proj
value: 0.0
- filter: down_proj
value: 0.0
- value: 1.0
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [24, 28]
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [27, 28]
parameters:
scale:
- filter: o_proj
value: 0.0
- filter: down_proj
value: 0.0
- value: 1.0
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [28, 32]
- sources:
- model: openchat/openchat-3.5-0106
layer_range: [31, 32]
parameters:
scale:
- filter: o_proj
value: 0.0
- filter: down_proj
value: 0.0
- value: 1.0
merge_method: passthrough
dtype: bfloat16
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 22.49 |
| IFEval (0-Shot) | 59.76 |
| BBH (3-Shot) | 24.08 |
| MATH Lvl 5 (4-Shot) | 6.80 |
| GPQA (0-shot) | 7.27 |
| MuSR (0-shot) | 11.51 |
| MMLU-PRO (5-shot) | 25.54 |
Citation
@misc{wu2024llamaproprogressivellama,
title={LLaMA Pro: Progressive LLaMA with Block Expansion},
author={Chengyue Wu and Yukang Gan and Yixiao Ge and Zeyu Lu and Jiahao Wang and Ye Feng and Ying Shan and Ping Luo},
year={2024},
eprint={2401.02415},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2401.02415},
}