Text Generation
MLX
English
sol_milkshake
causal-lm
decoder-only
small-language-model
recurrent-depth
ngpt
research
Instructions to use solintellegence/Sol-Milkshake with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use solintellegence/Sol-Milkshake with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # if on a CUDA device, also pip install mlx[cuda] # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("solintellegence/Sol-Milkshake") prompt = "Once upon a time in" text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- MLX LM
How to use solintellegence/Sol-Milkshake with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Generate some text mlx_lm.generate --model "solintellegence/Sol-Milkshake" --prompt "Once upon a time"
- Atomic Chat
Rewrite model card in plain language
Browse files
README.md
CHANGED
|
@@ -17,11 +17,11 @@ tags:
|
|
| 17 |
|
| 18 |
# Sol Milkshake
|
| 19 |
|
| 20 |
-
|
| 21 |
|
| 22 |
-
The
|
| 23 |
|
| 24 |
-
##
|
| 25 |
|
| 26 |
| Setting | Value |
|
| 27 |
|---|---|
|
|
@@ -42,15 +42,15 @@ The experiment combines hyperspherical optimization with shared transformer bloc
|
|
| 42 |
| Token embeddings | Tied to the output head |
|
| 43 |
| Weights | MLX NPZ |
|
| 44 |
|
| 45 |
-
##
|
| 46 |
|
| 47 |
-
Install MLX
|
| 48 |
|
| 49 |
```bash
|
| 50 |
pip install "mlx>=0.29" "tokenizers>=0.22" huggingface_hub
|
| 51 |
```
|
| 52 |
|
| 53 |
-
|
| 54 |
|
| 55 |
```python
|
| 56 |
from huggingface_hub import snapshot_download
|
|
@@ -73,7 +73,7 @@ text = generate(
|
|
| 73 |
print(text)
|
| 74 |
```
|
| 75 |
|
| 76 |
-
##
|
| 77 |
|
| 78 |
| Benchmark | Examples | Normalized accuracy |
|
| 79 |
|---|---:|---:|
|
|
@@ -83,20 +83,20 @@ print(text)
|
|
| 83 |
| PIQA | 1,838 | 52.50% |
|
| 84 |
| ArithMark-3 | 1,000 | 30.60% |
|
| 85 |
|
| 86 |
-
|
| 87 |
|
| 88 |
-
|
| 89 |
|
| 90 |
-
##
|
| 91 |
|
| 92 |
-
Pretraining
|
| 93 |
|
| 94 |
-
|
| 95 |
|
| 96 |
-
##
|
| 97 |
|
| 98 |
-
|
| 99 |
|
| 100 |
-
Milkshake is
|
| 101 |
|
| 102 |
-
The model
|
|
|
|
| 17 |
|
| 18 |
# Sol Milkshake
|
| 19 |
|
| 20 |
+
Milkshake tests recurrence and hyperspherical optimization in MLX on Apple Silicon. Five stored transformer blocks produce eleven block applications. Local token-pattern memory and rolling chunk memory accompany the shared blocks.
|
| 21 |
|
| 22 |
+
The model has 2,990,000 parameters. Its released checkpoint received 2,526,565,888 token exposures across pretraining and recovery. It generates text completions and hasn't been instruction-tuned.
|
| 23 |
|
| 24 |
+
## Configuration
|
| 25 |
|
| 26 |
| Setting | Value |
|
| 27 |
|---|---|
|
|
|
|
| 42 |
| Token embeddings | Tied to the output head |
|
| 43 |
| Weights | MLX NPZ |
|
| 44 |
|
| 45 |
+
## Run with MLX
|
| 46 |
|
| 47 |
+
Install MLX, the tokenizer library, and the Hugging Face Hub client:
|
| 48 |
|
| 49 |
```bash
|
| 50 |
pip install "mlx>=0.29" "tokenizers>=0.22" huggingface_hub
|
| 51 |
```
|
| 52 |
|
| 53 |
+
The downloaded repository contains the model implementation, loader, and generation function:
|
| 54 |
|
| 55 |
```python
|
| 56 |
from huggingface_hub import snapshot_download
|
|
|
|
| 73 |
print(text)
|
| 74 |
```
|
| 75 |
|
| 76 |
+
## Evaluation and selection
|
| 77 |
|
| 78 |
| Benchmark | Examples | Normalized accuracy |
|
| 79 |
|---|---:|---:|
|
|
|
|
| 83 |
| PIQA | 1,838 | 52.50% |
|
| 84 |
| ArithMark-3 | 1,000 | 30.60% |
|
| 85 |
|
| 86 |
+
Milkshake's local Intelligence Index score is 3.158. The four language-model tasks used their complete zero-shot splits and `lm-eval` 0.4.12. ArithMark-3 used independent tokenization and normalized accuracy. See `evals/` for the raw outputs.
|
| 87 |
|
| 88 |
+
We used this same benchmark suite to select the recovery checkpoint, so selection affects the reported scores. They aren't an untouched held-out estimate, and no independent evaluation has verified them.
|
| 89 |
|
| 90 |
+
## Pretraining and recovery
|
| 91 |
|
| 92 |
+
Pretraining drew on FineWeb-Edu, FinePDFs-Edu, English UltraFineWeb multi-domain and question-answer subsets, Cosmopedia, and FineMath. For recovery, the mixture was 70% original frozen curriculum, 15% Cosmopedia-v2 English, and 15% FinePhrase.
|
| 93 |
|
| 94 |
+
Recovery kept the tokenizer and prepared streams fixed. Filtering, deduplication, and decontamination rules also stayed the same. The run record is `training_state.json`.
|
| 95 |
|
| 96 |
+
## Repository contents
|
| 97 |
|
| 98 |
+
Load the MLX weights from `model.npz` with `modeling_sol_milkshake.py`. That file also provides generation; `sol_config.py` defines the architecture. The download includes `config.json`, tokenizer files, training metadata, and evaluation outputs.
|
| 99 |
|
| 100 |
+
Milkshake is intended for research on recurrence, memory, hyperspherical optimization, and MLX inference. Completions can be repetitive, incoherent, or incorrect.
|
| 101 |
|
| 102 |
+
The model license is [CC BY 4.0](LICENSE). Dataset licenses apply separately.
|