j0no12 commited on
Commit
eb48963
·
verified ·
1 Parent(s): 2b6ad91

Rewrite model card in plain language

Browse files
Files changed (1) hide show
  1. README.md +16 -16
README.md CHANGED
@@ -17,11 +17,11 @@ tags:
17
 
18
  # Sol Milkshake
19
 
20
- Sol Milkshake is a recurrent language model that runs with MLX on Apple Silicon. It has 2,990,000 parameters, and the released checkpoint received 2,526,565,888 token exposures across pretraining and recovery training.
21
 
22
- The experiment combines hyperspherical optimization with shared transformer blocks. It also has two kinds of memory: local token-pattern memory and rolling memory across chunks. Milkshake completes text; it hasn't been instruction-tuned.
23
 
24
- ## What's in the model
25
 
26
  | Setting | Value |
27
  |---|---|
@@ -42,15 +42,15 @@ The experiment combines hyperspherical optimization with shared transformer bloc
42
  | Token embeddings | Tied to the output head |
43
  | Weights | MLX NPZ |
44
 
45
- ## Try it on Apple Silicon
46
 
47
- Install MLX and the tokenizer library, along with the Hub client:
48
 
49
  ```bash
50
  pip install "mlx>=0.29" "tokenizers>=0.22" huggingface_hub
51
  ```
52
 
53
- Download the repository, then import the model code and loader included here:
54
 
55
  ```python
56
  from huggingface_hub import snapshot_download
@@ -73,7 +73,7 @@ text = generate(
73
  print(text)
74
  ```
75
 
76
- ## Results and checkpoint selection
77
 
78
  | Benchmark | Examples | Normalized accuracy |
79
  |---|---:|---:|
@@ -83,20 +83,20 @@ print(text)
83
  | PIQA | 1,838 | 52.50% |
84
  | ArithMark-3 | 1,000 | 30.60% |
85
 
86
- The reported Intelligence Index is 3.158. The four language-model tasks use the full zero-shot splits with `lm-eval` 0.4.12. ArithMark-3 uses independent tokenization and normalized accuracy. The raw outputs are in `evals/`.
87
 
88
- Recovery runs used this same suite to choose the checkpoint. That means the scores include the effect of checkpoint selection; they aren't an untouched held-out estimate. No independent evaluation has verified them.
89
 
90
- ## Training
91
 
92
- Pretraining used FineWeb-Edu and FinePDFs-Edu, English UltraFineWeb multi-domain and question-answer subsets, Cosmopedia, and FineMath. Recovery mixed 70% of the original frozen curriculum with 15% Cosmopedia-v2 English and 15% FinePhrase.
93
 
94
- The tokenizer and prepared streams stayed fixed during recovery, as did the existing filtering, deduplication, and decontamination rules. `training_state.json` records the run details.
95
 
96
- ## Files and limits
97
 
98
- `model.npz` contains the MLX weights. The loader and generator are in `modeling_sol_milkshake.py`; `sol_config.py` holds the architecture settings. You'll also find `config.json`, tokenizer files, training metadata, and evaluation outputs here.
99
 
100
- Milkshake is a small research model for studying recurrence, memory, and hyperspherical optimization, as well as MLX inference. It can produce repetitive or incoherent text and get answers wrong.
101
 
102
- The model uses [CC BY 4.0](LICENSE). The training datasets keep their own licenses.
 
17
 
18
  # Sol Milkshake
19
 
20
+ Milkshake tests recurrence and hyperspherical optimization in MLX on Apple Silicon. Five stored transformer blocks produce eleven block applications. Local token-pattern memory and rolling chunk memory accompany the shared blocks.
21
 
22
+ The model has 2,990,000 parameters. Its released checkpoint received 2,526,565,888 token exposures across pretraining and recovery. It generates text completions and hasn't been instruction-tuned.
23
 
24
+ ## Configuration
25
 
26
  | Setting | Value |
27
  |---|---|
 
42
  | Token embeddings | Tied to the output head |
43
  | Weights | MLX NPZ |
44
 
45
+ ## Run with MLX
46
 
47
+ Install MLX, the tokenizer library, and the Hugging Face Hub client:
48
 
49
  ```bash
50
  pip install "mlx>=0.29" "tokenizers>=0.22" huggingface_hub
51
  ```
52
 
53
+ The downloaded repository contains the model implementation, loader, and generation function:
54
 
55
  ```python
56
  from huggingface_hub import snapshot_download
 
73
  print(text)
74
  ```
75
 
76
+ ## Evaluation and selection
77
 
78
  | Benchmark | Examples | Normalized accuracy |
79
  |---|---:|---:|
 
83
  | PIQA | 1,838 | 52.50% |
84
  | ArithMark-3 | 1,000 | 30.60% |
85
 
86
+ Milkshake's local Intelligence Index score is 3.158. The four language-model tasks used their complete zero-shot splits and `lm-eval` 0.4.12. ArithMark-3 used independent tokenization and normalized accuracy. See `evals/` for the raw outputs.
87
 
88
+ We used this same benchmark suite to select the recovery checkpoint, so selection affects the reported scores. They aren't an untouched held-out estimate, and no independent evaluation has verified them.
89
 
90
+ ## Pretraining and recovery
91
 
92
+ Pretraining drew on FineWeb-Edu, FinePDFs-Edu, English UltraFineWeb multi-domain and question-answer subsets, Cosmopedia, and FineMath. For recovery, the mixture was 70% original frozen curriculum, 15% Cosmopedia-v2 English, and 15% FinePhrase.
93
 
94
+ Recovery kept the tokenizer and prepared streams fixed. Filtering, deduplication, and decontamination rules also stayed the same. The run record is `training_state.json`.
95
 
96
+ ## Repository contents
97
 
98
+ Load the MLX weights from `model.npz` with `modeling_sol_milkshake.py`. That file also provides generation; `sol_config.py` defines the architecture. The download includes `config.json`, tokenizer files, training metadata, and evaluation outputs.
99
 
100
+ Milkshake is intended for research on recurrence, memory, hyperspherical optimization, and MLX inference. Completions can be repetitive, incoherent, or incorrect.
101
 
102
+ The model license is [CC BY 4.0](LICENSE). Dataset licenses apply separately.