A weird model trained on a Pi. This model is a standalone version; no need to download anything else. (Please participate in a survey on the Community tab! I want to see why you use my model!) PLEASE FOR THE LOVE OF GOD FOLLOW BananaMind

Training Details

Category	Specification
Model Architecture	GPT‑2 (decoder‑only Transformer)
Total Parameters	346,104
Model Size (FP32)	1.32 MB (1,387,072 bytes)
Layers (`n_layer`)	2
Embedding Dimension (`n_embd`)	92
Attention Heads (`n_head`)	2
Context Length (`n_positions`)	1024
Vocabulary Size	502 (byte‑level BPE, tied embeddings)
Training Framework	Oobabooga TextGen WebUI (Transformers + PEFT)
Fine‑Tuning Method	LoRA (Target all linear layers)
LoRA Rank	8
LoRA Alpha	16
Optimizer	AdamW 8‑bit (`adamw_bnb_8bit`)
Learning Rate	2e‑4 (linear scheduler, 20 warmup steps)
Batch Size	4
Training Dataset	Capybara ShareGPT (random subset, 1.36 MB, 1,453,408 bytes)
Dataset Format	JSON array with `"text"` field (flattened ShareGPT)
Training Epochs	5
Loss Target	Stopped at ~6.0 (plateaued due to parameter limit)
Hardware	Raspberry Pi 5, 8GB RAM, ARM Cortex‑A76 (CPU only)
Training Time	~1 hour
Tokenizer	Custom byte‑level BPE (trained on dataset)
Tokenizer Special Tokens	`[BOS]`, `[EOS]`, `[PAD]`, `[UNK]`
Note	This is a proof‑of‑concept model. It fits on a 3.5‑inch floppy disk and was trained from scratch on a Raspberry Pi.

This thing still produces nonsense. Be warned. Chats should be formatted as:

### User:
<user prompt>
### Assistant:
<assistant response>

Cite as:

[Single Floppy 346K Merged](https://huggingface.co/NILKNARFGonzo/single-floppy-346k-base/) by [Nilky](https://huggingface.co/NILKNARFGonzo) is licenced under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)