Scratch GPT — MCQ Top-3 Ranker
A decoder-only transformer implemented from scratch in PyTorch — no pretrained
weights, no transformers model class. Pretrained as a character/BPE language
model on a Wikipedia corpus, then fine-tuned with a linear scoring head to rank
five multiple-choice options (A–E) and emit the top three, scored with MAP@3.
Interactive demo: https://huggingface.co/spaces/your-hf-username/scratch-gpt-mcq
Architecture
Built up from the primitives: single attention head → multi-head attention → position-wise MLP → pre-norm residual block → full model.
| Tokenizer | Custom BPE trained on the corpus (vocab 4,000); character-level variant also implemented |
| Blocks | 4 |
| Attention heads | 4 (head size 64) |
| d_model | 256 |
| Context length | 128 tokens |
| Dropout | 0.1 |
| Positional encoding | Learned embeddings |
| Normalisation | Pre-norm LayerNorm, residual around attention and MLP |
| Parameters | 5.24M |
Training
Stage 1 — language-model pretraining. Next-token cross-entropy on a Wikipedia corpus, AdamW at 3e-4, batch 64, 4000 iterations, 10% held out for validation. Reported as bits-per-character so the BPE and character tokenizers are comparable on the same axis.
Stage 2 — multiple-choice fine-tuning. Each (question, option) pair is
encoded as question + "\n" + option, tail-truncated to the context window so
the option always survives. The hidden state at the last real token goes through
a linear head to one scalar; the five scalars are softmaxed and trained with
cross-entropy against the correct letter. AdamW at 3e-4, 3 epochs, 8 questions
(40 sequences) per step. The train/validation split is grouped by prompt so no
prompt appears on both sides.
Files
| File | What it is |
|---|---|
gpt_wiki_mc.pt |
Fine-tuned multiple-choice model (state_dict + config) |
gpt_wiki.pt |
Pretrained language model before fine-tuning |
tokenizer_bpe.json |
BPE vocabulary and merge list |
modeling.py |
Model and tokenizer definitions needed to load the checkpoints |
Usage
import torch
from huggingface_hub import hf_hub_download
import modeling # download modeling.py from this repo
REPO = "your-hf-username/scratch-gpt-mcq"
tok = modeling.load_tokenizer(hf_hub_download(REPO, "tokenizer_bpe.json"))
mc, cfg = modeling.load_mc_model(hf_hub_download(REPO, "gpt_wiki_mc.pt"))
top3, scores = modeling.rank_options(
mc, tok, cfg,
"Which phenomenon explains the bending of light around a massive object?",
["Gravitational lensing", "Rayleigh scattering", "Total internal reflection",
"The photoelectric effect", "Bremsstrahlung"],
)
print(" ".join(top3)) # e.g. "A C B" -> submission format
Limitations
5.24M parameters trained on a small corpus for a few thousand steps. It is an architectural exercise, not a competitive QA system: the language-model samples are Wikipedia-flavoured but not factual, and the multiple-choice head generalises only to questions resembling the competition distribution. Not for any use where being wrong matters.