# Tokenizer Inspector

A small CLI tool for SLM builders: inspect and compare Hugging Face tokenizers before you commit one to a training run.

## What it tells you

- **Vocab size** and token type (byte-level BPE, SentencePiece Unigram, WordPiece, etc.)
- **Special tokens** — what's added on top of the learned vocab
- **Chars/token** on your test corpus — higher means more compact (fewer tokens to train on)
- **Vocab utilization** — what fraction of the vocab your data actually touches
- **Roundtrip check** — does `decode(encode(text))` give you back the same text?
- **Top tokens** — what the tokenizer is actually producing for your data
- **Side-by-side comparison** of two or more tokenizers on the same corpus

## Install

```bash
pip install tokenizers huggingface_hub
```

## Usage

```bash
# Inspect a single tokenizer by repo ID
python3 tokenizer_inspector.py gpt2

# Compare two tokenizers side by side
python3 tokenizer_inspector.py gpt2 HuggingFaceTB/SmolLM2-135M --compare

# Tokenize your own text
python3 tokenizer_inspector.py gpt2 --text "Hello, world! This is a test."

# Show the first 30 tokens produced
python3 tokenizer_inspector.py gpt2 --show-tokens

# Use a local tokenizer.json file
python3 tokenizer_inspector.py ./my_tokenizer.json

# JSON output (for scripting)
python3 tokenizer_inspector.py gpt2 --json
```

## Example output

```
============================================================
  COMPARISON (2 tokenizers)
============================================================
  Metric                               gpt2   HuggingFaceTB/Smol
  -------------------- -------------------- --------------------
  Vocab size                         50,257               49,152
  Token type                 byte-level BPE       byte-level BPE
  Special tokens                          1                   17
  Chars/token                           2.6                 2.72
  Corpus tokens                         116                  111
  Vocab util                          0.13%                0.12%
  Longest tok                           128                   81
  Roundtrip                              ok                 ok
```

## When to use this

- **Before choosing a tokenizer for training**: compare chars/token on a sample of your actual data, not just English prose
- **Debugging a bad tokenizer**: if roundtrip fails, your tokenizer is lossy
- **Vocab sizing**: if vocab utilization is very low (<1%), your vocab is oversized for your data
- **Special token audit**: see exactly what control tokens are in the vocab

## Notes

- The default corpus is mixed English + Python code. Use `--corpus` or `--text` to test on your own data.
- Chars/token is corpus-dependent. A tokenizer that's great for code may be worse for Turkish, and vice versa.
- This tool reads `tokenizer.json` (the fast-rs format). For SentencePiece `.model` files, it falls back to that format.