Upload folder using huggingface_hub

README.md

@@ -20,7 +20,7 @@ A **Byte-Level BPE** tokenizer trained on **jpn_Jpan** data from Fineweb-2-HQ.
 | Algorithm | Byte-Level BPE |
 | Language | `jpn_Jpan` |
 | Target Vocab Size | 8,000 |
-| Final Vocab Size | 9,957 |
+| Final Vocab Size | 9,956 |
 | Pre-tokenizer | custom:jpn_Jpan |
 | Number handling | ltr_3digit |
 | Contraction handling | True |
@@ -46,4 +46,4 @@ tokens = tokenizer.encode("Hello, world!")
 ## Sample Encoding
 | Text | Tokens | Token IDs |
 |------|--------|-----------|
-| `Hello, world! 12345 This is a test. こんにちは` | `H, ell, o, ,, Ġw, or, ld, !, Ġ, 123, 45, ĠT, h, is, Ġis, Ġa, Ġt, est, ., Ġãģĵ` | `42, 4990, 81, 14, 2135, 956, 5825, 3, 223, 9639, 4679, 1615, 74, 1271, 4943, 2560, 1094, 3790, 16, 4587` |
+| `Hello, world! 12345 This is a test. こんにちは` | `H, ell, o, ,, Ġw, or, ld, !, Ġ, 123, 45, ĠT, h, is, Ġis, Ġa, Ġt, est, ., Ġãģĵ` | `42, 4980, 81, 14, 2117, 953, 5795, 3, 223, 9470, 4690, 1611, 74, 1257, 4909, 2412, 1086, 3761, 16, 4547` |