Instructions to use naclbit/gpt-j-japanese-6.8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use naclbit/gpt-j-japanese-6.8b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="naclbit/gpt-j-japanese-6.8b")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("naclbit/gpt-j-japanese-6.8b")
model = AutoModelForCausalLM.from_pretrained("naclbit/gpt-j-japanese-6.8b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use naclbit/gpt-j-japanese-6.8b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "naclbit/gpt-j-japanese-6.8b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "naclbit/gpt-j-japanese-6.8b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/naclbit/gpt-j-japanese-6.8b

SGLang

How to use naclbit/gpt-j-japanese-6.8b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "naclbit/gpt-j-japanese-6.8b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "naclbit/gpt-j-japanese-6.8b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "naclbit/gpt-j-japanese-6.8b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "naclbit/gpt-j-japanese-6.8b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use naclbit/gpt-j-japanese-6.8b with Docker Model Runner:
```
docker model run hf.co/naclbit/gpt-j-japanese-6.8b
```

This pre-trained model is work in progress! Model weight download will be available in the future.

A 6.8 billion parameter pre-trained model for Japanese language, based on EleutherAI's Mesh Transformer JAX, that has a similar model structure to their GPT-J-6B pre-trained model.

EleutherAIによるMesh Transformer JAXをコードベースとした、GPT-J-6Bに似たストラクチャと約68.7億パラメータを持つ日本語pre-trainedモデルです。

We used T5Tokenizer and SentencePiece instead of GPT-2/3 tokenizer. Normalization done by SentencePiece is must for Japanese tokenizing as there are so much many more variations for common symbols than Western languages.
Tokenizer has a vocabulary of 52,500 tokens and trained on Japanese Wikipedia dump as of 01 Aug 2021.
The model fits within 16GB VRAM GPUs like P100 for inference up to 1688 context length. Full 2048 context length output requires 20GB VRAM or more (e.g. GTX3090/A5000).
The model was trained with TPUv3-128 generously provided by Google TRC for about 4 weeks. We are currently formatting additional datasets and preparing for more training time.

Specifications

Hyperparameter	Value
n_parameters	6,876,450,080
n_layers	32
d_model	4,096
d_ff	16,384
n_heads	16
d_head	256
n_ctx	2,048
n_vocab	52,512
position encoding	Rotary position encodings (RoPE)
RoPE dimensions	64

Instructions

We recommend to use finetuneanon's forked transformer codebase for inferencing as split checkpoint loads up a lot faster than monolithic checkpoint supported by HuggingFace Transformers repository.

The tokenizer still uses 50256 as the <|endoftext|> substitute. Therefore 50256 should be excluded when inferencing.

Datasets

Lack of quality Japanese corpus was one of the major challenges when we trained the model. We aimed to compile well-formatted corpuses outside of Common Crawl.

The dataset is normalized and sanitized against leading and trailing spaces, excessive CR/LF repetitions.

The whole dataset is about 400GB (as of October 2021) and 106B tokens (compared to 825GB/300B tokens for The Pile).

** Common Crawl

Jan-Dec 2018 72GB CC100-Japanese (https://metatext.io/datasets/cc100-japanese)
November 2018 106GB OSCAR-Japanese (https://oscar-corpus.com)
75GB Converted 860GB Google C4 Multilingual Japanese (re-formatted)

** Books

140GB Web Fictions, non-fictions and blogs corpus
5GB Books and Aozora Bunko corpus (weighted 2x)

** News

1GB Scientific news, medical news and web news corpus

** Wikipedia

Aug 2021 3GB Assorted and Deduplicated Japanese Wikipedia (weighted 2x)
Aug 2021 Wikibooks, Wikinews, Wikiquote, Wikisource, Wiktionary, Wikiversity and Wikivoyage

** Other Corpuses

2018 OpenSubtitles (https://opus.nlpl.eu/OpenSubtitles-v2018.php)
80-90's BBS Logs
Assorted Blogs Crawl
QED-ja
TED 2020-ja

Downloads last month: 353

Paper for naclbit/gpt-j-japanese-6.8b

RoFormer: Enhanced Transformer with Rotary Position Embedding

Paper • 2104.09864 • Published Apr 20, 2021 • 20