atlasia/Atlaset
Viewer โข Updated โข 1.17M โข 117 โข 33
How to use BounharAbdelaziz/Moroccan-Darija-Tokenizer with Transformers:
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("BounharAbdelaziz/Moroccan-Darija-Tokenizer", device_map="auto")This tokenizer is designed for Moroccan Darija, a dialectal variety of Arabic (ISO code: ary).
It has been trained using the Byte Pair Encoding (BPE) algorithm on the dataset: atlasia/AL-Atlas-Moroccan-Darija-Pretraining-Dataset.