Question about PIXEL-M4 pretraining data and `PRETRAINING.md`
Hi! I’m trying to reproduce the PIXEL-M4 pretraining setup and had a couple of questions.
The repository’s README.md points to .github/PRETRAINING.md, but that file currently says that the instructions are a work in progress. Is a more complete version of the pretraining instructions still planned?
I also noticed that run_pretraining_pixel_m4.py contains placeholder paths for the four preprocessed language shards:
en_path = "path/to/data/preprocessed-c4-train/preprocessed_c4_bigrams_529_sep_wspace_cleaned_rmCo"
hi_path = "path/to/data/preprocessed-mc4-train/preprocessed_mc4_bigrams_529_hi"
uk_path = "path/to/data/preprocessed-mc4-train/preprocessed_mc4_bigrams_529_uk"
zh_path = "path/to/data/preprocessed-mc4-train/preprocessed_mc4_bigrams_529_zh"
fallback_fonts_dir = "path/to/data/fallback_fonts_dd2248copy"
The preprocessing code also refers to files that don't appear to have a source documented in the repository, such as unicode-hash-map/results.jsonl and unrenderable_chars.json.
I found these four datasets on the Hugging Face Hub, which seem to correspond to the same four languages:
Team-PIXEL/PIXELSum_en_wiki_for_TATeam-PIXEL/PIXELSum_hi_wiki_for_TATeam-PIXEL/PIXELSum_uk_wiki_for_TATeam-PIXEL/PIXELSum_zh_wiki_for_TA
Are these datasets the pretraining corpus for PIXEL-M4 rendered?
So it looks like the text has already been rendered into PNGs, whereas run_pretraining_pixel_m4.py appears to expect a text column and performs rendering during preprocessing/training.
I mainly want to make sure I’m using the correct data and following the intended preprocessing pipeline rather than making assumptions.
If there is a specific dataset, preprocessing script, or artifact corresponding to the paths in run_pretraining_pixel_m4.py, I’d really appreciate a pointer to it. Thanks!