qikp's Educational Scorer (QES)

QES is a model with an identical purpose to HuggingFaceFW/fineweb-edu-classifier, and is trained on a subset of its data.

The biggest difference is that QES is a fine-tune of huawei-noah/TinyBERT_General_4L_312D instead of Snowflake/snowflake-arctic-embed-m.

Mozilla Firefox includes a model fine-tuned on the same base model as QES for form autofill, so the base model's reliability is proven.

Training data

The first parquet shard of HuggingFaceFW/fineweb-edu-llama3-annotations was used. Additionally, a padding data collator was used.

Training details

Training took 20 minutes and 49 seconds on a single T4 GPU from Google.

Model was trained as a FP32/FP16 hybrid as the Turing architecture does not support bfloat16.

The default batch size and learning rate was used.

The model was trained for 2 epochs.

Limitations

The model deviates by up to around three quarters of a point or so during limited internal testing compared to the final FineWeb-Edu dataset. This accuracy is not guaranteed.

As such, it should only be used in constrained circumstances or circumstances involving colossal amounts of data.

Additionally, models like QES are designed as an additional post-filtering step over already filtered data. Using QES on unfiltered web scrapes is likely going to miss spam and thin content.