ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Paper โข 2003.10555 โข Published
How to use sarnikowski/electra-small-discriminator-da-256-cased with Transformers:
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForPreTraining
tokenizer = AutoTokenizer.from_pretrained("sarnikowski/electra-small-discriminator-da-256-cased")
model = AutoModelForPreTraining.from_pretrained("sarnikowski/electra-small-discriminator-da-256-cased", device_map="auto")An ELECTRA model pretrained on a custom Danish corpus (~17.5gb). For details regarding data sources and training procedure, along with benchmarks on downstream tasks, go to: https://github.com/sarnikowski/danish_transformers/tree/main/electra
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("sarnikowski/electra-small-discriminator-da-256-cased")
model = AutoModel.from_pretrained("sarnikowski/electra-small-discriminator-da-256-cased")
If you have any questions feel free to open an issue on the danish_transformers repository, or send an email to p.sarnikowski@gmail.com