gpt2-medium-dutch / README.md
yhavinga's picture
Update model card
74babbc
|
raw
history blame
1.4 kB
metadata
language: nl
widget:
  - text: In het jaar 2030 zullen we
  - text: Toen ik gisteren volledig in de ban was van
  - text: >-
      Studenten en leraren van de Bogazici Universiteit in de Turkse stad
      Istanbul
  - text: In Israël was een strenge lockdown
tags:
  - gpt2-medium
  - gpt2
pipeline_tag: text-generation
datasets:
  - yhavinga/mc4_nl_cleaned

GPT2-Medium pre-trained on cleaned Dutch mC4 🇳🇱

Dataset:

Tokenizer:

  • New tokenizer trained on mC4 with the scripts from the Huggingface Transformers Flax examples

Training details:

  • Trained for 240k steps (29 dec 2021)
  • Block size: 512
  • Optimizer: adam, lr 8e-4, beta1 0.9, beta2 0.98
  • Warmup steps: 5000
  • Weight decay: 0.01

Work in progress. Dec 2021.