Update README.md
Browse files
README.md
CHANGED
|
@@ -6,4 +6,8 @@ datasets:
|
|
| 6 |
language:
|
| 7 |
- la
|
| 8 |
---
|
| 9 |
-
Pretrained from scratch using GPT-2 architecture and a dataset of Latin texts ([Corpus Corporum](https://huggingface.co/datasets/Fece228/latin-literature-dataset-170M))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
language:
|
| 7 |
- la
|
| 8 |
---
|
| 9 |
+
Pretrained from scratch using GPT-2 architecture and a dataset of Latin texts ([Corpus Corporum](https://huggingface.co/datasets/Fece228/latin-literature-dataset-170M))
|
| 10 |
+
64 token context, loss 4.5, trained on 1 epoch of 492 million tokens
|
| 11 |
+
GPT2 style tokenizer trained with min_frequency of 2000
|
| 12 |
+
|
| 13 |
+
Tends to get repetitive and is not very coherent, due to size and limited data.
|