Commit
•
341a438
1
Parent(s):
8902689
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,2 +1,9 @@
|
|
| 1 |
-
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ruGPT-3.5 13B
|
| 2 |
|
| 3 |
+
Language model for Russian language. Model has 13B parameters as you can guess from it's name. This is our biggest model so far and it was used for trainig GigaChat (read more about it in the [article](https://habr.com/ru/companies/sberbank/articles/730108/)).
|
| 4 |
+
|
| 5 |
+
## Dataset
|
| 6 |
+
|
| 7 |
+
Model was pretrained on a 300Gb of various domains, than additionaly trained on the 100 Gb of code and legal documets. Here is the dataset structure:
|
| 8 |
+
|
| 9 |
+
<img src="https://habrastorage.org/getpro/habr/upload_files/384/cd1/40f/384cd140fbd9b4e7dd5427801be13ca0.png" width="800px">
|