Commit
•
c27ce18
1
Parent(s):
1481ffb
Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
- ru
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
## Description
|
| 9 |
+
This is an instruction following model (based on Mistral v0.1 Base) optimized for Russian language.
|
| 10 |
+
It was trained using [kolibrify](https://github.com/oKatanaaa/kolibrify) on a multitude of instruction datasets.
|
| 11 |
+
|
| 12 |
+
The model uses ChatML template. Adding a system prompt will likely improve the model's performance on your task (experiment with it).
|
| 13 |
+
I recommend using the model with LMStudio.
|
| 14 |
+
|
| 15 |
+
Currently in pre-alpha, later releases will include more details regarding training procedure and data mix.
|
| 16 |
+
|
| 17 |
+
## Instruction following evals
|
| 18 |
+
The model was tested using the following benchmarks:
|
| 19 |
+
- [ruIFEval](https://github.com/NLP-Core-Team/ruIFEval)
|
| 20 |
+
- [ifeval](https://github.com/google-research/google-research/tree/master/instruction_following_eval)
|
| 21 |
+
| Eval name |Strict Value| Loose Value
|
| 22 |
+
|---------------------------------|----|----|
|
| 23 |
+
|Avg. |*49.92*|*52.52*|
|
| 24 |
+
|ifeval-prompt-level |49.72|52.31|
|
| 25 |
+
|ifeval-instruction-level |60.91|63.18|
|
| 26 |
+
|ru-ifeval-prompt-level |39.55|42.32|
|
| 27 |
+
|ru-ifeval-instruction-level |49.52|52.27|
|