Update README.md
Browse files
README.md
CHANGED
|
@@ -17,7 +17,7 @@ base_model: beratcmn/Llama-3-11.5B
|
|
| 17 |
---
|
| 18 |
# Llama-3-11.5B
|
| 19 |
|
| 20 |
-
This model is a Proof of Concept. First 2 Llama-3-8B models has been merged using `Mergekit` and pre-training continued using `QLora` and `Unsloth` for 1000 samples from `roneneldan/TinyStories
|
| 21 |
Loss still decreases each epoch so I believe this is a successful experiment where there is a lot of room to experiment.
|
| 22 |
|
| 23 |
Llama-3-11.5B is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
|
|
|
|
| 17 |
---
|
| 18 |
# Llama-3-11.5B
|
| 19 |
|
| 20 |
+
This model is a Proof of Concept. First 2 Llama-3-8B models has been merged using `Mergekit` and pre-training continued using `QLora` and `Unsloth` for 1000 samples from `roneneldan/TinyStories`.
|
| 21 |
Loss still decreases each epoch so I believe this is a successful experiment where there is a lot of room to experiment.
|
| 22 |
|
| 23 |
Llama-3-11.5B is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
|