abhinavnmagic
commited on
Commit
•
91a872b
1
Parent(s):
a8c9e50
Update README.md
Browse files
README.md
CHANGED
@@ -28,7 +28,7 @@ license: llama3.1
|
|
28 |
- **Model Developers:** Neural Magic
|
29 |
|
30 |
Quantized version of [Meta-Llama-3.1-405B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct).
|
31 |
-
It achieves an average score of
|
32 |
|
33 |
### Model Optimizations
|
34 |
|
@@ -146,11 +146,11 @@ This version of the lm-evaluation-harness includes versions of ARC-Challenge, GS
|
|
146 |
<tr>
|
147 |
<td>MMLU (5-shot)
|
148 |
</td>
|
149 |
-
<td>
|
150 |
</td>
|
151 |
-
<td>
|
152 |
</td>
|
153 |
-
<td>
|
154 |
</td>
|
155 |
</tr>
|
156 |
<tr>
|
@@ -176,37 +176,41 @@ This version of the lm-evaluation-harness includes versions of ARC-Challenge, GS
|
|
176 |
<tr>
|
177 |
<td>Hellaswag (10-shot)
|
178 |
</td>
|
179 |
-
<td>
|
180 |
</td>
|
181 |
-
|
|
|
|
|
182 |
</td>
|
183 |
</tr>
|
184 |
<tr>
|
185 |
<td>Winogrande (5-shot)
|
186 |
</td>
|
187 |
-
<td>
|
|
|
|
|
188 |
</td>
|
189 |
-
<td>
|
190 |
</td>
|
191 |
</tr>
|
192 |
<tr>
|
193 |
<td>TruthfulQA (0-shot)
|
194 |
</td>
|
195 |
-
<td>
|
196 |
</td>
|
197 |
-
<td>
|
198 |
</td>
|
199 |
-
<td>
|
200 |
</td>
|
201 |
</tr>
|
202 |
<tr>
|
203 |
<td><strong>Average</strong>
|
204 |
</td>
|
205 |
-
<td><strong>
|
206 |
</td>
|
207 |
-
<td><strong>
|
208 |
</td>
|
209 |
-
<td><strong>
|
210 |
</td>
|
211 |
</tr>
|
212 |
</table>
|
|
|
28 |
- **Model Developers:** Neural Magic
|
29 |
|
30 |
Quantized version of [Meta-Llama-3.1-405B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct).
|
31 |
+
It achieves an average score of 86.01 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 86.63.
|
32 |
|
33 |
### Model Optimizations
|
34 |
|
|
|
146 |
<tr>
|
147 |
<td>MMLU (5-shot)
|
148 |
</td>
|
149 |
+
<td>86.25
|
150 |
</td>
|
151 |
+
<td>85.97
|
152 |
</td>
|
153 |
+
<td>99.67%
|
154 |
</td>
|
155 |
</tr>
|
156 |
<tr>
|
|
|
176 |
<tr>
|
177 |
<td>Hellaswag (10-shot)
|
178 |
</td>
|
179 |
+
<td>88.33
|
180 |
</td>
|
181 |
+
<td>88.16
|
182 |
+
</td>
|
183 |
+
<td>99.80%
|
184 |
</td>
|
185 |
</tr>
|
186 |
<tr>
|
187 |
<td>Winogrande (5-shot)
|
188 |
</td>
|
189 |
+
<td>87.21
|
190 |
+
</td>
|
191 |
+
<td>85.95
|
192 |
</td>
|
193 |
+
<td>98.55%
|
194 |
</td>
|
195 |
</tr>
|
196 |
<tr>
|
197 |
<td>TruthfulQA (0-shot)
|
198 |
</td>
|
199 |
+
<td>64.64
|
200 |
</td>
|
201 |
+
<td>64.75
|
202 |
</td>
|
203 |
+
<td>100.17%
|
204 |
</td>
|
205 |
</tr>
|
206 |
<tr>
|
207 |
<td><strong>Average</strong>
|
208 |
</td>
|
209 |
+
<td><strong>86.63</strong>
|
210 |
</td>
|
211 |
+
<td><strong>86.01</strong>
|
212 |
</td>
|
213 |
+
<td><strong>99.28%</strong>
|
214 |
</td>
|
215 |
</tr>
|
216 |
</table>
|