Update README.md
Browse files
README.md
CHANGED
@@ -18,7 +18,8 @@ model-index:
|
|
18 |
value: 73.04
|
19 |
name: normalized accuracy
|
20 |
source:
|
21 |
-
url:
|
|
|
22 |
name: Open LLM Leaderboard
|
23 |
- task:
|
24 |
type: text-generation
|
@@ -34,7 +35,8 @@ model-index:
|
|
34 |
value: 88.61
|
35 |
name: normalized accuracy
|
36 |
source:
|
37 |
-
url:
|
|
|
38 |
name: Open LLM Leaderboard
|
39 |
- task:
|
40 |
type: text-generation
|
@@ -51,7 +53,8 @@ model-index:
|
|
51 |
value: 75.49
|
52 |
name: accuracy
|
53 |
source:
|
54 |
-
url:
|
|
|
55 |
name: Open LLM Leaderboard
|
56 |
- task:
|
57 |
type: text-generation
|
@@ -67,7 +70,8 @@ model-index:
|
|
67 |
- type: mc2
|
68 |
value: 69.38
|
69 |
source:
|
70 |
-
url:
|
|
|
71 |
name: Open LLM Leaderboard
|
72 |
- task:
|
73 |
type: text-generation
|
@@ -84,7 +88,8 @@ model-index:
|
|
84 |
value: 85.32
|
85 |
name: accuracy
|
86 |
source:
|
87 |
-
url:
|
|
|
88 |
name: Open LLM Leaderboard
|
89 |
- task:
|
90 |
type: text-generation
|
@@ -101,8 +106,11 @@ model-index:
|
|
101 |
value: 67.7
|
102 |
name: accuracy
|
103 |
source:
|
104 |
-
url:
|
|
|
105 |
name: Open LLM Leaderboard
|
|
|
|
|
106 |
---
|
107 |
this is [miqu-1-70b](https://huggingface.co/miqudev/miqu-1-70b), dequantised from q5 to f16 && transposed to pytorch. shapes have been rotated less wrongly than in [alpindale/miqu-1-70b-pytorch](https://huggingface.co/alpindale/miqu-1-70b-pytorch/tree/main)
|
108 |
|
@@ -110,7 +118,7 @@ usage
|
|
110 |
```python
|
111 |
from transformers import LlamaForCausalLM as LLM, LlamaTokenizer as LT
|
112 |
|
113 |
-
lt = LT.from_pretrained("
|
114 |
t = lt("[INST] eloquent high camp prose about a cute catgirl [/INST]", return_tensors='pt').input_ids.cuda()
|
115 |
|
116 |
llm = LLM.from_pretrained("152334H/miqu-1-70b-sf", device_map='auto') # note: you may need many gpus for this
|
@@ -144,7 +152,7 @@ So let us raise our teacups in honor of this fabulous feline, this queen of camp
|
|
144 |
|
145 |
![](https://thicc-af.mywaifulist.moe/waifus/miku-nakano-the-quintessential-quintuplets/phUEiEhPOL75GTDLncGy2dUbkDVMfYExZ2A1RBeQ.png?class=thumbnail)
|
146 |
|
147 |
-
some benchmarks
|
148 |
|
149 |
```
|
150 |
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|
@@ -245,5 +253,4 @@ Detailed results can be found [here](https://huggingface.co/datasets/open-llm-le
|
|
245 |
|MMLU (5-Shot) |75.49|
|
246 |
|TruthfulQA (0-shot) |69.38|
|
247 |
|Winogrande (5-shot) |85.32|
|
248 |
-
|GSM8k (5-shot) |67.70|
|
249 |
-
|
|
|
18 |
value: 73.04
|
19 |
name: normalized accuracy
|
20 |
source:
|
21 |
+
url: >-
|
22 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
23 |
name: Open LLM Leaderboard
|
24 |
- task:
|
25 |
type: text-generation
|
|
|
35 |
value: 88.61
|
36 |
name: normalized accuracy
|
37 |
source:
|
38 |
+
url: >-
|
39 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
40 |
name: Open LLM Leaderboard
|
41 |
- task:
|
42 |
type: text-generation
|
|
|
53 |
value: 75.49
|
54 |
name: accuracy
|
55 |
source:
|
56 |
+
url: >-
|
57 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
58 |
name: Open LLM Leaderboard
|
59 |
- task:
|
60 |
type: text-generation
|
|
|
70 |
- type: mc2
|
71 |
value: 69.38
|
72 |
source:
|
73 |
+
url: >-
|
74 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
75 |
name: Open LLM Leaderboard
|
76 |
- task:
|
77 |
type: text-generation
|
|
|
88 |
value: 85.32
|
89 |
name: accuracy
|
90 |
source:
|
91 |
+
url: >-
|
92 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
93 |
name: Open LLM Leaderboard
|
94 |
- task:
|
95 |
type: text-generation
|
|
|
106 |
value: 67.7
|
107 |
name: accuracy
|
108 |
source:
|
109 |
+
url: >-
|
110 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
111 |
name: Open LLM Leaderboard
|
112 |
+
language:
|
113 |
+
- en
|
114 |
---
|
115 |
this is [miqu-1-70b](https://huggingface.co/miqudev/miqu-1-70b), dequantised from q5 to f16 && transposed to pytorch. shapes have been rotated less wrongly than in [alpindale/miqu-1-70b-pytorch](https://huggingface.co/alpindale/miqu-1-70b-pytorch/tree/main)
|
116 |
|
|
|
118 |
```python
|
119 |
from transformers import LlamaForCausalLM as LLM, LlamaTokenizer as LT
|
120 |
|
121 |
+
lt = LT.from_pretrained("152334H/miqu-1-70b-sf")
|
122 |
t = lt("[INST] eloquent high camp prose about a cute catgirl [/INST]", return_tensors='pt').input_ids.cuda()
|
123 |
|
124 |
llm = LLM.from_pretrained("152334H/miqu-1-70b-sf", device_map='auto') # note: you may need many gpus for this
|
|
|
152 |
|
153 |
![](https://thicc-af.mywaifulist.moe/waifus/miku-nakano-the-quintessential-quintuplets/phUEiEhPOL75GTDLncGy2dUbkDVMfYExZ2A1RBeQ.png?class=thumbnail)
|
154 |
|
155 |
+
## some benchmarks
|
156 |
|
157 |
```
|
158 |
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|
|
|
253 |
|MMLU (5-Shot) |75.49|
|
254 |
|TruthfulQA (0-shot) |69.38|
|
255 |
|Winogrande (5-shot) |85.32|
|
256 |
+
|GSM8k (5-shot) |67.70|
|
|