fblgit/UNA-ThePitbull-21.4-v1

For a better performance check out our v2 at fblgit/UNA-ThePitbull-21.4B-v2

UNA-ThePitbull 21.4B v1

Introducing the best LLM in the industry. Nearly as good as a 70B, just a 21.4B based on saltlux/luxia-21.4b-alignment-v1.0

This model has not been poisoned to score high and be useless. We release him becaues its the real deal of EQ & IQ all together in a crazy powerful smart and conversational model. So far the #1 of them at 25/5/2024

Quant version available at bartowski/UNA-ThePitbull-21.4-v1-GGUF

For a better performance check out our v2 at fblgit/UNA-ThePitbull-21.4B-v2

Evaluations

Can only be compared with its non-una base model: the original luxia-21.4b.

UNA (VLLM) Evaluations

|    Tasks     |Version|     Filter     |n-shot|  Metric   |Value |   |Stderr|
|--------------|------:|----------------|-----:|-----------|-----:|---|-----:|
|gsm8k         |      3|strict-match    |     5|exact_match|0.7566|±  |0.0118|
|              |       |flexible-extract|     5|exact_match|0.7582|±  |0.0118|
|hellaswag     |      1|none            |    10|acc        |0.8168|±  |0.0039|
|              |       |none            |    10|acc_norm   |0.9188|±  |0.0027|
|winogrande    |      1|none            |     5|acc        |0.8635|±  |0.0097|
|mmlu          |    N/A|none            |     0|acc        |0.6444|±  |0.0038|
|arc_challenge |      1|none            |    25|acc        |0.7747|±  |0.0122|
|              |       |none            |    25|acc_norm   |0.7850|±  |0.0120|
|truthfulqa_mc2|      2|none            |     0|acc        |0.7902|±  |0.0134|
|mathqa        |      1|none            |     0|acc        |0.4030|±  | 0.009|
|              |       |none            |     0|acc_norm   |0.4034|±  | 0.009|
|pubmedqa      |      1|none            |     0|acc        |0.6860|±  |0.0208|
|boolq         |      2|none            |     0|acc        |0.8401|±  |0.0064|

Original (VLLM) Evaluations

|    Tasks     |Version|     Filter     |n-shot|  Metric   |Value |   |Stderr|
|--------------|------:|----------------|-----:|-----------|-----:|---|-----:|
|gsm8k         |      3|strict-match    |     5|exact_match|0.7528|±  |0.0119|
|              |       |flexible-extract|     5|exact_match|0.7521|±  |0.0119|
|hellaswag     |      1|none            |    10|acc        |0.8117|±  |0.0039|
|              |       |none            |    10|acc_norm   |0.9167|±  |0.0028|
|winogrande    |      1|none            |     5|acc        |0.8682|±  |0.0095|
|mmlu          |    N/A|none            |     0|acc        |0.6448|±  |0.0038|
|arc_challenge |      1|none            |    25|acc        |0.7688|±  |0.0123|
|              |       |none            |    25|acc_norm   |0.7730|±  |0.0122|
|truthfulqa_mc2|      2|none            |     0|acc        |0.7895|±  |0.0133|
|mathqa        |      1|none            |     0|acc        |0.4000|±  | 0.009|
|              |       |none            |     0|acc_norm   |0.4003|±  | 0.009|
|pubmedqa      |      1|none            |     0|acc        |0.6680|±  |0.0211|
|boolq         |      2|none            |     0|acc        |0.8346|±  |0.0065|

UNA Details

Only MLP were Uniformed leaving room for further optimisations. You should be able to perform a SFT+DPO again on this model at moderate speeds. 1e-4/2e-5/etc.