metadata
license: afl-3.0
library_name: transformers
tags:
- UNA
- juanako
datasets:
- jondurbin/py-dpo-v0.1
- Replete-AI/code_bagel_hermes-2.5
- mlabonne/orpo-dpo-mix-40k
UNA-ThePitbull 21.4B v2
Introducing the best LLM in the industry. Nearly as good as a 70B, just a 21.4B based on saltlux/luxia-21.4b-alignment-v1.0
This model has not been poisoned to score high and be useless. We release him becaues its the real deal of EQ & IQ all together in a crazy powerful smart and conversational model.
Quant version available at ... soon ..
Difference V1 vs V2
On V2 we implemented a different UNA strategy and covered partially the MLP's and Attention Layers. We also performed further SFT over V1 and further DPO over V1 and we'll release some of those soon as well.
Changes
- SFT over V1 with
Replete-AI/code_bagel_hermes-2.5at 1.0e-4 till 5.0e-5 - DPO with: 1.0e-4 to min_lr 5.0e-5
mlabonne/orpo-dpo-mix-40kjondurbin/py-dpo-v0.1
Evaluations
Can only be compared with its non-una base model: the original luxia-21.4b and ThePitbull-v1
UNA v2 (VLLM) Evaluations:
vllm (pretrained=/data/tools/mergekit/una-thepitbull-v5,dtype=bfloat16,gpu_memory_utilization=0.8,max_model_len=2048,data_parallel_size=2,tensor_parallel_size=4), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
| Tasks |Version| Filter |n-shot| Metric |Value | |Stderr|
|--------------|------:|----------------|-----:|-----------|-----:|---|-----:|
|gsm8k | 3|strict-match | 5|exact_match|0.7695|± |0.0116|+
| | |flexible-extract| 5|exact_match|0.7695|± |0.0116|+
|hellaswag | 1|none | 10|acc |0.8110|± |0.0039|
| | |none | 10|acc_norm |0.9169|± |0.0028|+
|winogrande | 1|none | 5|acc |0.8777|± |0.0092|+
|mmlu |N/A |none | 0|acc |0.6427|± |0.0038|-
|arc_challenge | 1|none | 25|acc |0.7713|± |0.0123|
| | |none | 25|acc_norm |0.7875|± |0.0120|+
|truthfulqa_mc2| 2|none | 0|acc |0.7824|± |0.0135|-
|mathqa | 1|none | 0|acc |0.4037|± | 0.009|
| | |none | 0|acc_norm |0.4034|± | 0.009|+
|pubmedqa | 1|none | 0|acc |0.7260|± | 0.020|+
|boolq | 2|none | 0|acc |0.8602|± |0.0061|+
UNA v1 (VLLM) Evaluations
| Tasks |Version| Filter |n-shot| Metric |Value | |Stderr|
|--------------|------:|----------------|-----:|-----------|-----:|---|-----:|
|gsm8k | 3|strict-match | 5|exact_match|0.7566|± |0.0118|
| | |flexible-extract| 5|exact_match|0.7582|± |0.0118|
|hellaswag | 1|none | 10|acc |0.8168|± |0.0039|
| | |none | 10|acc_norm |0.9188|± |0.0027|
|winogrande | 1|none | 5|acc |0.8635|± |0.0097|
|mmlu | N/A|none | 0|acc |0.6444|± |0.0038|
|arc_challenge | 1|none | 25|acc |0.7747|± |0.0122|
| | |none | 25|acc_norm |0.7850|± |0.0120|
|truthfulqa_mc2| 2|none | 0|acc |0.7902|± |0.0134|
|mathqa | 1|none | 0|acc |0.4030|± | 0.009|
| | |none | 0|acc_norm |0.4034|± | 0.009|
|pubmedqa | 1|none | 0|acc |0.6860|± |0.0208|
|boolq | 2|none | 0|acc |0.8401|± |0.0064|
Original (VLLM) Evaluations
| Tasks |Version| Filter |n-shot| Metric |Value | |Stderr|
|--------------|------:|----------------|-----:|-----------|-----:|---|-----:|
|gsm8k | 3|strict-match | 5|exact_match|0.7528|± |0.0119|
| | |flexible-extract| 5|exact_match|0.7521|± |0.0119|
|hellaswag | 1|none | 10|acc |0.8117|± |0.0039|
| | |none | 10|acc_norm |0.9167|± |0.0028|
|winogrande | 1|none | 5|acc |0.8682|± |0.0095|
|mmlu | N/A|none | 0|acc |0.6448|± |0.0038|
|arc_challenge | 1|none | 25|acc |0.7688|± |0.0123|
| | |none | 25|acc_norm |0.7730|± |0.0122|
|truthfulqa_mc2| 2|none | 0|acc |0.7895|± |0.0133|
|mathqa | 1|none | 0|acc |0.4000|± | 0.009|
| | |none | 0|acc_norm |0.4003|± | 0.009|
|pubmedqa | 1|none | 0|acc |0.6680|± |0.0211|
|boolq | 2|none | 0|acc |0.8346|± |0.0065|
Citations
- saltlux
- mlabonne
- jondurbin & Replete-AI
- bartowski & TheBloke
If you use UNA models dont forget to cite:
@misc{unathepitbull21b,
title={ThePitbull: Uniform Neural Alignment},
author={Xavier Murias},
year={2024},
publisher = {Juanako.AI},
journal = {HuggingFace repository},
howpublished = {\url{https://huggingface.co/fblgit/UNA-ThePitbull-21.4-v1}},
}