File size: 4,462 Bytes
2c6e000 064b785 2c6e000 064b785 2c6e000 064b785 2c6e000 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 |
---
license: apache-2.0
library_name: peft
tags:
- alignment-handbook
- generated_from_trainer
- trl
- dpo
- generated_from_trainer
datasets:
- HuggingFaceH4/ultrafeedback_binarized
base_model: mistralai/Mistral-7B-Instruct-v0.2
model-index:
- name: SausageLM-7b-Instruct-v0.01-dpo-qlora
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# SausageLM-7b-Instruct-v0.01-dpo-qlora
This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) on the HuggingFaceH4/ultrafeedback_binarized dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4204
- Rewards/chosen: -1.9644
- Rewards/rejected: -3.5978
- Rewards/accuracies: 0.8020
- Rewards/margins: 1.6333
- Logps/rejected: -778.7791
- Logps/chosen: -552.1046
- Logits/rejected: 1.3639
- Logits/chosen: 0.3998
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 1
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 8
- total_train_batch_size: 16
- total_eval_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1
### Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.4906 | 0.08 | 300 | 0.5340 | -1.1814 | -1.8425 | 0.7310 | 0.6611 | -603.2533 | -473.8014 | -1.6234 | -1.7536 |
| 0.4794 | 0.16 | 600 | 0.4701 | -1.3882 | -2.4799 | 0.7700 | 1.0918 | -666.9945 | -494.4773 | 1.2460 | 0.4450 |
| 0.4519 | 0.24 | 900 | 0.4566 | -1.4239 | -2.6724 | 0.7730 | 1.2485 | -686.2431 | -498.0537 | 1.0803 | 0.1979 |
| 0.4034 | 0.31 | 1200 | 0.4487 | -1.9028 | -3.5170 | 0.7870 | 1.6142 | -770.7061 | -545.9451 | 1.7156 | 0.7244 |
| 0.4193 | 0.39 | 1500 | 0.4420 | -1.8864 | -3.4847 | 0.7840 | 1.5983 | -767.4712 | -544.3021 | 0.9998 | 0.0019 |
| 0.409 | 0.47 | 1800 | 0.4365 | -2.0591 | -3.7221 | 0.7920 | 1.6630 | -791.2130 | -561.5723 | 1.4876 | 0.5341 |
| 0.4037 | 0.55 | 2100 | 0.4334 | -2.1275 | -3.8835 | 0.7970 | 1.7560 | -807.3529 | -568.4110 | 1.9485 | 0.9489 |
| 0.3829 | 0.63 | 2400 | 0.4248 | -1.8791 | -3.4902 | 0.8010 | 1.6111 | -768.0193 | -543.5670 | 1.5421 | 0.5047 |
| 0.47 | 0.71 | 2700 | 0.4211 | -1.8565 | -3.4027 | 0.8030 | 1.5462 | -759.2699 | -541.3088 | 1.5152 | 0.5343 |
| 0.3769 | 0.79 | 3000 | 0.4205 | -1.9199 | -3.5317 | 0.8010 | 1.6119 | -772.1762 | -547.6463 | 1.5142 | 0.5326 |
| 0.3921 | 0.86 | 3300 | 0.4216 | -2.0430 | -3.7240 | 0.8050 | 1.6810 | -791.3992 | -559.9616 | 1.5287 | 0.5531 |
| 0.4249 | 0.94 | 3600 | 0.4204 | -1.9591 | -3.5883 | 0.8000 | 1.6292 | -777.8283 | -551.5704 | 1.3533 | 0.3917 |
### Framework versions
- PEFT 0.7.1
- Transformers 4.36.2
- Pytorch 2.1.2+cu121
- Datasets 2.14.6
- Tokenizers 0.15.0 |