quantized / README.md
narekvslife's picture
dpo_0uxjem3c
97a7051
metadata
license: apache-2.0
library_name: peft
tags:
  - trl
  - dpo
  - unsloth
  - generated_from_trainer
base_model: unsloth/llama-3-8b-Instruct-bnb-4bit
model-index:
  - name: dpo
    results: []

dpo

This model is a fine-tuned version of unsloth/llama-3-8b-Instruct-bnb-4bit on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 0.6257
  • Rewards/chosen: 0.8141
  • Rewards/rejected: 0.4945
  • Rewards/accuracies: 0.6431
  • Rewards/margins: 0.3196
  • Logps/rejected: -229.7856
  • Logps/chosen: -249.2073
  • Logits/rejected: -0.6789
  • Logits/chosen: -0.6135

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 0
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 100
  • training_steps: 750
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Rewards/chosen Rewards/rejected Rewards/accuracies Rewards/margins Logps/rejected Logps/chosen Logits/rejected Logits/chosen
0.6904 0.0372 28 0.6811 0.2766 0.2476 0.5770 0.0290 -232.2545 -254.5816 -0.5471 -0.5010
0.6591 0.0745 56 0.6623 0.9939 0.8694 0.5927 0.1245 -226.0365 -247.4085 -0.5351 -0.4798
0.6297 0.1117 84 0.6542 1.1966 0.9862 0.6136 0.2104 -224.8689 -245.3818 -0.4689 -0.4120
0.5985 0.1489 112 0.6540 1.5211 1.2525 0.6087 0.2687 -222.2059 -242.1367 -0.4989 -0.4262
0.6603 0.1862 140 0.6459 0.7737 0.5130 0.6304 0.2607 -229.6009 -249.6110 -0.5779 -0.5054
0.619 0.2234 168 0.6411 0.9352 0.6917 0.6222 0.2435 -227.8137 -247.9963 -0.5842 -0.5261
0.6497 0.2606 196 0.6427 0.8696 0.6404 0.6282 0.2292 -228.3268 -248.6518 -0.5798 -0.5255
0.6014 0.2979 224 0.6397 0.8941 0.6357 0.6263 0.2583 -228.3730 -248.4069 -0.6397 -0.5816
0.594 0.3351 252 0.6361 0.7069 0.4027 0.6319 0.3043 -230.7038 -250.2785 -0.6434 -0.5848
0.5898 0.3723 280 0.6356 1.0373 0.7462 0.6278 0.2911 -227.2686 -246.9745 -0.6340 -0.5714
0.639 0.4096 308 0.6342 0.7199 0.4321 0.6342 0.2878 -230.4095 -250.1490 -0.6956 -0.6293
0.6289 0.4468 336 0.6363 0.4299 0.1879 0.6248 0.2420 -232.8515 -253.0488 -0.6705 -0.6155
0.6304 0.4840 364 0.6321 0.7719 0.5053 0.6435 0.2667 -229.6779 -249.6284 -0.6279 -0.5652
0.6126 0.5213 392 0.6325 0.5194 0.2033 0.6375 0.3161 -232.6973 -252.1539 -0.6785 -0.6117
0.5974 0.5585 420 0.6254 0.7418 0.4269 0.6428 0.3149 -230.4618 -249.9303 -0.6823 -0.6170
0.6185 0.5957 448 0.6267 0.9534 0.6106 0.6409 0.3428 -228.6247 -247.8141 -0.6532 -0.5866
0.604 0.6330 476 0.6284 0.8011 0.4691 0.6394 0.3320 -230.0398 -249.3374 -0.6842 -0.6177
0.6154 0.6702 504 0.6269 0.8353 0.5307 0.6431 0.3046 -229.4234 -248.9947 -0.6705 -0.6051
0.5936 0.7074 532 0.6277 0.7287 0.4206 0.6469 0.3082 -230.5248 -250.0604 -0.6887 -0.6226
0.6291 0.7447 560 0.6260 0.8539 0.5327 0.6439 0.3211 -229.4030 -248.8091 -0.6758 -0.6096
0.6169 0.7819 588 0.6255 0.8797 0.5669 0.6461 0.3127 -229.0613 -248.5513 -0.6690 -0.6041
0.5934 0.8191 616 0.6256 0.8582 0.5399 0.6461 0.3183 -229.3312 -248.7658 -0.6753 -0.6095
0.6004 0.8564 644 0.6257 0.8263 0.5074 0.6450 0.3189 -229.6564 -249.0845 -0.6761 -0.6110
0.6282 0.8936 672 0.6256 0.8133 0.4949 0.6442 0.3184 -229.7819 -249.2152 -0.6748 -0.6101
0.5572 0.9309 700 0.6258 0.8122 0.4938 0.6442 0.3184 -229.7925 -249.2255 -0.6781 -0.6129
0.595 0.9681 728 0.6256 0.8140 0.4943 0.6428 0.3197 -229.7873 -249.2078 -0.6788 -0.6134

Framework versions

  • PEFT 0.11.1
  • Transformers 4.41.2
  • Pytorch 2.3.0+cu121
  • Datasets 2.19.2
  • Tokenizers 0.19.1