Edit model card

sablo-pebble-mistral-dpo-lora-HelpSteer_binarized-2

This model is a fine-tuned version of sablo/sablo-pebble-mistral on the sablo/HelpSteer_binarized dataset. It achieves the following results on the evaluation set:

  • Loss: 0.5195
  • Rewards/chosen: -1.3821
  • Rewards/rejected: -2.4510
  • Rewards/accuracies: 0.7358
  • Rewards/margins: 1.0689
  • Logps/rejected: -158.5470
  • Logps/chosen: -147.7195
  • Logits/rejected: -2.0952
  • Logits/chosen: -2.1922

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-06
  • train_batch_size: 4
  • eval_batch_size: 8
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss Rewards/chosen Rewards/rejected Rewards/accuracies Rewards/margins Logps/rejected Logps/chosen Logits/rejected Logits/chosen
0.65 0.2 200 0.6563 0.1070 0.0177 0.6509 0.0893 -76.2561 -98.0835 -2.0464 -2.1421
0.456 0.39 400 0.5446 -1.2305 -1.8748 0.7217 0.6444 -139.3410 -142.6661 -2.1203 -2.2102
0.4388 0.59 600 0.5325 -1.8012 -2.8927 0.7123 1.0915 -173.2708 -161.6904 -2.1017 -2.1954
0.6137 0.79 800 0.5198 -1.4487 -2.5199 0.7382 1.0712 -160.8413 -149.9388 -2.0962 -2.1935
0.5866 0.98 1000 0.5195 -1.3821 -2.4510 0.7358 1.0689 -158.5470 -147.7195 -2.0952 -2.1922

Framework versions

  • PEFT 0.7.1
  • Transformers 4.36.2
  • Pytorch 2.0.1+cu118
  • Datasets 2.14.6
  • Tokenizers 0.15.0
Downloads last month
9
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Model tree for dctanner/sablo-pebble-mistral-dpo-lora-HelpSteer_binarized-2

Adapter
(3)
this model

Dataset used to train dctanner/sablo-pebble-mistral-dpo-lora-HelpSteer_binarized-2