openhermes-mistral-dpo-gptq

This model is a fine-tuned version of TheBloke/OpenHermes-2-Mistral-7B-GPTQ on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.7494
• Rewards/chosen: -1.3887
• Rewards/rejected: -1.3727
• Rewards/accuracies: 0.25
• Rewards/margins: -0.0160
• Logps/rejected: -254.9227
• Logps/chosen: -296.4655
• Logits/rejected: -3.2845
• Logits/chosen: -3.3612

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 2
• training_steps: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6671	0.01	10	0.7157	-0.0276	-0.0086	0.375	-0.0190	-241.2815	-282.8551	-2.8170	-2.9092
0.7091	0.01	20	0.7164	-0.5539	-0.6101	0.5	0.0562	-247.2965	-288.1179	-3.0463	-3.1341
0.6568	0.01	30	0.7347	-0.7194	-0.7557	0.5	0.0363	-248.7529	-289.7729	-3.1075	-3.1928
0.6534	0.02	40	0.7316	-1.3152	-1.3098	0.25	-0.0054	-254.2937	-295.7312	-3.2685	-3.3467
0.8009	0.03	50	0.7494	-1.3887	-1.3727	0.25	-0.0160	-254.9227	-296.4655	-3.2845	-3.3612

Framework versions

• Transformers 4.35.2
• Pytorch 2.1.0+cu121
• Datasets 2.16.1
• Tokenizers 0.15.0