zephyr-7b-dpo-full-prometheus_consistent-reward-scale-01

This model is a fine-tuned version of alignment-handbook/zephyr-7b-sft-full on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.5426
• Rewards/chosen: -1.6823
• Rewards/rejected: -3.0793
• Rewards/accuracies: 0.7543
• Rewards/margins: 1.3969
• Logps/rejected: -527.0015
• Logps/chosen: -443.8361
• Logits/rejected: 3.9256
• Logits/chosen: 2.3196

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 8
• eval_batch_size: 8
• seed: 55
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 128
• total_eval_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6745	0.1143	50	0.6652	-0.0243	-0.1726	0.6853	0.1483	-236.3333	-278.0297	-2.4359	-2.5373
0.6243	0.2286	100	0.6148	-0.4908	-1.0266	0.7026	0.5358	-321.7332	-324.6850	-2.3884	-2.4967
0.5613	0.3429	150	0.5828	-1.5541	-2.6485	0.7241	1.0943	-483.9227	-431.0166	1.8439	0.7832
0.5618	0.4571	200	0.5594	-1.3157	-2.5436	0.7457	1.2279	-473.4366	-407.1749	2.6334	1.3535
0.558	0.5714	250	0.5526	-1.3313	-2.5932	0.7629	1.2619	-478.3948	-408.7301	3.1078	1.5316
0.5399	0.6857	300	0.5465	-1.7354	-3.0859	0.75	1.3504	-527.6615	-449.1465	3.8100	2.3192
0.5536	0.8	350	0.5458	-1.8434	-3.1830	0.7672	1.3396	-537.3768	-459.9458	4.1008	2.5496
0.5612	0.9143	400	0.5426	-1.6823	-3.0793	0.7543	1.3969	-527.0015	-443.8361	3.9256	2.3196

Framework versions

• Transformers 4.44.0.dev0
• Pytorch 2.1.2
• Datasets 2.20.0
• Tokenizers 0.19.1