zephyr-7b-dpo-full-prometheus_consistent-high-curriculum

This model is a fine-tuned version of alignment-handbook/zephyr-7b-sft-full on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.5203
• Rewards/chosen: -0.8639
• Rewards/rejected: -1.8895
• Rewards/accuracies: 0.7672
• Rewards/margins: 1.0256
• Logps/rejected: -408.0299
• Logps/chosen: -361.9968
• Logits/rejected: 0.2281
• Logits/chosen: -0.7985

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 8
• eval_batch_size: 8
• seed: 55
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 128
• total_eval_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6603	0.1143	50	0.6510	-0.0704	-0.2218	0.7069	0.1514	-241.2574	-282.6420	-2.3962	-2.4982
0.6072	0.2286	100	0.5890	-0.1773	-0.6076	0.7457	0.4303	-279.8370	-293.3325	-2.2765	-2.4106
0.5601	0.3429	150	0.5646	-0.8683	-1.7726	0.7414	0.9043	-396.3318	-362.4335	-0.2023	-1.0345
0.5553	0.4571	200	0.5395	-1.0446	-1.9038	0.7414	0.8592	-409.4591	-380.0676	0.1453	-0.7609
0.5165	0.5714	250	0.5305	-0.7966	-1.8137	0.7543	1.0172	-400.4511	-355.2614	0.3109	-0.7944
0.5262	0.6857	300	0.5260	-0.9894	-1.9874	0.7586	0.9981	-417.8188	-374.5410	0.7809	-0.4146
0.5269	0.8	350	0.5224	-0.9044	-1.9323	0.7716	1.0279	-412.3088	-366.0488	0.4302	-0.6784
0.5243	0.9143	400	0.5203	-0.8639	-1.8895	0.7672	1.0256	-408.0299	-361.9968	0.2281	-0.7985

Framework versions

• Transformers 4.44.0.dev0
• Pytorch 2.1.2
• Datasets 2.20.0
• Tokenizers 0.19.1