zephyr-7b-dpo-full-ultrabin-reward-scale-01-random

This model is a fine-tuned version of alignment-handbook/zephyr-7b-sft-full on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5400
• Rewards/chosen: -2.0899
• Rewards/rejected: -3.6849
• Rewards/accuracies: 0.7539
• Rewards/margins: 1.5951
• Logps/rejected: -631.1537
• Logps/chosen: -471.6162
• Logits/rejected: 4.0473
• Logits/chosen: 2.8493

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 8
• eval_batch_size: 8
• seed: 55
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 128
• total_eval_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6757	0.1046	50	0.6523	0.0185	-0.0845	0.6914	0.1030	-271.1101	-260.7814	-2.5756	-2.6127
0.5878	0.2092	100	0.5948	-1.0094	-1.7751	0.7305	0.7657	-440.1679	-363.5702	0.4396	0.1394
0.5614	0.3138	150	0.5571	-1.1386	-2.1151	0.7344	0.9765	-474.1727	-376.4938	1.4578	0.8611
0.5467	0.4184	200	0.5479	-1.3072	-2.5365	0.75	1.2293	-516.3164	-393.3519	1.9356	0.9992
0.5401	0.5230	250	0.5436	-1.2459	-2.5137	0.7539	1.2678	-514.0286	-387.2157	2.2974	1.4064
0.5223	0.6276	300	0.5449	-1.7241	-3.1547	0.7461	1.4306	-578.1356	-435.0401	3.0431	2.1942
0.5133	0.7322	350	0.5299	-1.6714	-3.0561	0.7578	1.3847	-568.2688	-429.7706	2.9402	1.9451
0.5234	0.8368	400	0.5391	-2.0819	-3.6352	0.7539	1.5533	-626.1800	-470.8206	4.0879	2.9776
0.5253	0.9414	450	0.5400	-2.0899	-3.6849	0.7539	1.5951	-631.1537	-471.6162	4.0473	2.8493

Framework versions

• Transformers 4.44.0.dev0
• Pytorch 2.1.2
• Datasets 2.20.0
• Tokenizers 0.19.1