Sheared-LLaMA-1.3B-dpo-full-3-epoch-hydrox-safe

This model is a fine-tuned version of princeton-nlp/Sheared-LLaMA-1.3B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0041
• Rewards/chosen: 1.7270
• Rewards/rejected: -15.3712
• Rewards/accuracies: 0.9983
• Rewards/margins: 17.0982
• Logps/rejected: -656.3423
• Logps/chosen: -371.7201
• Logits/rejected: 2.3459
• Logits/chosen: 0.3641

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 64
• total_eval_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6612	0.03	100	0.6499	0.0765	-0.0151	0.8300	0.0916	-502.7813	-388.2253	3.2379	0.9032
0.4585	0.07	200	0.4458	0.5224	-0.1242	0.9301	0.6466	-503.8723	-383.7663	3.2494	0.9081
0.2519	0.1	300	0.2540	1.2036	-0.4814	0.9470	1.6851	-507.4445	-376.9535	3.2790	0.9127
0.17	0.14	400	0.1751	1.5794	-1.0033	0.9562	2.5827	-512.6629	-373.1959	3.3007	0.9173
0.1179	0.17	500	0.1215	1.8423	-2.0791	0.9588	3.9214	-523.4217	-370.5673	3.2925	0.9104
0.1032	0.2	600	0.1078	2.0902	-2.7647	0.9596	4.8549	-530.2773	-368.0876	3.2574	0.9180
0.0614	0.24	700	0.0881	2.2830	-3.4190	0.9638	5.7021	-536.8207	-366.1595	3.2243	0.9190
0.0666	0.27	800	0.0751	2.3690	-4.0591	0.9689	6.4281	-543.2214	-365.2995	3.1788	0.9025
0.0706	0.31	900	0.0662	2.4002	-4.5254	0.9722	6.9257	-547.8843	-364.9874	3.1624	0.9102
0.0711	0.34	1000	0.0577	2.4230	-4.9179	0.9764	7.3409	-551.8096	-364.7598	3.1467	0.9093
0.0623	0.37	1100	0.0572	2.4840	-5.3620	0.9773	7.8459	-556.2499	-364.1504	3.1186	0.9011
0.0443	0.41	1200	0.0526	2.4237	-5.4784	0.9798	7.9021	-557.4146	-364.7530	3.1196	0.8961
0.0416	0.44	1300	0.0477	2.3874	-6.2247	0.9823	8.6120	-564.8768	-365.1163	3.0683	0.8720
0.0365	0.48	1400	0.0448	2.2887	-6.8360	0.9806	9.1246	-570.9899	-366.1031	3.0491	0.8667
0.0341	0.51	1500	0.0442	2.2795	-6.9547	0.9848	9.2343	-572.1777	-366.1945	3.0299	0.8500
0.0406	0.54	1600	0.0414	2.0896	-7.0003	0.9848	9.0899	-572.6334	-368.0941	3.0437	0.8442
0.0427	0.58	1700	0.0387	2.0380	-7.1141	0.9857	9.1521	-573.7712	-368.6102	3.0458	0.8383
0.0225	0.61	1800	0.0421	2.2150	-7.1052	0.9891	9.3203	-573.6826	-366.8395	3.0443	0.8362
0.0298	0.65	1900	0.0364	2.0854	-7.7136	0.9882	9.7990	-579.7668	-368.1361	3.0306	0.8392
0.0255	0.68	2000	0.0353	2.1351	-7.6852	0.9907	9.8203	-579.4824	-367.6387	3.0204	0.8292
0.019	0.71	2100	0.0296	2.1215	-8.1790	0.9916	10.3005	-584.4203	-367.7745	3.0052	0.8412
0.0198	0.75	2200	0.0248	2.1218	-8.4302	0.9907	10.5520	-586.9324	-367.7719	2.9878	0.8183
0.0192	0.78	2300	0.0238	2.0950	-8.2588	0.9924	10.3538	-585.2184	-368.0402	2.9758	0.7942
0.0191	0.82	2400	0.0213	2.1701	-8.6399	0.9941	10.8101	-589.0295	-367.2885	2.9719	0.8049
0.0215	0.85	2500	0.0224	2.1220	-9.1960	0.9933	11.3180	-594.5902	-367.7695	2.9391	0.7799
0.0579	0.88	2600	0.0193	2.0368	-9.3428	0.9933	11.3796	-596.0587	-368.6217	2.9297	0.7933
0.0163	0.92	2700	0.0180	1.9057	-9.4956	0.9941	11.4013	-597.5867	-369.9328	2.9114	0.7628
0.019	0.95	2800	0.0194	1.9915	-9.4265	0.9933	11.4179	-596.8949	-369.0752	2.9223	0.7736
0.0166	0.99	2900	0.0182	2.0770	-9.1954	0.9958	11.2724	-594.5848	-368.2201	2.9186	0.7592
0.0121	1.02	3000	0.0180	1.9094	-9.4964	0.9941	11.4059	-597.5947	-369.8957	2.8957	0.7557
0.011	1.05	3100	0.0150	2.0009	-9.9345	0.9966	11.9354	-601.9758	-368.9812	2.8560	0.7294
0.0106	1.09	3200	0.0139	2.0861	-9.6153	0.9966	11.7014	-598.7830	-368.1290	2.8565	0.7071
0.0095	1.12	3300	0.0134	1.9755	-10.3936	0.9958	12.3691	-606.5661	-369.2344	2.8290	0.7083
0.0115	1.16	3400	0.0129	1.9719	-10.3851	0.9949	12.3569	-606.4811	-369.2712	2.8212	0.7184
0.0152	1.19	3500	0.0124	2.0357	-10.2131	0.9958	12.2488	-604.7615	-368.6329	2.8217	0.7140
0.01	1.22	3600	0.0116	2.0147	-10.9243	0.9966	12.9390	-611.8733	-368.8428	2.7589	0.6517
0.0135	1.26	3700	0.0116	1.9527	-10.8649	0.9966	12.8176	-611.2795	-369.4628	2.8017	0.7064
0.0078	1.29	3800	0.0112	1.7362	-11.5598	0.9966	13.2960	-618.2281	-371.6281	2.7623	0.6879
0.0114	1.33	3900	0.0106	1.8313	-11.3667	0.9983	13.1980	-616.2969	-370.6765	2.7616	0.6728
0.0077	1.36	4000	0.0101	1.9160	-11.5484	0.9992	13.4645	-618.1147	-369.8296	2.7534	0.6694
0.0057	1.39	4100	0.0098	1.8898	-11.3187	0.9983	13.2085	-615.8172	-370.0915	2.7553	0.6617
0.0056	1.43	4200	0.0091	2.0721	-11.6748	0.9992	13.7469	-619.3782	-368.2689	2.7234	0.6265
0.006	1.46	4300	0.0088	1.8416	-12.1884	0.9983	14.0300	-624.5148	-370.5739	2.7058	0.6225
0.0071	1.5	4400	0.0083	2.0151	-11.7393	0.9983	13.7544	-620.0233	-368.8386	2.7124	0.6231
0.0101	1.53	4500	0.0083	2.0864	-11.5153	0.9992	13.6016	-617.7830	-368.1264	2.7206	0.6407
0.0054	1.56	4600	0.0083	1.9930	-11.3424	0.9975	13.3354	-616.0542	-369.0597	2.7246	0.6099
0.0116	1.6	4700	0.0080	1.9298	-11.3167	0.9975	13.2464	-615.7971	-369.6923	2.7200	0.6008
0.0116	1.63	4800	0.0074	1.8809	-11.4685	0.9975	13.3494	-617.3154	-370.1813	2.6917	0.5698
0.0087	1.67	4900	0.0073	1.8993	-11.8845	0.9983	13.7838	-621.4749	-369.9968	2.6861	0.5798
0.0031	1.7	5000	0.0072	1.8755	-12.3032	0.9975	14.1787	-625.6624	-370.2348	2.6435	0.5411
0.0115	1.73	5100	0.0076	1.9283	-11.9068	0.9958	13.8351	-621.6979	-369.7066	2.6527	0.5393
0.0065	1.77	5200	0.0074	1.9870	-11.9105	0.9949	13.8975	-621.7357	-369.1199	2.6790	0.5763
0.006	1.8	5300	0.0068	1.7994	-12.4601	0.9958	14.2595	-627.2310	-370.9959	2.6264	0.5393
0.0076	1.84	5400	0.0064	2.0449	-12.2057	0.9966	14.2506	-624.6871	-368.5407	2.6409	0.5465
0.0042	1.87	5500	0.0062	1.9941	-12.4399	0.9983	14.4340	-627.0295	-369.0491	2.6332	0.5433
0.0079	1.9	5600	0.0061	1.9119	-12.4000	0.9983	14.3118	-626.6300	-369.8711	2.6300	0.5377
0.0066	1.94	5700	0.0062	2.0544	-12.1682	0.9983	14.2226	-624.3120	-368.4457	2.6248	0.5288
0.0071	1.97	5800	0.0061	2.0943	-12.2702	0.9975	14.3645	-625.3325	-368.0468	2.6248	0.5422
0.0021	2.01	5900	0.0057	1.9195	-12.9348	0.9983	14.8543	-631.9785	-369.7946	2.5712	0.5186
0.0029	2.04	6000	0.0057	1.8384	-13.3904	0.9983	15.2288	-636.5340	-370.6057	2.5405	0.4960
0.0035	2.07	6100	0.0056	1.6150	-14.2858	0.9975	15.9009	-645.4886	-372.8395	2.4718	0.4415
0.0053	2.11	6200	0.0053	1.8268	-13.9429	0.9983	15.7696	-642.0590	-370.7222	2.4921	0.4576
0.0044	2.14	6300	0.0052	1.9443	-13.8117	0.9975	15.7560	-640.7470	-369.5464	2.5079	0.4705
0.0026	2.18	6400	0.0053	2.0456	-13.7455	0.9975	15.7911	-640.0853	-368.5343	2.5139	0.4823
0.0026	2.21	6500	0.0050	2.0028	-13.6496	0.9983	15.6524	-639.1260	-368.9618	2.5135	0.4823
0.0029	2.24	6600	0.0050	1.8856	-13.7926	0.9975	15.6782	-640.5563	-370.1337	2.4828	0.4459
0.0023	2.28	6700	0.0049	1.9422	-14.0760	0.9983	16.0182	-643.3903	-369.5678	2.4698	0.4471
0.003	2.31	6800	0.0048	1.8633	-14.4649	0.9983	16.3282	-647.2790	-370.3570	2.4646	0.4562
0.0058	2.35	6900	0.0049	1.8085	-14.8512	0.9975	16.6597	-651.1427	-370.9051	2.4275	0.4292
0.0032	2.38	7000	0.0048	1.9006	-14.6340	0.9983	16.5346	-648.9703	-369.9842	2.4387	0.4425
0.0018	2.41	7100	0.0047	1.8215	-15.0376	0.9983	16.8592	-653.0067	-370.7746	2.4153	0.4296
0.001	2.45	7200	0.0046	1.8195	-15.0112	0.9983	16.8307	-652.7422	-370.7950	2.4153	0.4248
0.0057	2.48	7300	0.0045	1.8920	-14.4156	0.9983	16.3077	-646.7868	-370.0694	2.4336	0.4234
0.004	2.52	7400	0.0044	1.7826	-14.6522	0.9983	16.4348	-649.1526	-371.1638	2.4101	0.4117
0.0025	2.55	7500	0.0044	1.8202	-14.7043	0.9983	16.5245	-649.6732	-370.7875	2.4040	0.4069
0.0035	2.58	7600	0.0044	1.8712	-14.7562	0.9983	16.6273	-650.1921	-370.2782	2.4019	0.4087
0.002	2.62	7700	0.0043	1.8406	-14.8610	0.9983	16.7017	-651.2407	-370.5836	2.3996	0.4114
0.002	2.65	7800	0.0043	1.8042	-15.0820	0.9992	16.8862	-653.4503	-370.9484	2.3936	0.4147
0.0046	2.69	7900	0.0042	1.8043	-15.2990	0.9983	17.1033	-655.6204	-370.9472	2.3757	0.3993
0.0025	2.72	8000	0.0042	1.8289	-15.3097	0.9983	17.1386	-655.7274	-370.7011	2.3634	0.3853
0.0023	2.75	8100	0.0041	1.7995	-15.2380	0.9983	17.0375	-655.0099	-370.9947	2.3619	0.3779
0.0025	2.79	8200	0.0040	1.8013	-15.2440	0.9983	17.0453	-655.0703	-370.9769	2.3668	0.3827
0.002	2.82	8300	0.0040	1.8040	-15.2101	0.9983	17.0141	-654.7317	-370.9499	2.3660	0.3834
0.0023	2.86	8400	0.0040	1.7441	-15.3132	0.9983	17.0572	-655.7621	-371.5493	2.3498	0.3680
0.002	2.89	8500	0.0040	1.7551	-15.3278	0.9983	17.0828	-655.9080	-371.4393	2.3509	0.3714
0.004	2.92	8600	0.0040	1.7500	-15.3290	0.9983	17.0790	-655.9205	-371.4897	2.3518	0.3701
0.0041	2.96	8700	0.0040	1.7294	-15.3645	0.9983	17.0940	-656.2756	-371.6956	2.3478	0.3660
0.0029	2.99	8800	0.0040	1.7305	-15.3609	0.9983	17.0914	-656.2390	-371.6845	2.3464	0.3647

Framework versions

• Transformers 4.35.0
• Pytorch 2.1.1+cu121
• Datasets 2.14.6
• Tokenizers 0.14.1