GPT-2_para3M_2epoch_256

This model is a fine-tuned version of gpt2 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.1100

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss
4.1873	0.01	500	4.0187
3.5461	0.02	1000	3.4287
3.2706	0.04	1500	3.1495
3.105	0.05	2000	2.9773
2.9885	0.06	2500	2.8566
2.8931	0.07	3000	2.7720
2.8307	0.08	3500	2.7016
2.7912	0.09	4000	2.6474
2.7295	0.11	4500	2.5972
2.6927	0.12	5000	2.5641
2.6756	0.13	5500	2.5248
2.6536	0.14	6000	2.4972
2.6186	0.15	6500	2.4730
2.5947	0.17	7000	2.4492
2.591	0.18	7500	2.4313
2.5706	0.19	8000	2.4172
2.5441	0.2	8500	2.3991
2.5266	0.21	9000	2.3838
2.5259	0.22	9500	2.3740
2.5173	0.24	10000	2.3629
2.5122	0.25	10500	2.3549
2.5004	0.26	11000	2.3409
2.4902	0.27	11500	2.3364
2.4735	0.28	12000	2.3242
2.4784	0.29	12500	2.3193
2.4754	0.31	13000	2.3126
2.4587	0.32	13500	2.3077
2.4613	0.33	14000	2.3050
2.4562	0.34	14500	2.2968
2.4422	0.35	15000	2.2913
2.4307	0.37	15500	2.2870
2.4339	0.38	16000	2.2814
2.445	0.39	16500	2.2801
2.4257	0.4	17000	2.2747
2.425	0.41	17500	2.2709
2.4095	0.42	18000	2.2672
2.4137	0.44	18500	2.2632
2.4284	0.45	19000	2.2601
2.419	0.46	19500	2.2569
2.4221	0.47	20000	2.2504
2.3951	0.48	20500	2.2507
2.4054	0.5	21000	2.2515
2.3977	0.51	21500	2.2442
2.4009	0.52	22000	2.2422
2.3941	0.53	22500	2.2388
2.3909	0.54	23000	2.2349
2.4016	0.55	23500	2.2380
2.389	0.57	24000	2.2326
2.3864	0.58	24500	2.2287
2.3795	0.59	25000	2.2285
2.3817	0.6	25500	2.2266
2.3789	0.61	26000	2.2256
2.3801	0.62	26500	2.2210
2.3687	0.64	27000	2.2189
2.378	0.65	27500	2.2194
2.3735	0.66	28000	2.2157
2.3758	0.67	28500	2.2142
2.3616	0.68	29000	2.2133
2.3731	0.7	29500	2.2085
2.3606	0.71	30000	2.2115
2.3516	0.72	30500	2.2072
2.3551	0.73	31000	2.2067
2.3626	0.74	31500	2.2033
2.3516	0.75	32000	2.2031
2.3658	0.77	32500	2.2008
2.3554	0.78	33000	2.1992
2.3524	0.79	33500	2.1988
2.3509	0.8	34000	2.1996
2.3474	0.81	34500	2.1949
2.3431	0.83	35000	2.1943
2.3413	0.84	35500	2.1907
2.3592	0.85	36000	2.1917
2.3636	0.86	36500	2.1919
2.3529	0.87	37000	2.1881
2.3371	0.88	37500	2.1875
2.3413	0.9	38000	2.1856
2.3463	0.91	38500	2.1839
2.3303	0.92	39000	2.1859
2.3432	0.93	39500	2.1790
2.3455	0.94	40000	2.1801
2.344	0.95	40500	2.1761
2.3442	0.97	41000	2.1759
2.3331	0.98	41500	2.1760
2.3391	0.99	42000	2.1748
2.3275	1.0	42500	2.1760
2.3308	1.01	43000	2.1712
2.3191	1.03	43500	2.1727
2.3182	1.04	44000	2.1682
2.3184	1.05	44500	2.1683
2.3177	1.06	45000	2.1668
2.3163	1.07	45500	2.1643
2.321	1.08	46000	2.1631
2.3164	1.1	46500	2.1655
2.3231	1.11	47000	2.1631
2.3139	1.12	47500	2.1591
2.3223	1.13	48000	2.1588
2.3133	1.14	48500	2.1588
2.2995	1.16	49000	2.1569
2.308	1.17	49500	2.1578
2.3062	1.18	50000	2.1539
2.3203	1.19	50500	2.1538
2.3116	1.2	51000	2.1526
2.294	1.21	51500	2.1520
2.2941	1.23	52000	2.1499
2.3053	1.24	52500	2.1502
2.3154	1.25	53000	2.1507
2.3057	1.26	53500	2.1485
2.3106	1.27	54000	2.1464
2.3035	1.28	54500	2.1457
2.304	1.3	55000	2.1445
2.2985	1.31	55500	2.1439
2.296	1.32	56000	2.1421
2.2917	1.33	56500	2.1411
2.2936	1.34	57000	2.1406
2.2866	1.36	57500	2.1383
2.2973	1.37	58000	2.1396
2.2865	1.38	58500	2.1378
2.2929	1.39	59000	2.1370
2.2858	1.4	59500	2.1351
2.2857	1.41	60000	2.1350
2.3019	1.43	60500	2.1338
2.289	1.44	61000	2.1330
2.2874	1.45	61500	2.1318
2.2858	1.46	62000	2.1305
2.2875	1.47	62500	2.1298
2.2859	1.49	63000	2.1294
2.28	1.5	63500	2.1275
2.2866	1.51	64000	2.1277
2.2851	1.52	64500	2.1281
2.2806	1.53	65000	2.1258
2.2889	1.54	65500	2.1245
2.2745	1.56	66000	2.1249
2.2739	1.57	66500	2.1230
2.2853	1.58	67000	2.1226
2.2773	1.59	67500	2.1228
2.2742	1.6	68000	2.1214
2.2656	1.61	68500	2.1200
2.2756	1.63	69000	2.1194
2.2806	1.64	69500	2.1193
2.271	1.65	70000	2.1186
2.2671	1.66	70500	2.1185
2.2718	1.67	71000	2.1168
2.2781	1.69	71500	2.1172
2.2744	1.7	72000	2.1164
2.2744	1.71	72500	2.1156
2.2603	1.72	73000	2.1154
2.2703	1.73	73500	2.1141
2.267	1.74	74000	2.1141
2.2614	1.76	74500	2.1141
2.263	1.77	75000	2.1133
2.2668	1.78	75500	2.1128
2.2642	1.79	76000	2.1128
2.2637	1.8	76500	2.1128
2.2692	1.82	77000	2.1118
2.2631	1.83	77500	2.1117
2.2567	1.84	78000	2.1116
2.2707	1.85	78500	2.1112
2.2707	1.86	79000	2.1109
2.2664	1.87	79500	2.1114
2.266	1.89	80000	2.1113
2.2645	1.9	80500	2.1108
2.2767	1.91	81000	2.1106
2.274	1.92	81500	2.1102
2.2587	1.93	82000	2.1102
2.2736	1.94	82500	2.1100
2.2633	1.96	83000	2.1102
2.2652	1.97	83500	2.1100
2.2655	1.98	84000	2.1101
2.2683	1.99	84500	2.1100

Framework versions

• Transformers 4.32.0
• Pytorch 2.0.1+cu117
• Datasets 2.14.4
• Tokenizers 0.13.2