BL-pythia-31m-simpleRW-lite-2048-scratch

This model is a fine-tuned version of EleutherAI/pythia-31m on the None dataset. It achieves the following results on the evaluation set:

• Loss: 4.7136
• Accuracy: 0.2662

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

2040 ***** eval metrics *****
2041   epoch                   =        3.0
2042   eval_accuracy           =     0.2668
2043   eval_loss               =     4.7076
2044   eval_runtime            = 0:00:21.04
2045   eval_samples            =        500
2046   eval_samples_per_second =     23.759
2047   eval_steps_per_second   =      11.88
2048   perplexity              =   110.7897

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 2
• eval_batch_size: 2
• seed: 80085
• gradient_accumulation_steps: 64
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.99) and epsilon=1e-07
• lr_scheduler_type: inverse_sqrt
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
7.0159	0.13	100	7.1022	0.1180
6.2257	0.27	200	6.3526	0.1508
5.8611	0.4	300	5.9888	0.1735
5.5514	0.54	400	5.7552	0.1855
5.3824	0.67	500	5.5883	0.1948
5.344	0.81	600	5.4697	0.2017
5.1925	0.94	700	5.3717	0.2073
5.0814	1.08	800	5.2932	0.2121
5.0865	1.21	900	5.2280	0.2162
4.9602	1.35	1000	5.1672	0.2207
4.957	1.48	1100	5.1144	0.2247
4.8489	1.62	1200	5.0617	0.2299
4.79	1.75	1300	5.0122	0.2349
4.8005	1.89	1400	4.9637	0.2400
4.7409	2.02	1500	4.9216	0.2448
4.6674	2.16	1600	4.8815	0.2488
4.6729	2.29	1700	4.8475	0.2526
4.7071	2.43	1800	4.8156	0.2555
4.4937	2.56	1900	4.7841	0.2588
4.5153	2.7	2000	4.7573	0.2615
4.5512	2.83	2100	4.7345	0.2637
4.5153	2.96	2200	4.7136	0.2662

Framework versions

• Transformers 4.34.0.dev0
• Pytorch 2.2.0.dev20230915+cu118
• Datasets 2.14.5
• Tokenizers 0.13.3