bert-large-uncased-nsp-2000-1e-06-8

This model is a fine-tuned version of bert-large-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5950

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 32
• eval_batch_size: 1024
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	63	0.7048
No log	2.0	126	0.6967
No log	3.0	189	0.6940
0.7137	4.0	252	0.6924
0.7137	5.0	315	0.6917
0.7137	6.0	378	0.6908
0.7023	7.0	441	0.6899
0.7023	8.0	504	0.6890
0.7023	9.0	567	0.6877
0.7027	10.0	630	0.6861
0.7027	11.0	693	0.6838
0.7027	12.0	756	0.6810
0.6984	13.0	819	0.6770
0.6984	14.0	882	0.6711
0.6984	15.0	945	0.6639
0.6778	16.0	1008	0.6549
0.6778	17.0	1071	0.6465
0.6778	18.0	1134	0.6391
0.6778	19.0	1197	0.6324
0.6535	20.0	1260	0.6271
0.6535	21.0	1323	0.6214
0.6535	22.0	1386	0.6153
0.6335	23.0	1449	0.6111
0.6335	24.0	1512	0.6059
0.6335	25.0	1575	0.6026
0.6146	26.0	1638	0.5998
0.6146	27.0	1701	0.5977
0.6146	28.0	1764	0.5962
0.6011	29.0	1827	0.5953
0.6011	30.0	1890	0.5950

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.1+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1