bert-large-uncased-nsp-10000-1e-06-16

This model is a fine-tuned version of bert-large-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3296

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 64
• eval_batch_size: 1024
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	157	0.7022
0.7432	2.0	314	0.6874
0.6949	3.0	471	0.6383
0.6413	4.0	628	0.5845
0.6413	5.0	785	0.5452
0.5838	6.0	942	0.4705
0.5095	7.0	1099	0.4255
0.4276	8.0	1256	0.4012
0.3848	9.0	1413	0.3864
0.3848	10.0	1570	0.3709
0.358	11.0	1727	0.3579
0.3262	12.0	1884	0.3495
0.3081	13.0	2041	0.3476
0.3081	14.0	2198	0.3432
0.2827	15.0	2355	0.3390
0.2728	16.0	2512	0.3378
0.2584	17.0	2669	0.3337
0.2506	18.0	2826	0.3375
0.2506	19.0	2983	0.3306
0.2337	20.0	3140	0.3296
0.2196	21.0	3297	0.3327
0.2146	22.0	3454	0.3334
0.2148	23.0	3611	0.3343

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.1+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1