deberta-v3-small-kaggle-mlm

This model is a fine-tuned version of microsoft/deberta-v3-small on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.6169

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
3.0931	1.0	6848	2.8467
2.6186	2.0	13696	2.4089
2.3498	3.0	20544	2.2224
2.2399	4.0	27392	2.1105
2.1226	5.0	34240	2.0204
2.0768	6.0	41088	1.9402
2.0251	7.0	47936	1.8767
1.9587	8.0	54784	1.8527
1.9209	9.0	61632	1.8108
1.8829	10.0	68480	1.8113
1.8454	11.0	75328	1.7698
1.8077	12.0	82176	1.7504
1.7991	13.0	89024	1.7390
1.7896	14.0	95872	1.7138
1.7608	15.0	102720	1.6847
1.7636	16.0	109568	1.6863
1.7416	17.0	116416	1.6816
1.7363	18.0	123264	1.6651
1.7013	19.0	130112	1.6465
1.6828	20.0	136960	1.6528
1.6889	21.0	143808	1.6406
1.6882	22.0	150656	1.6358
1.6742	23.0	157504	1.6338
1.6657	24.0	164352	1.6062
1.6685	25.0	171200	1.6086
1.6701	26.0	178048	1.6256
1.6755	27.0	184896	1.6186
1.6505	28.0	191744	1.6013
1.6573	29.0	198592	1.6108
1.6497	30.0	205440	1.6009

Framework versions

• Transformers 4.41.2
• Pytorch 2.3.1+cu121
• Datasets 2.19.2
• Tokenizers 0.19.1