mms-1b-all-bem-natbed-combined

This model is a fine-tuned version of facebook/mms-1b-all on the GENBED - BEM dataset. It achieves the following results on the evaluation set:

• Loss: 0.5565
• Wer: 0.4959

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 30.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
8.2239	0.1252	100	1.3301	0.8960
0.8992	0.2503	200	0.7774	0.6657
0.8263	0.3755	300	0.7550	0.5891
0.7846	0.5006	400	0.6995	0.5665
0.9046	0.6258	500	0.6796	0.5546
0.7688	0.7509	600	0.6881	0.5436
0.7139	0.8761	700	0.6999	0.5557
0.7922	1.0013	800	0.6811	0.5421
0.7929	1.1264	900	0.6760	0.5434
0.7508	1.2516	1000	0.6555	0.5660
0.7534	1.3767	1100	0.6411	0.5372
0.7316	1.5019	1200	0.6420	0.5320
0.7147	1.6270	1300	0.6726	0.5294
0.6734	1.7522	1400	0.6308	0.5253
0.7084	1.8773	1500	0.6205	0.5439
0.6714	2.0025	1600	0.6119	0.5231
0.6888	2.1277	1700	0.6350	0.5167
0.6871	2.2528	1800	0.6183	0.5118
0.6882	2.3780	1900	0.5974	0.5333
0.6769	2.5031	2000	0.5995	0.5301
0.6801	2.6283	2100	0.5880	0.5378
0.6695	2.7534	2200	0.5973	0.5051
0.6557	2.8786	2300	0.6027	0.5056
0.6525	3.0038	2400	0.5946	0.4996
0.6829	3.1289	2500	0.5882	0.4998
0.6627	3.2541	2600	0.6010	0.4985
0.6146	3.3792	2700	0.5770	0.5009
0.6205	3.5044	2800	0.5739	0.5021
0.7025	3.6295	2900	0.5806	0.5224
0.6379	3.7547	3000	0.6210	0.5064
0.6104	3.8798	3100	0.5702	0.5034
0.6607	4.0050	3200	0.5756	0.4891
0.6776	4.1302	3300	0.5679	0.4886
0.6343	4.2553	3400	0.5598	0.4899
0.5818	4.3805	3500	0.5807	0.4964
0.6085	4.5056	3600	0.5932	0.4915
0.6648	4.6308	3700	0.5580	0.4860
0.6359	4.7559	3800	0.5565	0.4959
0.6139	4.8811	3900	0.5605	0.4886
0.5995	5.0063	4000	0.5720	0.4803
0.6349	5.1314	4100	0.5506	0.5012
0.6134	5.2566	4200	0.5603	0.4784
0.5989	5.3817	4300	0.5714	0.4844
0.6083	5.5069	4400	0.5698	0.4758

Framework versions

• Transformers 4.46.0.dev0
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.0