t5-small-finetuned-billsum

This model is a fine-tuned version of t5-small on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.3166
• Rouge1: 58.6163
• Rouge2: 41.6107
• Rougel: 51.5177
• Rougelsum: 52.8486
• Gen Len: 62.2894

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
2.5373	0.4219	500	1.7306	44.8884	29.9047	38.8732	39.7098	49.8626
1.8175	0.8439	1000	1.5164	53.4663	36.8936	46.4756	47.6133	57.6953
1.6775	1.2658	1500	1.4401	55.9549	38.7969	48.6918	49.9216	60.5867
1.6	1.6878	2000	1.4016	56.8423	39.5972	49.5877	50.8088	61.5580
1.5717	2.1097	2500	1.3736	57.4282	40.2126	50.1498	51.3818	61.9033
1.5389	2.5316	3000	1.3570	57.6909	40.5046	50.4987	51.7769	62.0116
1.5183	2.9536	3500	1.3426	58.2372	41.1473	51.0517	52.3423	62.1297
1.499	3.3755	4000	1.3310	58.326	41.2564	51.1817	52.4513	62.2423
1.4845	3.7975	4500	1.3232	58.4925	41.5426	51.3865	52.6942	62.2276
1.4888	4.2194	5000	1.3203	58.5475	41.5865	51.4574	52.791	62.2710
1.48	4.6414	5500	1.3166	58.6163	41.6107	51.5177	52.8486	62.2894

Framework versions

• Transformers 4.42.4
• Pytorch 2.3.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1