PRIMERA-lora-finetuned

This model is a fine-tuned version of allenai/PRIMERA on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.4326
• Rouge1: 13.8331
• Rouge2: 6.379
• Rougel: 11.3582
• Rougelsum: 13.0901
• Gen Len: 19.0217

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 16
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
1.9399	1.0	4725	1.6257	13.1779	5.0715	10.6386	11.9435	19.3478
1.8124	2.0	9450	1.5723	12.8712	5.1519	10.5072	11.9192	19.3478
1.7698	3.0	14175	1.5261	13.6339	5.5927	10.9273	12.3707	19.3478
1.7171	4.0	18900	1.4990	13.0984	6.164	10.9698	12.1863	19.6739
1.7096	5.0	23625	1.4832	12.7164	5.4214	10.3325	11.8134	19.3478
1.6889	6.0	28350	1.4743	13.0677	4.9162	10.3945	11.9299	19.3478
1.6667	7.0	33075	1.4652	13.8986	6.1753	10.9896	12.8072	19.3478
1.6836	8.0	37800	1.4573	13.0179	5.5771	10.4498	12.2198	19.0217
1.6611	9.0	42525	1.4523	12.8773	5.3502	10.294	11.9712	19.0217
1.6842	10.0	47250	1.4503	13.1982	5.0089	10.4547	12.2554	19.0217
1.6373	11.0	51975	1.4432	13.0444	5.533	10.3895	12.3086	19.0217
1.6882	12.0	56700	1.4408	13.7092	5.9873	11.1084	12.8159	19.3478
1.6707	13.0	61425	1.4395	13.6103	6.2231	10.7643	12.7343	19.0217
1.677	14.0	66150	1.4366	13.5232	6.3114	10.9432	12.7234	19.0217
1.6825	15.0	70875	1.4347	13.7907	6.1456	11.0682	12.803	19.0217
1.6516	16.0	75600	1.4326	13.8331	6.379	11.3582	13.0901	19.0217

Framework versions

• PEFT 0.12.0
• Transformers 4.43.3
• Pytorch 2.2.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1