bart-qmsum-meeting-summarization

This model is a fine-tuned version of sshleifer/distilbart-cnn-12-6 on the QMSum dataset. It achieves the following results on the evaluation set:

• Loss: 4.3354
• Rouge1: 39.5539
• Rouge2: 12.1134
• Rougel: 23.9163
• Rougelsum: 36.0299
• Gen Len: 117.225

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-07
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 200
• label_smoothing_factor: 0.1

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
5.5573	2.17	100	5.4074	23.6282	4.1122	14.584	21.2263	84.75
5.4721	4.35	200	5.2899	24.61	4.272	15.2096	22.2997	87.2
5.3407	6.52	300	5.1360	25.8272	4.3314	15.9926	23.3416	87.95
5.1527	8.7	400	4.9751	27.7207	5.31	16.7055	24.8357	88.35
5.0058	10.87	500	4.8372	30.1847	6.8615	18.934	27.2424	89.95
4.8807	13.04	600	4.7488	33.1208	9.1784	20.655	30.1198	101.3
4.7931	15.22	700	4.6891	33.2266	8.4253	20.0334	30.4093	108.925
4.7272	17.39	800	4.6467	35.0475	9.326	21.0655	31.8413	111.7
4.6904	19.57	900	4.6102	34.869	9.6046	21.395	32.4346	115.05
4.6547	21.74	1000	4.5829	36.3392	10.9936	22.1524	33.6863	119.875
4.594	23.91	1100	4.5602	35.9717	10.3827	21.6118	32.8302	119.5
4.5714	26.09	1200	4.5424	36.3656	10.6282	22.2187	33.6494	118.0
4.542	28.26	1300	4.5256	36.7386	10.615	22.2487	34.1927	115.675
4.5092	30.43	1400	4.5116	37.1597	10.7751	22.6747	34.396	118.55
4.5031	32.61	1500	4.4981	37.6108	10.9732	22.8342	34.6833	117.125
4.4682	34.78	1600	4.4875	37.5057	11.1328	22.8973	34.7114	117.65
4.4387	36.96	1700	4.4775	38.1278	11.3597	23.1307	35.1869	115.65
4.4085	39.13	1800	4.4682	37.9578	11.4355	23.1149	35.4961	119.6
4.4166	41.3	1900	4.4592	38.1467	11.3208	23.045	35.0824	120.05
4.3971	43.48	2000	4.4517	37.9922	11.5071	23.3983	34.6918	114.425
4.3638	45.65	2100	4.4438	38.1666	11.4985	23.5518	35.1484	117.2
4.3522	47.83	2200	4.4377	37.7572	11.3984	23.4437	35.0453	113.725
4.3398	50.0	2300	4.4320	38.5833	11.4575	23.6411	35.3437	116.125
4.3341	52.17	2400	4.4247	38.2705	12.0374	23.5807	34.9985	110.8
4.3024	54.35	2500	4.4201	39.0206	12.2041	23.4394	35.6291	114.5
4.3117	56.52	2600	4.4147	38.6555	12.1079	23.5655	35.5287	111.325
4.2659	58.7	2700	4.4107	39.2235	12.025	23.934	36.2243	113.3
4.2946	60.87	2800	4.4055	39.0301	12.1833	23.8999	36.0487	110.325
4.2431	63.04	2900	4.4009	39.0498	12.3215	23.9686	36.0277	112.775
4.2439	65.22	3000	4.3968	38.8786	12.0985	23.8308	35.8575	115.175
4.2244	67.39	3100	4.3922	38.7614	12.1721	23.7736	35.6744	113.55
4.235	69.57	3200	4.3895	38.6858	11.3994	23.6392	35.3456	114.125
4.2064	71.74	3300	4.3859	39.0258	12.0435	24.2528	35.8378	113.5
4.1934	73.91	3400	4.3835	39.0467	11.5556	23.6704	35.5643	111.5
4.1859	76.09	3500	4.3800	38.776	11.729	24.1254	35.3894	112.9
4.1762	78.26	3600	4.3775	38.9465	11.9112	23.8123	35.5453	114.125
4.1848	80.43	3700	4.3744	39.2783	11.6539	23.8236	35.8465	110.225
4.1386	82.61	3800	4.3730	38.8894	11.4784	23.7534	35.5464	113.15
4.1483	84.78	3900	4.3710	39.2734	12.0285	23.8171	35.6884	115.95
4.1428	86.96	4000	4.3688	39.6134	12.0616	23.7454	36.0363	113.375
4.133	89.13	4100	4.3663	38.935	11.4781	23.8766	35.4061	114.15
4.1211	91.3	4200	4.3648	39.1488	11.8399	23.9935	35.3107	113.975
4.1076	93.48	4300	4.3650	38.9764	11.9963	23.4994	35.7214	116.25
4.121	95.65	4400	4.3597	38.9418	11.8416	24.0272	35.6597	111.325
4.0936	97.83	4500	4.3602	39.266	12.5616	24.2046	36.1883	114.275
4.0841	100.0	4600	4.3588	39.4659	12.2132	24.0521	36.249	115.475
4.0768	102.17	4700	4.3578	39.4167	12.0587	24.025	35.9668	114.375
4.0711	104.35	4800	4.3541	39.6943	12.1095	24.0925	36.3496	115.65
4.072	106.52	4900	4.3539	40.2024	12.4618	24.2863	36.8844	113.475
4.0646	108.7	5000	4.3540	39.4299	11.8085	23.686	36.0454	113.975
4.0508	110.87	5100	4.3517	39.9217	11.9379	24.2299	36.6362	115.5
4.0549	113.04	5200	4.3498	40.3496	12.2558	24.0271	36.9715	112.5
4.0428	115.22	5300	4.3497	40.1349	12.0628	24.0622	36.9169	113.95
4.0391	117.39	5400	4.3480	40.1209	12.3587	24.3456	36.8411	116.025
4.0195	119.57	5500	4.3474	39.5209	12.1325	24.2622	36.4357	111.975
4.0054	121.74	5600	4.3468	40.2885	12.4453	24.2373	36.932	117.375
4.0286	123.91	5700	4.3465	39.3943	11.8399	23.9786	35.991	116.475
4.005	126.09	5800	4.3459	38.7442	11.7408	23.8948	35.3673	117.625
3.991	128.26	5900	4.3444	39.6276	12.1549	23.9542	36.3832	115.675
4.0137	130.43	6000	4.3427	39.8331	12.2687	24.187	36.6144	115.475
3.9755	132.61	6100	4.3438	39.1907	12.1033	24.2339	35.9126	114.525
4.0134	134.78	6200	4.3422	39.4298	11.862	24.0847	35.5744	115.025
3.9935	136.96	6300	4.3416	39.4158	11.6968	23.9636	35.8155	114.35
3.9606	139.13	6400	4.3409	39.1239	11.7046	23.6846	36.0431	114.775
3.9834	141.3	6500	4.3404	39.6375	12.2746	24.2636	36.1425	116.175
3.9687	143.48	6600	4.3409	39.1494	12.1404	24.0778	35.4932	118.05
3.9861	145.65	6700	4.3394	39.6258	12.2497	23.9662	36.4054	116.8
3.9755	147.83	6800	4.3400	39.3121	11.7831	23.6584	35.9636	118.125
3.9591	150.0	6900	4.3390	39.6957	11.9406	24.0599	36.3021	114.9
3.9599	152.17	7000	4.3389	39.4271	11.4159	24.1437	35.9056	115.8
3.9456	154.35	7100	4.3384	39.4862	11.726	23.883	35.9839	116.375
3.9341	156.52	7200	4.3386	39.6915	11.8028	24.346	36.406	116.425
3.9648	158.7	7300	4.3383	39.9311	11.7135	23.985	36.2617	118.075
3.9486	160.87	7400	4.3372	39.8375	12.0014	24.0969	36.5902	118.8
3.9533	163.04	7500	4.3371	40.2678	12.3137	24.1916	37.1632	118.075
3.9344	165.22	7600	4.3369	39.5588	11.6805	24.1474	36.2021	114.875
3.9314	167.39	7700	4.3368	39.8649	11.9824	24.5459	36.3921	113.65
3.9558	169.57	7800	4.3363	39.8428	12.0892	24.0175	36.67	112.7
3.928	171.74	7900	4.3364	39.2281	11.8456	23.7212	36.2005	113.95
3.9351	173.91	8000	4.3363	39.9798	12.4387	23.7687	36.6472	115.45
3.9326	176.09	8100	4.3363	39.9772	12.1193	24.1518	36.5791	117.4
3.9387	178.26	8200	4.3363	39.8629	12.1719	23.9446	36.345	115.075
3.9204	180.43	8300	4.3358	39.9738	12.3072	23.8641	36.4802	116.3
3.9418	182.61	8400	4.3357	40.1451	12.4144	24.1553	36.4251	116.025
3.9289	184.78	8500	4.3357	39.7241	12.0543	24.0752	36.0847	115.8
3.9176	186.96	8600	4.3358	39.7969	12.0967	24.123	36.2664	118.6
3.9097	189.13	8700	4.3356	39.4096	11.9872	24.0609	35.8662	117.2
3.938	191.3	8800	4.3354	39.4695	11.9343	24.0295	35.9372	117.025
3.9239	193.48	8900	4.3352	39.3231	12.0965	23.9131	35.9555	117.275
3.91	195.65	9000	4.3354	39.5932	12.1808	23.9233	36.0864	116.925
3.9234	197.83	9100	4.3354	39.5539	12.1134	23.9163	36.0299	117.225
3.9263	200.0	9200	4.3354	39.5539	12.1134	23.9163	36.0299	117.225

Framework versions

• Transformers 4.18.0
• Pytorch 1.11.0+cu113
• Datasets 2.1.0
• Tokenizers 0.12.1