GIZ/CONDITIONAL-multilabel-bge_f

CONDITIONAL-multilabel-bge

This model is a fine-tuned version of BAAI/bge-base-en-v1.5 on the Policy-Classification dataset. It achieves the following results on the evaluation set:

• Loss: 0.5295
• Precision-micro: 0.5138
• Precision-samples: 0.1866
• Precision-weighted: 0.5169
• Recall-micro: 0.7378
• Recall-samples: 0.1874
• Recall-weighted: 0.7378
• F1-micro: 0.6058
• F1-samples: 0.1852
• F1-weighted: 0.6065

Model description

The purpose of this model is to predict multiple labels simultaneously from a given input data. Specifically, the model will predict 2 labels - ConditionalLabel, UnconditionalLabel - that are relevant to a particular task or application

• Conditional: In context of climate policy documents if certain Target/Action/Plan/Policy commitment is being made conditionally.
• Unconditional: In context of climate policy documents if certain Target/Action/Plan/Policy commitment is being made unconditionally.

Intended uses & limitations

The dataset sometimes does not include the sub-heading/heading which indicates that the paragraph belongs to Conditional/Unconditional category. But has been copied from the relevant document from those sub-headings. This makes the assessment of Conditonality very difficult. Annotator when given only the paragraph without the full long context had a difficulty in assessing the conditionality of commitments being made in paragraph.

Training and evaluation data

• Training Dataset: 5901

  Class	Positive Count of Class
  ConditionalLabel	1986
  UnconditionalLabel	1312

• Validation Dataset: 1190

  Class	Positive Count of Class
  ConditionalLabel	192
  UnconditionalLabel	136

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 4.02e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 200
• num_epochs: 6

Training results

Training Loss	Epoch	Step	Validation Loss	Precision-micro	Precision-samples	Precision-weighted	Recall-micro	Recall-samples	Recall-weighted	F1-micro	F1-samples	F1-weighted
0.5361	1.0	369	0.4405	0.3405	0.1655	0.4102	0.6311	0.1622	0.6311	0.4423	0.1622	0.4503
0.3692	2.0	738	0.3437	0.4631	0.1794	0.4929	0.6890	0.1761	0.6890	0.5539	0.1762	0.5604
0.182	3.0	1107	0.3915	0.4702	0.1857	0.4871	0.7470	0.1891	0.7470	0.5771	0.1854	0.5800
0.0757	4.0	1476	0.4713	0.4960	0.1882	0.4986	0.7530	0.1908	0.7530	0.5981	0.1877	0.5987
0.0298	5.0	1845	0.4971	0.5161	0.1840	0.5184	0.7317	0.1857	0.7317	0.6053	0.1829	0.6058
0.0152	6.0	2214	0.5295	0.5138	0.1866	0.5169	0.7378	0.1874	0.7378	0.6058	0.1852	0.6065

label	precision	recall	f1-score	support
ConditionalLabel	0.490	0.760	0.595	192.0
UnconditionalLabel	0.555	0.706	0.621	136.0

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• Carbon Emitted: 0.02845 kg of CO2
• Hours Used: 0.702 hours

Training Hardware

• On Cloud: yes
• GPU Model: 1 x Tesla T4
• CPU Model: Intel(R) Xeon(R) CPU @ 2.00GHz
• RAM Size: 12.67 GB

Framework versions

• Transformers 4.38.1
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2