OSainz commited on
Commit
c62102c
1 Parent(s): 2c9b9c6

First model version

Browse files
README.md CHANGED
@@ -1,3 +1,63 @@
1
  ---
2
- license: cc-by-4.0
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ pipeline_tag: zero-shot-classification
3
+ datasets:
4
+ - snli
5
+ - anli
6
+ - multi_nli
7
+ - multi_nli_mismatch
8
+ - fever
9
  ---
10
+
11
+ # A2T Entailment model
12
+
13
+ **Important:** These pretrained entailment models are intended to be used with the [Ask2Transformers](https://github.com/osainz59/Ask2Transformers) library but are also fully compatible with the `ZeroShotTextClassificationPipeline` from [Transformers](https://github.com/huggingface/Transformers).
14
+
15
+
16
+ Textual Entailment (or Natural Language Inference) has turned out to be a good choice for zero-shot text classification problems [(Yin et al., 2019](https://aclanthology.org/D19-1404/); [Wang et al., 2021](https://arxiv.org/abs/2104.14690); [Sainz and Rigau, 2021)](https://aclanthology.org/2021.gwc-1.6/). Recent research addressed Information Extraction problems with the same idea [(Lyu et al., 2021](https://aclanthology.org/2021.acl-short.42/); [Sainz et al., 2021](https://aclanthology.org/2021.emnlp-main.92/); [Sainz et al., 2022a](), [Sainz et al., 2022b)](https://arxiv.org/abs/2203.13602). The A2T entailment models are first trained with NLI datasets such as MNLI [(Williams et al., 2018)](), SNLI [(Bowman et al., 2015)]() or/and ANLI [(Nie et al., 2020)]() and then fine-tuned to specific tasks that were previously converted to textual entailment format.
17
+
18
+ For more information please, take a look to the [Ask2Transformers]() library or the following published papers:
19
+
20
+ - [Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction (Sainz et al., EMNLP 2021)](https://aclanthology.org/2021.emnlp-main.92/)
21
+ - [Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning (Sainz et al., Findings of NAACL-HLT 2022)]()
22
+
23
+ ## About the model
24
+
25
+ The model name describes the configuration used for training as follows:
26
+
27
+ <!-- $$\text{HiTZ/A2T\_[pretrained\_model]\_[NLI\_datasets]\_[finetune\_datasets]}$$ -->
28
+
29
+ <h3 align="center">HiTZ/A2T_[pretrained_model]_[NLI_datasets]_[finetune_datasets]</h3>
30
+
31
+
32
+ - `pretrained_model`: The checkpoint used for initialization. For example: RoBERTa<sub>large</sub>.
33
+ - `NLI_datasets`: The NLI datasets used for pivot training.
34
+ - `S`: Standford Natural Language Inference (SNLI) dataset.
35
+ - `M`: Multi Natural Language Inference (MNLI) dataset.
36
+ - `F`: Fever-nli dataset.
37
+ - `A`: Adversarial Natural Language Inference (ANLI) dataset.
38
+ - `finetune_datasets`: The datasets used for fine tuning the entailment model. Note that for more than 1 dataset the training was performed sequentially. For example: ACE-arg.
39
+
40
+ Some models like `HiTZ/A2T_RoBERTa_SMFA_ACE-arg` have been trained marking some information between square brackets (`'[['` and `']]'`) like the event trigger span. Make sure you follow the same preprocessing in order to obtain the best results.
41
+
42
+ ## Cite
43
+
44
+ If you use this model, consider citing the following publications:
45
+
46
+ ```bibtex
47
+ @inproceedings{sainz-etal-2021-label,
48
+ title = "Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction",
49
+ author = "Sainz, Oscar and
50
+ Lopez de Lacalle, Oier and
51
+ Labaka, Gorka and
52
+ Barrena, Ander and
53
+ Agirre, Eneko",
54
+ booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
55
+ month = nov,
56
+ year = "2021",
57
+ address = "Online and Punta Cana, Dominican Republic",
58
+ publisher = "Association for Computational Linguistics",
59
+ url = "https://aclanthology.org/2021.emnlp-main.92",
60
+ doi = "10.18653/v1/2021.emnlp-main.92",
61
+ pages = "1199--1212",
62
+ }
63
+ ```
added_tokens.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"<trg>": 50265}
config.json ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "RobertaForSequenceClassification"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.1,
6
+ "bos_token_id": 0,
7
+ "eos_token_id": 2,
8
+ "gradient_checkpointing": false,
9
+ "hidden_act": "gelu",
10
+ "hidden_dropout_prob": 0.1,
11
+ "hidden_size": 1024,
12
+ "id2label": {
13
+ "0": "entailment",
14
+ "1": "neutral",
15
+ "2": "contradiction"
16
+ },
17
+ "initializer_range": 0.02,
18
+ "intermediate_size": 4096,
19
+ "label2id": {
20
+ "contradiction": 2,
21
+ "entailment": 0,
22
+ "neutral": 1
23
+ },
24
+ "layer_norm_eps": 1e-05,
25
+ "max_position_embeddings": 514,
26
+ "model_type": "roberta",
27
+ "num_attention_heads": 16,
28
+ "num_hidden_layers": 24,
29
+ "pad_token_id": 1,
30
+ "position_embedding_type": "absolute",
31
+ "problem_type": "single_label_classification",
32
+ "torch_dtype": "float32",
33
+ "transformers_version": "4.9.0.dev0",
34
+ "type_vocab_size": 1,
35
+ "use_cache": true,
36
+ "vocab_size": 50266
37
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c3ae6b82e5015fbe286f6e696940e39a391e667df242c6d48cde79a61ecb0681
3
+ size 1421624918
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true}}
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "errors": "replace", "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "do_lower_case": false, "model_max_length": 512, "special_tokens_map_file": "/sc01a4/users/osainz006/.cache/huggingface/transformers/248872fd529229b514ff0a34d7cf36cb19ac435ad80a0d104e69e6079104c687.cb2244924ab24d706b02fd7fcedaea4531566537687a539ebb94db511fd122a0", "name_or_path": "/gscratch3/users/osainz006/A2T/wikievents/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli_2_5_5_42_25_4e-6_0.01_f1-score", "tokenizer_class": "RobertaTokenizer"}
vocab.json ADDED
The diff for this file is too large to render. See raw diff