Initial model version

README.md

@@ -1,3 +1,213 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+language:
+- en
+metrics:
+- f1
+- recall
+- precision
+tags:
+- NER
+pipeline_tag: token-classification
+library_name: gliner
+datasets:
+- knowledgator/GLINER-multi-task-synthetic-data
+- EmergentMethods/AskNews-NER-v0
+- urchade/pile-mistral-v0.1
+- MultiCoNER/multiconer_v2
+- DFKI-SLT/few-nerd
+base_model: knowledgator/gliner-multitask-large-v0.5
+---
+
+![Illustration](images/gliner_merge_model_illustration.png)
+
+The `xomad/gliner-model-merge-large-v1.0` model is developed from the pretrained model `knowledgator/gliner-multitask-large-v0.5` to explore the capabilities of model merging techniques, resulting in a significant performance boost of 3.25 points, elevating the model's capability from 0.6276 to 0.6601 F1-score.
+
+The model is trained exclusively on datasets with commercial-friendly licenses to ensure broad applicability under the Apache-2.0 license. The following datasets were used in the training process:
+- [knowledgator/GLINER-multi-task-synthetic-data](https://huggingface.co/datasets/knowledgator/GLINER-multi-task-synthetic-data)
+- [EmergentMethods/AskNews-NER-v0](https://huggingface.co/datasets/EmergentMethods/AskNews-NER-v0)
+- [urchade/pile-mistral-v0.1](https://huggingface.co/datasets/urchade/pile-mistral-v0.1)
+- [MultiCoNER/multiconer_v2](https://huggingface.co/datasets/MultiCoNER/multiconer_v2)
+- [DFKI-SLT/few-nerd](https://huggingface.co/datasets/DFKI-SLT/few-nerd)
+
+### ⚙️ Finetuning process
+The process begins with the base model `knowledgator/gliner-multitask-large-v0.5`. Our model `xomad/gliner-model-merge-large-v1.0` is fine-tuned separately on each of the above datasets , and we save multiple checkpoints along the fine-tuning process. We put all these checkpoints together into a pool and then we apply the [Model soups](https://arxiv.org/abs/2203.05482) technique to produce different merged models:
+- `uniform_merged`
+- `greedy_on_random`
+- `greedy_on_sorted`
+
+Following this, we apply [WiSE-FT](https://openaccess.thecvf.com/content/CVPR2022/html/Wortsman_Robust_Fine-Tuning_of_Zero-Shot_Models_CVPR_2022_paper.html?ref=roboflow-blog) merging technique to pairs of models selected from a group of the above 3 models and the original model to produce the `wise_ft_merged` model. This concludes the 1st finetuning phase.
+
+The process is then repeated in the 2nd finetuning phase, using the `wise_ft_merged` as the new starting point, to produce the final model. The whole finetuning flow is illustrated in the following figure:
+
+![Finetuning flow](images/finetune_flow.png)
+
+The performance of the pool of fine-tuned models and the merged models are evaluated on the `CrossNER`, TwitterNER benchmarks, and plotted in the following 2 figures (as `crossner_f1` and `other_f1` respectively).
+
+The 1st finetuning phase plot:
+![1st finetuning phase](images/model_soups.png)
+
+The 2nd finetuning phase plot:
+![2nd finetuning phase](images/model_soups2.png)
+
+
+### 🛠️ Installation 	
+To use this model, you must install the [GLiNER Python library](https://github.com/urchade/GLiNER):
+
+```bash
+pip install gliner
+```
+
+Once you've downloaded the GLiNER library, you can import the GLiNER class. You can then load this model using GLiNER.from_pretrained.
+
+### 💻 Usage
+
+```python
+from gliner import GLiNER
+
+model = GLiNER.from_pretrained("xomad/gliner-model-merge-large-v1.0")
+
+text = """
+Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975 to develop and sell BASIC interpreters for the Altair 8800. During his career at Microsoft, Gates held the positions of chairman, chief executive officer, president and chief software architect, while also being the largest individual shareholder until May 2014.
+"""
+
+labels = ["founder", "computer", "software", "position", "date", "company"]
+
+entities = model.predict_entities(text, labels)
+
+for entity in entities:
+    print(entity["text"], "=>", entity["label"])
+```
+
+Output:
+```
+Microsoft => company
+Bill Gates => founder
+Paul Allen => founder
+April 4, 1975 => date
+BASIC => software
+Altair 8800 => computer
+Microsoft => company
+chairman => position
+chief executive officer => position
+president => position
+chief software architect => position
+May 2014 => date
+```
+
+
+### 📊 Benchmarks:
+
+![Model Performance](images/performance.png)
+
+Performance on different zero-shot NER benchmarks (CrossNER, mit-movie and mit-restaurant), numbers reported from https://huggingface.co/knowledgator/gliner-multitask-large-v0.5:
+
+|  Model                                                                                                  |   F1 Score  |
+|---------------------------------------------------------------------------------------------------------|-------------|
+| [xomad/gliner-model-merge-large-v1.0](https://huggingface.co/xomad/gliner-model-merge-large-v1.0)         |  **0.6601** |
+| [knowledgator/gliner-multitask-v0.5](https://huggingface.co/knowledgator/gliner-multitask-v0.5)         |  _0.6276_   |
+| [numind/NuNER_Zero-span](https://huggingface.co/numind/NuNER_Zero-span)                                 |  0.6196     |
+| [gliner-community/gliner_large-v2.5](https://huggingface.co/gliner-community/gliner_large-v2.5)         |  0.615      |
+| [EmergentMethods/gliner_large_news-v2.1](https://huggingface.co/EmergentMethods/gliner_large_news-v2.1) |  0.5876     |
+| [urchade/gliner_large-v2.1](https://huggingface.co/urchade/gliner_large-v2.1)                           |  0.5754     |
+
+
+Detailed performance on different datasets:
+
+
+| Model                              | Dataset            | Precision | Recall | F1 Score | F1 Score (Decimal) |
+|------------------------------------|--------------------|-----------|--------|----------|--------------------|
+| xomad/gliner-model-merge-large-v1.0 | CrossNER_AI         | 62.66%    | 57.48% | 59.96%   | 0.5996             |
+|                                    | CrossNER_literature | 73.28%    | 66.42% | 69.68%   | 0.6968             |
+|                                    | CrossNER_music      | 74.89%    | 70.67% | 72.72%   | 0.7272             |
+|                                    | CrossNER_politics   | 79.46%    | 77.57% | 78.51%   | 0.7851             |
+|                                    | CrossNER_science    | 74.72%    | 70.24% | 72.41%   | 0.7241             |
+|                                    | mit-movie           | 67.33%    | 57.89% | 62.25%   | 0.6225             |
+|                                    | mit-restaurant      | 54.94%    | 40.41% | 46.57%   | 0.4657             |
+|                                    | **Average**        |           |        |          | **0.6601**         |
+| numind/NuNER_Zero-span             | CrossNER_AI        | 63.82%    | 56.82% | 60.12%   | 0.6012             |
+|                                    | CrossNER_literature| 73.53%    | 58.06% | 64.89%   | 0.6489             |
+|                                    | CrossNER_music     | 72.69%    | 67.40% | 69.95%   | 0.6995             |
+|                                    | CrossNER_politics  | 77.28%    | 68.69% | 72.73%   | 0.7273             |
+|                                    | CrossNER_science   | 70.08%    | 63.12% | 66.42%   | 0.6642             |
+|                                    | mit-movie          | 63.00%    | 48.88% | 55.05%   | 0.5505             |
+|                                    | mit-restaurant     | 54.81%    | 37.62% | 44.62%   | 0.4462             |
+|                                    | **Average**        |           |        |          | **0.6196**         |
+| knowledgator/gliner-multitask-v0.5 | CrossNER_AI         | 51.00%    | 51.11% | 51.05%   | 0.5105             |
+|                                    | CrossNER_literature | 72.65%    | 65.62% | 68.96%   | 0.6896             |
+|                                    | CrossNER_music      | 74.91%    | 73.70% | 74.30%   | 0.7430             |
+|                                    | CrossNER_politics   | 78.84%    | 77.71% | 78.27%   | 0.7827             |
+|                                    | CrossNER_science    | 69.20%    | 65.48% | 67.29%   | 0.6729             |
+|                                    | mit-movie           | 61.29%    | 52.59% | 56.60%   | 0.5660             |
+|                                    | mit-restaurant      | 50.65%    | 38.13% | 43.51%   | 0.4351             |
+|                                    | **Average**        |           |        |          | **0.6276**         |
+| gliner-community/gliner_large-v2.5 | CrossNER_AI         | 50.85%    | 63.03% | 56.29%   | 0.5629             |
+|                                    | CrossNER_literature | 64.92%    | 67.21% | 66.04%   | 0.6604             |
+|                                    | CrossNER_music      | 70.88%    | 73.10% | 71.97%   | 0.7197             |
+|                                    | CrossNER_politics   | 72.67%    | 72.93% | 72.80%   | 0.7280             |
+|                                    | CrossNER_science    | 61.71%    | 68.85% | 65.08%   | 0.6508             |
+|                                    | mit-movie           | 54.63%    | 52.83% | 53.71%   | 0.5371             |
+|                                    | mit-restaurant      | 47.99%    | 42.13% | 44.87%   | 0.4487             |
+|                                    | **Average**        |           |        |          | **0.6154**         |
+| urchade/gliner_large-v2.1          | CrossNER_AI        | 54.98%    | 52.00% | 53.45%   | 0.5345             |
+|                                    | CrossNER_literature| 59.33%    | 56.47% | 57.87%   | 0.5787             |
+|                                    | CrossNER_music     | 67.39%    | 66.77% | 67.08%   | 0.6708             |
+|                                    | CrossNER_politics  | 66.07%    | 63.76% | 64.90%   | 0.6490             |
+|                                    | CrossNER_science   | 61.45%    | 62.56% | 62.00%   | 0.6200             |
+|                                    | mit-movie          | 55.94%    | 47.36% | 51.29%   | 0.5129             |
+|                                    | mit-restaurant     | 53.34%    | 40.83% | 46.25%   | 0.4625             |
+|                                    | **Average**        |           |        |          | **0.5754**         |
+| EmergentMethods/gliner_large_news-v2.1| CrossNER_AI    | 59.60%    | 54.55% | 56.96%   | 0.5696             |
+|                                    | CrossNER_literature| 65.41%    | 56.16% | 60.44%   | 0.6044             |
+|                                    | CrossNER_music     | 67.47%    | 63.08% | 65.20%   | 0.6520             |
+|                                    | CrossNER_politics  | 66.05%    | 60.07% | 62.92%   | 0.6292             |
+|                                    | CrossNER_science   | 68.44%    | 63.57% | 65.92%   | 0.6592             |
+|                                    | mit-movie          | 65.85%    | 49.59% | 56.57%   | 0.5657             |
+|                                    | mit-restaurant     | 54.71%    | 35.94% | 43.38%   | 0.4338             |
+|                                    | **Average**        |           |        |          | **0.5876**         |
+
+
+### Authors
+
+Hoan Nguyen, at xomad.com
+
+### Citations
+
+```
+@misc{wortsman2022modelsoupsaveragingweights,
+      title={Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time}, 
+      author={Mitchell Wortsman and Gabriel Ilharco and Samir Yitzhak Gadre and Rebecca Roelofs and Raphael Gontijo-Lopes and Ari S. Morcos and Hongseok Namkoong and Ali Farhadi and Yair Carmon and Simon Kornblith and Ludwig Schmidt},
+      year={2022},
+      eprint={2203.05482},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2203.05482}, 
+}
+
+@InProceedings{Wortsman_2022_CVPR,
+    author    = {Wortsman, Mitchell and Ilharco, Gabriel and Kim, Jong Wook and Li, Mike and Kornblith, Simon and Roelofs, Rebecca and Lopes, Raphael Gontijo and Hajishirzi, Hannaneh and Farhadi, Ali and Namkoong, Hongseok and Schmidt, Ludwig},
+    title     = {Robust Fine-Tuning of Zero-Shot Models},
+    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+    month     = {June},
+    year      = {2022},
+    pages     = {7959-7971}
+}
+
+@misc{stepanov2024gliner,
+      title={GLiNER multi-task: Generalist Lightweight Model for Various Information Extraction Tasks}, 
+      author={Ihor Stepanov and Mykhailo Shtopko},
+      year={2024},
+      eprint={2406.12925},
+      archivePrefix={arXiv},
+      primaryClass={id='cs.LG' full_name='Machine Learning' is_active=True alt_name=None in_archive='cs' is_general=False description='Papers on all aspects of machine learning research (supervised, unsupervised, reinforcement learning, bandit problems, and so on) including also robustness, explanation, fairness, and methodology. cs.LG is also an appropriate primary category for applications of machine learning methods.'}
+}
+
+@misc{zaratiana2023gliner,
+      title={GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer}, 
+      author={Urchade Zaratiana and Nadi Tomeh and Pierre Holat and Thierry Charnois},
+      year={2023},
+      eprint={2311.08526},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```

added_tokens.json

@@ -0,0 +1,6 @@
+{
+  "<<ENT>>": 128002,
+  "<<SEP>>": 128003,
+  "[FLERT]": 128001,
+  "[MASK]": 128000
+}

gliner_config.json

@@ -0,0 +1,134 @@
+{
+  "class_token_index": 128002,
+  "dropout": 0.1,
+  "embed_ent_token": true,
+  "encoder_config": {
+    "_name_or_path": "microsoft/deberta-v3-large",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_probs_dropout_prob": 0.1,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "gelu",
+    "hidden_dropout_prob": 0.1,
+    "hidden_size": 1024,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-07,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 512,
+    "max_relative_positions": -1,
+    "min_length": 0,
+    "model_type": "deberta-v2",
+    "no_repeat_ngram_size": 0,
+    "norm_rel_ebd": "layer_norm",
+    "num_attention_heads": 16,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 24,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 0,
+    "pooler_dropout": 0,
+    "pooler_hidden_act": "gelu",
+    "pooler_hidden_size": 1024,
+    "pos_att_type": [
+      "p2c",
+      "c2p"
+    ],
+    "position_biased_input": false,
+    "position_buckets": 256,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "relative_attention": true,
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "share_att_key": true,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "type_vocab_size": 0,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "vocab_size": 128004
+  },
+  "ent_token": "<<ENT>>",
+  "eval_every": 1000,
+  "fine_tune": true,
+  "freeze_token_rep": false,
+  "fuse_layers": false,
+  "has_rnn": true,
+  "hidden_size": 512,
+  "labels_encoder": null,
+  "labels_encoder_config": null,
+  "log_dir": "logs_final",
+  "loss_alpha": 0.75,
+  "loss_gamma": 0,
+  "loss_reduction": "sum",
+  "lr_encoder": "5e-6",
+  "lr_others": "7e-6",
+  "max_len": 768,
+  "max_neg_type_ratio": 1,
+  "max_types": 30,
+  "max_width": 100,
+  "model_name": "microsoft/deberta-v3-large",
+  "model_type": "gliner",
+  "name": "token level gliner large",
+  "num_post_fusion_layers": 1,
+  "post_fusion_schema": "",
+  "random_drop": true,
+  "root_dir": "gliner_logs",
+  "save_total_limit": 10,
+  "scheduler_type": "linear",
+  "sep_token": "<<SEP>>",
+  "shuffle_types": true,
+  "size_sup": -1,
+  "span_mode": "token_level",
+  "subtoken_pooling": "first",
+  "train_batch_size": 8,
+  "transformers_version": "4.44.2",
+  "val_data_dir": "none",
+  "vocab_size": 128004,
+  "warmup_ratio": 0.1,
+  "weight_decay_encoder": 0.01,
+  "weight_decay_other": 0.01,
+  "words_splitter_type": "whitespace"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:089504018e1ac4f24a9f4e13f09490240430567fb863af27aecf71fe9d74ced8
+size 1761030542

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd
+size 2464616

tokenizer_config.json

@@ -0,0 +1,86 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128000": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "[FLERT]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128002": {
+      "content": "<<ENT>>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128003": {
+      "content": "<<SEP>>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "[CLS]",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "eos_token": "[SEP]",
+  "mask_token": "[MASK]",
+  "max_length": null,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "sp_model_kwargs": {},
+  "split_by_punct": false,
+  "tokenizer_class": "DebertaV2Tokenizer",
+  "unk_token": "[UNK]",
+  "vocab_type": "spm"
+}