updated about section

app.py

@@ -10,7 +10,11 @@ from src.about import (
     CITATION_BUTTON_TEXT,
     EVALUATION_QUEUE_TEXT,
     INTRODUCTION_TEXT,
-    LLM_BENCHMARKS_TEXT,
+    LLM_BENCHMARKS_TEXT_1,
+    EVALUATION_EXAMPLE_IMG,
+    LLM_BENCHMARKS_TEXT_2,
+    ENTITY_DISTRIBUTION_IMG,
+    LLM_BENCHMARKS_TEXT_3,
     TITLE,
     LOGO
 )
@@ -341,7 +345,7 @@ with demo:
                 with gr.Column(min_width=320):
                     # with gr.Box(elem_id="box-filter"):
                     filter_columns_type = gr.CheckboxGroup(
-                        label="Model types",
+                        label="Model Types",
                         choices=[t.to_str() for t in ModelType],
                         value=[t.to_str() for t in ModelType],
                         interactive=True,
@@ -418,7 +422,11 @@ with demo:
                 )
 
         with gr.TabItem("📝 About", elem_id="llm-benchmark-tab-table", id=2):
-            gr.Markdown(LLM_BENCHMARKS_TEXT, elem_classes="markdown-text")
+            gr.Markdown(LLM_BENCHMARKS_TEXT_1, elem_classes="markdown-text")
+            gr.HTML(EVALUATION_EXAMPLE_IMG, elem_classes="logo")
+            gr.Markdown(LLM_BENCHMARKS_TEXT_2, elem_classes="markdown-text")
+            gr.HTML(ENTITY_DISTRIBUTION_IMG, elem_classes="logo")
+            gr.Markdown(LLM_BENCHMARKS_TEXT_3, elem_classes="markdown-text")
 
         with gr.TabItem("🚀 Submit here! ", elem_id="llm-benchmark-tab-table", id=3):
             with gr.Column():

src/about.py

@@ -57,32 +57,173 @@ Disclaimer: It is important to note that the purpose of this evaluation is purel
 """
 
 # Which evaluations are you running? how can people reproduce what you have?
-LLM_BENCHMARKS_TEXT = f"""
+LLM_BENCHMARKS_TEXT_1 = f"""
 
-The Named Clinical Entity Recognition Leaderboard is aimed at advancing the field of natural language processing in healthcare. It provides a standardized platform for evaluating and comparing the performance of various language models in recognizing named clinical entities, a critical task for applications such as clinical documentation, decision support, and information extraction. By fostering transparency and facilitating benchmarking, the leaderboard's goal is to drive innovation and improvement in NLP models. It also helps researchers identify the strengths and weaknesses of different approaches, ultimately contributing to the development of more accurate and reliable tools for clinical use. Despite its exploratory nature, the leaderboard aims to play a role in guiding research and ensuring that advancements are grounded in rigorous and comprehensive evaluations. 
 ## About
 
+The Named Clinical Entity Recognition Leaderboard is aimed at advancing the field of natural language processing in healthcare. It provides a standardized platform for evaluating and comparing the performance of various language models in recognizing named clinical entities, a critical task for applications such as clinical documentation, decision support, and information extraction. By fostering transparency and facilitating benchmarking, the leaderboard's goal is to drive innovation and improvement in NLP models. It also helps researchers identify the strengths and weaknesses of different approaches, ultimately contributing to the development of more accurate and reliable tools for clinical use. Despite its exploratory nature, the leaderboard aims to play a role in guiding research and ensuring that advancements are grounded in rigorous and comprehensive evaluations. 
+
 ## How it works
 
 ### Evaluation method and metrics
 When training a Named Entity Recognition (NER) system, the most common evaluation methods involve measuring precision, recall, and F1-score at the token level. While these metrics are useful for fine-tuning the NER system, evaluating the predicted named entities for downstream tasks requires metrics at the full named-entity level. We include both evaluation methods: token-based and span-based. We provide an example below which helps in understanding the difference between the methods.
 Example Sentence: "The patient was diagnosed with a skin cancer disease."
 For simplicity, let's assume the an example sentence which contains 10 tokens, with a single two-token disease entity (as shown in the figure below).
+"""
+EVALUATION_EXAMPLE_IMG = """<img src="file/assets/ner_evaluation_example.png" alt="Clinical X HF" width="750" height="500">"""
+LLM_BENCHMARKS_TEXT_2 = """
+Token-based evaluation involves obtaining the set of token labels (ground-truth annotations) for the annotated entities and the set of token predictions, comparing these sets, and computing a classification report. Hence, the results for the example above are shown below.
+**Token-based metrics:**
+
+  
+  
+| Model   | TP  | FP  | FN  | Precision | Recall | F1-Score |
+| ------- | --- | --- | --- | --------- | ------ | -------- |
+| Model D | 0   | 1   | 0   | 0.00      | 0.00   | 0.00     |
+| Model C | 1   | 1   | 1   | 0.50      | 0.50   | 0.50     |
+| Model B | 2   | 2   | 0   | 0.50      | 1.00   | 0.67     |
+| Model A | 2   | 1   | 0   | 0.67      | 1.00   | 0.80     |
+
+
+Where,
+$$ Precision = TP / (TP + FP)$$
+$$ Recall = TP / (TP + FN)$$
+$$ f1score = 2 * (Prec * Rec) / (Prec + Rec)$$
+  
+  
+
+With this token-based approach, we have a broad idea of the performance of the model at the token level. However, it may misrepresent the performance at the entity level when the entity includes more than 1 token (which may be more relevant for certain applications). In addition, depending on the annotations of certain datasets, we may not want to penalize a model for a "partial" match with a certain entity.  
+The span-based method attempts to address some of these issues, by determining the full or partial matches at the entity level to classify the predictions as correct, incorrect, missed and spurious. These are then used to calculate precision, recall and F1-score. Given this, for the example below.  
+ 
+**Span-based metrics:**
+
+
+| Model   | Correct | Incorrect | Missed | Spurious | Precision | Recall | F1-Score |
+| ------- | ------- | --------- | ------ | -------- | --------- | ------ | -------- |
+| Model A | 1       | 0         | 0      | 0        | 1.00      | 1.00   | 1.00     |
+| Model B | 1       | 0         | 0      | 0        | 1.00      | 1.00   | 1.00     |
+| Model C | 1       | 0         | 0      | 0        | 1.00      | 1.00   | 1.00     |
+| Model D | 0       | 0         | 1      | 1        | 0.00      | 0.00   | 0.00     |
+
+
+Where,
+$$ Precision = COR / (COR + INC + SPU)$$
+$$ Recall = COR / (COR + INC + MIS)$$
+$$ f1score = 2 * (Prec * Rec) / (Prec + Rec)$$
+
+
+This span-based approach is equivalent to the Partial Match ("Type") in the nervaluate (NER evaluation considering partial match scoring) python package.
+Further examples are presented the section below (Other example evaluations).
+
+## Datasets
+The following datasets (test splits only) have been included in the evaluation. 
+
+### [NCBI Disease](https://huggingface.co/datasets/m42-health/clinical_ncbi)
+The NCBI Disease corpus includes mention and concept level annotations on PubMed abstracts. It covers annotations of diseases. 
+
+|            | Counts |
+| ---------- | ------ |
+| Samples    | 100    |
+| Annotation | 960    |
+
+
+### [CHIA](https://huggingface.co/datasets/m42-health/clinical_chia)
+This is a large, annotated corpus of patient eligibility criteria extracted from registered clinical trials (ClinicalTrials.gov). Annotations cover 15 different entity types, including conditions, drugs, procedures, and measurements.
+
+
+|            | Counts |
+| ---------- | ------ |
+| Samples    | 194    |
+| Annotation | 3981   |
 
-### Datasets
-📈 We evaluate the models on 4 datasets, encompassing 6 entity types
-- [NCBI](https://huggingface.co/datasets/m42-health/clinical_ncbi) 
-- [CHIA](https://huggingface.co/datasets/m42-health/clinical_chia)
-- [BIORED](https://huggingface.co/datasets/m42-health/clinical_biored)
-- [BC5CD](https://huggingface.co/datasets/m42-health/clinical_bc5cdr)
 
-### Evaluation Metrics
-We perceive NER objects as span(with character offsets) instead of token level artifacts. This enables us to expand to nested NER scenarios easily.
+### [BC5CDR](https://huggingface.co/datasets/m42-health/clinical_bc5cdr)
+The BC5CDR corpus consists of 1500 PubMed articles with annotated chemicals and diseases.
 
 
-## Reproducibility
+|            | Counts |
+| ---------- | ------ |
+| Samples    | 500    |
+| Annotation | 9928   |
+
+
+### [BIORED](https://huggingface.co/datasets/m42-health/clinical_biored)
+The BIORED corpus includes a set of PubMed abstracts with annotations of multiple entity types (e.g., gene/protein, disease, chemical).
+
+
+|            | Counts |
+| ---------- | ------ |
+| Samples    | 100    |
+| Annotation | 3535   |
+
+
+Datasets summary
+
+A summary of the datasets used are summarized here.
+
+
+| Dataset | # samples | # annotations | # original entities | # clinical entities |
+| ------- | --------- | ------------- | ------------------- | ------------------- |
+| NCBI    | 100       | 960           | 4                   | 1                   |
+| CHIA    | 194       | 3981          | 16                  | 4                   |
+| BIORED  | 500       | 9928          | 2                   | 4                   |
+| BC5CDR  | 100       | 3535          | 6                   | 2                   |
+
+
+## Clinical Entity Types
+
+The above datasets are modified to cater to the clinical setting. For this, the entity types that are clinically relevant are retained and the rest are dropped. Further, the clinical entity type is standardized across the dataset to obtain a total of 6 clinical entity types shown below.
+
+
+| Clinical Entity | Combined Annotation |
+| --------------- | ------------------- |
+| Condition       | 7514                |
+| Drug            | 6443                |
+| Procedure       | 300                 |
+| Measurement     | 258                 |
+| Gene            | 1180                |
+| Gene Variant    | 241                 |
+
+
+The pie chart on the left below the distribution of clinical entities and their original dataset types.
+"""
+
+ENTITY_DISTRIBUTION_IMG = """<img src="file/assets/entity_distribution.png" alt="Clinical X HF" width="750" height="500">"""
+LLM_BENCHMARKS_TEXT_3="""
+## Decoder Model Evaluation
+Evaluating encoder models, such as BERT, for token classification tasks (e.g., NER) is straightforward given that these models process the entire input sequence simultaneously. This allows them to generate token-level classifications by leveraging bidirectional context, facilitating a direct comparison of predicted tags against the gold standard labels for each token in the input sequence.  
+
+In contrast, decoder-only models, like GPT, generate responses sequentially, predicting one token at a time based on the preceding context. Evaluating the performance of these models for token classification tasks requires a different approach. First, we prompt the decoder-only LLM with a specific task of tagging the different entity types within a given text. This task is clearly defined to the model, ensuring it understands which types of entities to identify (i.e., conditions, drugs, procedures, etc).   
+An example of the task prompt is shown below.
+```
+## Instruction
+Your task is to generate an HTML version of an input text, marking up specific entities related to healthcare. The entities to be identified are: symptom, disorder. Use HTML <span > tags to highlight these entities. Each <span > should have a class attribute indicating the type of the entity. Do NOT provide further examples and just consider the input provided below. Do NOT provide an explanation nor notes about the reasoning. Do NOT reformat nor summarize the input text. Follow the instruction and the format of the example below.
+ 
+## Entity markup guide
+Use <span class='symptom' > to denote a symptom.
+Use <span class='disorder' > to denote a disorder.
+```
+To ensure deterministic and consistent outputs, the temperature for generation is kept at 0.0. The model then generates a sequential response that includes the tagged entities, as shown in the example below.
+```
+## Input:
+He had been diagnosed with osteoarthritis of the knees and had undergone arthroscopy years prior to admission.
+## Output:
+He had been diagnosed with <span class="disease" >osteoarthritis of the knees</span >and had undergone <span class="procedure" >arthroscopy</span >years prior to admission.
+```
+
+After the tagged output is generated, it is parsed to extract the tagged entities. The parsed data are then compared against the gold standard labels, and performance metrics are computed as above. This evaluation method ensures a consistent and objective assessment of decoder-only LLM's performance in NER tasks, despite the differences in their architecture compared to encoder models. 
+
+# Reproducibility
 To reproduce our results, follow the steps detailed [here](https://github.com/WadoodAbdul/medics_ner/blob/master/docs/reproducing_results.md)
 
+# Disclaimer and Advisory
+The Leaderboard is maintained by the authors and affiliated entity as part of our ongoing contribution to open research in the field of NLP in healthcare. The leaderboard is intended for academic and exploratory purposes only. The language models evaluated on this platform (to the best knowledge of the authors) have not been approved for clinical use, and their performance should not be interpreted as clinically validated or suitable for real-world medical applications.  
+
+Users are advised to approach the results with an understanding of the inherent limitations and the experimental nature of this evaluation. The authors and affiliated entity do not endorse any specific model or approach, and the leaderboard is provided without any warranties or guarantees. Researchers and practitioners are encouraged to use the leaderboard as a resource to guide further research and development, keeping in mind the necessity for rigorous testing and validation in clinical settings before any practical application.
+
+
+
+
 """
 
 EVALUATION_QUEUE_TEXT = """

src/display/utils.py

@@ -159,7 +159,7 @@ class PromptTemplateName(Enum):
     UniversalNERTemplate = "universal_ner"
     LLMHTMLHighlightedSpansTemplate = "llm_html_highlighted_spans"
     LLamaNERTemplate = "llama_70B_ner_v0.3"
-    MixtralNERTemplate = "mixtral_ner_v0.3.jinja"
+    MixtralNERTemplate = "mixtral_ner_v0.3"