updated files for hhem-2.1-open

README.md

@@ -1,165 +1,95 @@
 ---
-license: apache-2.0
 language: en
-tags:
-- microsoft/deberta-v3-base
-datasets:
-- multi_nli
-- snli
-- fever
-- tals/vitaminc
-- paws
-metrics:
-- accuracy
-- auc
-- balanced accuracy
-pipeline_tag: text-classification
-widget:
-- text: "A man walks into a bar and buys a drink [SEP] A bloke swigs alcohol at a pub"
-  example_title: "Positive"
-- text: "A boy is jumping on skateboard in the middle of a red bridge. [SEP] The boy skates down the sidewalk on a blue bridge"
-  example_title: "Negative"
-
+license: apache-2.0
+base_model: google/flan-t5-base
+pipline_tag: text-classficiation
 ---
-<img src="candle.png" width="50" height="50" style="display: inline;">  In Loving memory of Simon Mark Hughes...
 
-# Introduction
-The HHEM model is an open source model, created by [Vectara](https://vectara.com), for detecting hallucinations in LLMs. It is particularly useful in the context of building retrieval-augmented-generation (RAG) applications where a set of facts is summarized by an LLM, but the model can also be used in other contexts.
+<img src="https://huggingface.co/vectara/hallucination_evaluation_model/resolve/main/candle.png" width="50" height="50" style="display: inline;">  In Loving memory of Simon Mark Hughes...
+
+
+HHEM-2.1-Open is a major upgrade to HHEM-1.0-Open created by [Vectara](https://vectara.com) in November 2023. The HHEM model series are designed for detecting hallucinations in LLMs. They are particularly useful in the context of building retrieval-augmented-generation (RAG) applications where a set of facts is summarized by an LLM, and HHEM can be used to measure the extent to which this summary is factually consistent with the facts.
 
 If you are interested to learn more about RAG or experiment with Vectara, you can [sign up](https://console.vectara.com/signup/?utm_source=huggingface&utm_medium=space&utm_term=hhem-model&utm_content=console&utm_campaign=) for a free Vectara account.
-Vectara now implements an improved version of HHEM which is a calibrated score with longer sequence length, and non-English language support. See more details [here](https://vectara.com/blog/automating-hallucination-detection-introducing-vectara-factual-consistency-score/) 
-Now let's dive into the details of the model.
 
-## Cross-Encoder for Hallucination Detection
-This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class. 
-The model outputs a probabilitity from 0 to 1, 0 being a hallucination and 1 being factually consistent. 
-The predictions can be thresholded at 0.5 to predict whether a document is consistent with its source.
+## Hallucination Detection 101
+By "hallucinated" or "factually inconsistent", we mean that a text (hypothesis, to be judged) is not supported by another text (evidence/premise, given). You **always need two** pieces of text to determine whether a text is hallucinated or not. When applied to RAG (retrieval augmented generation), the LLM is provided by several pieces of text (often called facts or context) retrieved from some dataset, and a hallucination would indicate that the summary (hypothesis) is not supported by those facts (evidence). 
 
-## Training Data
-This model is based on [microsoft/deberta-v3-base](https://huggingface.co/microsoft/deberta-v3-base) and is trained initially on NLI data to determine textual entailment, before being further fine tuned on summarization datasets with samples annotated for factual consistency including [FEVER](https://huggingface.co/datasets/fever), [Vitamin C](https://huggingface.co/datasets/tals/vitaminc) and [PAWS](https://huggingface.co/datasets/paws).
+A common type of hallucination in RAG is **factual but hallucinated**. For example, given the premise _"The capital of France is Berlin"_, the hypothesis _"The capital of France is Paris"_ is hallucinated -- although it is true in the world knowledge. This happens when LLMs do not generate content based on the textual data provided to them as part of the RAG retrieval process, but rather generate content based on their pre-trained knowledge.
 
-## Performance
+## Using HHEM-2.1-Open
 
-* [TRUE Dataset](https://arxiv.org/pdf/2204.04991.pdf) (Minus Vitamin C, FEVER and PAWS) - 0.872 AUC Score
-* [SummaC Benchmark](https://aclanthology.org/2022.tacl-1.10.pdf) (Test Split) - 0.764 Balanced Accuracy, 0.831 AUC Score
-* [AnyScale Ranking Test for Hallucinations](https://www.anyscale.com/blog/llama-2-is-about-as-factually-accurate-as-gpt-4-for-summaries-and-is-30x-cheaper) - 86.6 % Accuracy
+HHEM-2.1-Open can be loaded easily using the `transformers` library. Just remember to set `trust_remote_code=True` to take advantage of the pre-/post-processing code we provided for your convenience. The **input** of the model is a list of pairs of (premise, hypothesis). For each pair, the model will **return** a score between 0 and 1, where 0 means that the hypothesis is not evidenced at all by the premise and 1 means the hypothesis is fully supported by the premise. 
 
-## LLM Hallucination Leaderboard
-If you want to stay up to date with results of the latest tests using this model to evaluate the top LLM models, we have a [public leaderboard](https://huggingface.co/spaces/vectara/leaderboard) that is periodically updated, and results are also available on the [GitHub repository](https://github.com/vectara/hallucination-leaderboard).
+```python
+from transformers import AutoModelForSequenceClassification
+
+# Load the model
+model = AutoModelForSequenceClassification.from_pretrained(
+    'vectara/hallucination_evaluation_model', trust_remote_code=True)
+
+pairs = [ # Test data, List[Tuple[str, str]]
+    ("The capital of France is Berlin.", "The capital of France is Paris."), # factual but hallucinated
+    ('I am in California', 'I am in United States.'), # Consistent
+    ('I am in United States', 'I am in California.'), # Hallucinated
+    ("A person on a horse jumps over a broken down airplane.", "A person is outdoors, on a horse."),
+    ("A boy is jumping on skateboard in the middle of a red bridge.", "The boy skates down the sidewalk on a red bridge"),
+    ("A man with blond-hair, and a brown shirt drinking out of a public water fountain.", "A blond man wearing a brown shirt is reading a book."),
+    ("Mark Wahlberg was a fan of Manny.", "Manny was a fan of Mark Wahlberg.")
+]
 
-# Using HHEM
+# Use the model to predict
+model.predict(pairs) # note the predict() method. Do not do model(pairs). 
+# tensor([0.0111, 0.6474, 0.1290, 0.8969, 0.1846, 0.0050, 0.0543])
+```
 
-## Using the Inference API Widget on the Right
-To use the model with the widget, you need to pass both documents as a single string separated with [SEP]. For example:
+Note that the order of a pair is important. For example, notice how the 2nd and 3rd examples in the `pairs` list are consistent and hallcuianted, respectively. 
 
-* A man walks into a bar and buys a drink [SEP] A bloke swigs alcohol at a pub
-* A person on a horse jumps over a broken down airplane. [SEP] A person is at a diner, ordering an omelette.
-* A person on a horse jumps over a broken down airplane. [SEP] A person is outdoors, on a horse.
 
-etc. See examples below for expected probability scores.
+## HHEM-2.1-Open vs. HHEM-1.0
 
-## Usage with Sentencer Transformers (Recommended)
+The major difference between HHEM-2.1-Open and the original HHEM-1.0 is that HHEM-2.1-Open has an unlimited context length, while HHEM-1.0 is capped at 512 tokens. The longer context length allows HHEM-2.1-Open to provide more accurate hallucination detection for RAG which often needs more than 512 tokens. 
 
-### Inference
-The model can be used like this, on pairs of documents, passed as a list of list of strings (```List[List[str]]]```):
+The tables below compare the two models on the [AggreFact](https://arxiv.org/pdf/2205.12854) and [RAGTruth](https://arxiv.org/abs/2401.00396) benchmarks. In particualr, on AggreFact, we focus on its SOTA subset (denoted as `AggreFact-SOTA`) which contains summaries generated by Google's T5, Meta's BART, and Google's Pegasus, which are the three latest models in the AggreFact benchmark. The results on RAGTruth's summarization (denoted as `RAGTruth-Summ`) and QA (denoted as `RAGTruth-QA`) subsets are reported separately.
 
-```python
-from sentence_transformers import CrossEncoder
-
-model = CrossEncoder('vectara/hallucination_evaluation_model')
-scores = model.predict([
-    ["A man walks into a bar and buys a drink", "A bloke swigs alcohol at a pub"],
-    ["A person on a horse jumps over a broken down airplane.", "A person is at a diner, ordering an omelette."],
-    ["A person on a horse jumps over a broken down airplane.", "A person is outdoors, on a horse."],
-    ["A boy is jumping on skateboard in the middle of a red bridge.", "The boy skates down the sidewalk on a blue bridge"],
-    ["A man with blond-hair, and a brown shirt drinking out of a public water fountain.", "A blond drinking water in public."],
-    ["A man with blond-hair, and a brown shirt drinking out of a public water fountain.", "A blond man wearing a brown shirt is reading a book."],
-    ["Mark Wahlberg was a fan of Manny.", "Manny was a fan of Mark Wahlberg."],  
-])
-```
+Table 1: Performance on AggreFact-SOTA
+| model     |    Balanced Accuracy | F1     | Recall | Precision |
+|:----------------------|---------:|-------:|-------:|----------:|
+| HHEM-1.0              | 0.7887   | 0.9047 | 0.7081 | 0.6728    |
+| HHEM-2.1-Open         | 0.7655   | 0.6677 | 0.6848 | 0.6513    |
 
-This returns a numpy array representing a factual consistency score. A score < 0.5 indicates a likely hallucination):
-```
-array([0.61051559, 0.00047493709, 0.99639291, 0.00021221573, 0.99599433, 0.0014127002, 0.002.8262993], dtype=float32)
-```
+Table 2: Performance on RAGTruth-Summ
+| model     |    Balanced Accuracy | F1         | Recall    | Precision |
+|:----------------------|---------:|-----------:|----------:|----------:|
+| HHEM-1.0              | 0.5336 | 0.1577   | 0.0931 | 0.5135 |
+| HHEM-2.1-Open         | 0.6442 | 0.4883 | 0.3186 | 0.7558 |
 
-Note that the model is designed to work with entire documents, so long as they fit into the 512 token context window (across both documents). 
-Also note that the order of the documents is important, the first document is the source document, and the second document is validated against the first for factual consistency, e.g. as a summary of the first or a claim drawn from the source.
+Table 3: Performance on RAGTruth-QA
+| model     |    Balanced Accuracy | F1         | Recall    | Precision |
+|:----------------------|---------:|-----------:|----------:|----------:|
+| HHEM-1.0              | 0.5258 | 0.1940   | 0.1625 | 0.2407 |
+| HHEM-2.1-Open         | 0.7428 | 0.6000 | 0.5438 | 0.6692 |
 
-### Training
+The tables above show that HHEM-2.1-Open has a significant improvement over HHEM-1.0 in the RAGTruth-Summ and RAGTruth-QA benchmarks, while it has a slight decrease in the AggreFact-SOTA benchmark. However when intepreting these results, please note that AggreFact-SOTA is evaluated on relatively older types of LLMs:
+- LLMs in AggreFact-SOTA: T5, BART, and Pegasus;
+- LLMs in RAGTruth: GPT-4-0613, GPT-3.5-turbo-0613, Llama-2-7B/13B/70B-chat, and Mistral-7B-instruct. 
 
-```python
-from sentence_transformers.cross_encoder import CrossEncoder
-from sentence_transformers.cross_encoder.evaluation import CEBinaryClassificationEvaluator
-from sentence_transformers import InputExample
-
-num_epochs = 5
-model_save_path = "./model_dump"
-model_name = 'cross-encoder/nli-deberta-v3-base' # base model, use 'vectara/hallucination_evaluation_model' if you want to further fine-tune ours
-
-model = CrossEncoder(model_name, num_labels=1, automodel_args={'ignore_mismatched_sizes':True})
-
-# Load some training examples as such, using a pandas dataframe with source and summary columns:
-train_examples, test_examples = [], []
-for i, row in df_train.iterrows():
-    train_examples.append(InputExample(texts=[row['source'], row['summary']], label=int(row['label'])))
-
-for i, row in df_test.iterrows():
-    test_examples.append(InputExample(texts=[row['source'], row['summary']], label=int(row['label'])))
-test_evaluator = CEBinaryClassificationEvaluator.from_input_examples(test_examples, name='test_eval')
-
-# Then train the model as such as per the Cross Encoder API:
-train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=train_batch_size)
-warmup_steps = math.ceil(len(train_dataloader) * num_epochs * 0.1) #10% of train data for warm-up
-model.fit(train_dataloader=train_dataloader,
-          evaluator=test_evaluator,
-          epochs=num_epochs,
-          evaluation_steps=10_000,
-          warmup_steps=warmup_steps,
-          output_path=model_save_path,
-          show_progress_bar=True)
-```
+Therefore, we conclude that HHEM-2.1-Open is better than HHEM-1.0. 
 
-## Usage with Transformers AutoModel
-You can use the model also directly with Transformers library (without the SentenceTransformers library):
+## Want something more powerful?
 
-```python
-from transformers import AutoTokenizer, AutoModelForSequenceClassification
-import torch
-import numpy as np
-
-model = AutoModelForSequenceClassification.from_pretrained('vectara/hallucination_evaluation_model')
-tokenizer = AutoTokenizer.from_pretrained('vectara/hallucination_evaluation_model')
-
-pairs = [
-    ["A man walks into a bar and buys a drink", "A bloke swigs alcohol at a pub"],
-    ["A person on a horse jumps over a broken down airplane.", "A person is at a diner, ordering an omelette."],
-    ["A person on a horse jumps over a broken down airplane.", "A person is outdoors, on a horse."],
-    ["A boy is jumping on skateboard in the middle of a red bridge.", "The boy skates down the sidewalk on a blue bridge"],
-    ["A man with blond-hair, and a brown shirt drinking out of a public water fountain.", "A blond drinking water in public."],
-    ["A man with blond-hair, and a brown shirt drinking out of a public water fountain.", "A blond man wearing a brown shirt is reading a book."],
-    ["Mark Wahlberg was a fan of Manny.", "Manny was a fan of Mark Wahlberg."], 
-]
+As you may have already sensed from the name, HHEM-2.1-Open is the open source version of the premium HHEM-2.1. HHEM-2.1 (without the `-Open`) is offered exclusively via Vectara's RAG-as-a-service platform. The major difference between HHEM-2.1 and HHEM-2.1-Open is that HHEM-2.1 is cross-lingual on three languages: English, German, and French, while HHEM-2.1-Open is English-only. "Cross-lingual" means any combination of the three languages, e.g., documents in German, query in English, results in French.
 
-inputs = tokenizer.batch_encode_plus(pairs, return_tensors='pt', padding=True)
+### Why RAG in Vectara? 
 
-model.eval()
-with torch.no_grad():
-    outputs = model(**inputs)
-    logits = outputs.logits.cpu().detach().numpy()
-    # convert logits to probabilities
-    scores = 1 / (1 + np.exp(-logits)).flatten()
-```
+Vectara provides a Trusted Generative AI platform. The platform allows organizations to rapidly create an AI assistant experience which is grounded in the data, documents, and knowledge that they have. Vectara's serverless RAG-as-a-Service also solves critical problems required for enterprise adoption, namely: reduces hallucination, provides explainability / provenance, enforces access control, allows for real-time updatability of the knowledge, and mitigates intellectual property / bias concerns from large language models.
 
-This returns a numpy array representing a factual consistency score. A score < 0.5 indicates a likely hallucination):
-```
-array([0.61051559, 0.00047493709, 0.99639291, 0.00021221573, 0.99599433, 0.0014127002, 0.002.8262993], dtype=float32)
-```
+To start benefiting from HHEM-2.1, you can [sign up](https://console.vectara.com/signup/?utm_source=huggingface&utm_medium=space&utm_term=hhem-model&utm_content=console&utm_campaign=) for a free Vectara account, and you will get the HHEM-2.1 score returned with every query automatically.
 
-## Contact Details
-Feel free to contact us with any questions:
-* X/Twitter - https://twitter.com/vectara or http://twitter.com/ofermend
-* Discussion [forums](https://discuss.vectara.com/)
-* Discord [server](https://discord.gg/GFb8gMz6UH)
+Here are some additional resources:
+1. Vectara [API documentation](https://docs.vectara.com/docs).
+2. Quick start using Forrest's [vectara-python-cli](https://vectara-python-cli.readthedocs.io/en/latest/crash_course.html).
+3. Learn more about Vectara's [Boomerang embedding model](https://vectara.com/blog/introducing-boomerang-vectaras-new-and-improved-retrieval-model/), [Slingshot reranker](https://vectara.com/blog/deep-dive-into-vectara-multilingual-reranker-v1-state-of-the-art-reranker-across-100-languages/), and [Mockingbird LLM](https://vectara.com/blog/mockingbird-a-rag-and-structured-output-focused-llm/)
 
-For more information about [Vectara](https://vectara.com) and how to use our RAG-as-a-service API platform, check out our [documentation](https://docs.vectara.com/docs/).
+## LLM Hallucination Leaderboard
+If you want to stay up to date with results of the latest tests using this model to evaluate the top LLM models, we have a [public leaderboard](https://huggingface.co/spaces/vectara/leaderboard) that is periodically updated, and results are also available on the [GitHub repository](https://github.com/vectara/hallucination-leaderboard).

added_tokens.json

@@ -1,3 +0,0 @@
-{
-  "[MASK]": 128000
-}

config.json

@@ -1,41 +1,12 @@
 {
-  "_name_or_path": "./cross-encoder-binary-bce_loss/distilbert_nli_epoch_crossencoder_binary-bce-3-score_0.9844",
   "architectures": [
-    "DebertaV2ForSequenceClassification"
+    "HHEMv2ForSequenceClassification"
   ],
-  "attention_probs_dropout_prob": 0.1,
-  "hidden_act": "gelu",
-  "hidden_dropout_prob": 0.1,
-  "hidden_size": 768,
-  "id2label": {
-    "0": "LABEL_0"
+  "auto_map": {
+    "AutoConfig": "configuration_hhem_v2.HHEMv2Config",
+    "AutoModelForSequenceClassification": "modeling_hhem_v2.HHEMv2ForSequenceClassification"
   },
-  "initializer_range": 0.02,
-  "intermediate_size": 3072,
-  "label2id": {
-    "LABEL_0": 0
-  },
-  "layer_norm_eps": 1e-07,
-  "max_position_embeddings": 512,
-  "max_relative_positions": -1,
-  "model_type": "deberta-v2",
-  "norm_rel_ebd": "layer_norm",
-  "num_attention_heads": 12,
-  "num_hidden_layers": 12,
-  "pad_token_id": 0,
-  "pooler_dropout": 0,
-  "pooler_hidden_act": "gelu",
-  "pooler_hidden_size": 768,
-  "pos_att_type": [
-    "p2c",
-    "c2p"
-  ],
-  "position_biased_input": false,
-  "position_buckets": 256,
-  "relative_attention": true,
-  "share_att_key": true,
+  "model_type": "HHEMv2Config",
   "torch_dtype": "float32",
-  "transformers_version": "4.33.3",
-  "type_vocab_size": 0,
-  "vocab_size": 128100
+  "transformers_version": "4.39.3"
 }

configuration_hhem_v2.py

@@ -0,0 +1,22 @@
+from transformers import PretrainedConfig
+
+class HHEMv2Config(PretrainedConfig):
+    model_type = "HHEMv2"
+    foundation = "google/flan-t5-base"
+    prompt = "<pad> Determine if the hypothesis is true given the premise?\n\nPremise: {text1}\n\nHypothesis: {text2}"
+    
+    def __init___(self,
+                  foundation="xyz",
+                  prompt="abc",
+                  **kwargs):
+        super().__init__(**kwargs)
+        self.foundation = foundation
+        self.prompt = prompt
+
+
+# FIXME: The default values passed to the constructor are not used.
+# Instead, the values set as global before the constructor are used. 
+# To test, run this: 
+# config = HHEMv2Config()
+# print(config.foundation)
+# The output will not be xyz but google/flan-t5-base.

leaderboard_summaries.csv

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ab3c650afa79db152974ad311258fb573b1815a020b6f7e8a4c3a8636eb27487
-size 19446482

spm.model → model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd
-size 2464616
+oid sha256:634de18a38cf1e991c1acd0f7a9e0d30f7ea187fba42bb4798f862d3edd31e72
+size 438535352

modeling_hhem_v2.py

@@ -0,0 +1,65 @@
+import torch
+
+from peft import PeftModel
+from transformers import PreTrainedModel, AutoConfig, T5ForTokenClassification, AutoModel, AutoTokenizer, AutoModelForTokenClassification
+
+from .configuration_hhem_v2 import HHEMv2Config
+
+class HHEMv2Model(PreTrainedModel):
+    config_class = HHEMv2Config
+    
+    def __init__(self, config):
+        super().__init__(config)
+    #     self.t5 = T5ForTokenClassification.from_config(
+    #         AutoConfig.from_pretrained(config.foundation)
+    #     )
+
+    # def populate(self, model): 
+    #     self.t5 = model
+
+    # def forward(self, **kwarg):
+    #     return self.t5.transformer(**kwarg)
+
+class HHEMv2ForSequenceClassification(PreTrainedModel):
+    config_class = HHEMv2Config
+    
+    def __init__(self, config=HHEMv2Config()):
+        super().__init__(config)
+        self.t5 = T5ForTokenClassification(
+            AutoConfig.from_pretrained(config.foundation)
+        )
+        self.prompt = config.prompt
+        self.tokenzier = AutoTokenizer.from_pretrained(config.foundation)
+
+    def populate(self, model: AutoModel): 
+        """Initiate the model with the pretrained model
+                
+        This method should only be called by Vectara employee who prepares the model for publishing. Users do not need to call this method.
+
+        """
+        self.t5 = model
+
+    # TODO: Figure out how to publish only the adapter yet still able to do end-to-end pulling and inference.
+    # def populate_lora(self, checkpoint: str): 
+    #     base_model = AutoModelForTokenClassification.from_pretrained(self.config.foundation)
+    #     combined_model = PeftModel.from_pretrained(base_model, checkpoint, is_trainable=False)
+    #     self.t5 = combined_model
+
+    def forward(self, **kwargs):
+        return self.t5(**kwargs)
+
+    def predict(self, text_pairs):
+        tokenizer = self.tokenzier
+        pair_dict = [{'text1': pair[0], 'text2': pair[1]} for pair in text_pairs]
+        inputs = tokenizer(
+            [self.prompt.format(**pair) for pair in pair_dict], return_tensors='pt', padding=True)
+        self.t5.eval()
+        with torch.no_grad():
+            outputs = self.t5(**inputs)
+        logits = outputs.logits    
+        logits = logits[:, 0, :] # tok_cls
+        transformed_probs = torch.softmax(logits, dim=-1)
+        raw_scores = transformed_probs[:, 1] # the probability of class 1
+        return raw_scores
+
+    

pytorch_model.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:eb3789996c599ab27a5506005f03229a0a4600fe3634d121c48731ae95cde40e
-size 737761269

special_tokens_map.json

@@ -1,9 +0,0 @@
-{
-  "bos_token": "[CLS]",
-  "cls_token": "[CLS]",
-  "eos_token": "[SEP]",
-  "mask_token": "[MASK]",
-  "pad_token": "[PAD]",
-  "sep_token": "[SEP]",
-  "unk_token": "[UNK]"
-}

tokenizer_config.json

@@ -1,23 +0,0 @@
-{
-  "bos_token": "[CLS]",
-  "clean_up_tokenization_spaces": true,
-  "cls_token": "[CLS]",
-  "do_lower_case": false,
-  "eos_token": "[SEP]",
-  "mask_token": "[MASK]",
-  "max_length": 512,
-  "model_max_length": 512,
-  "pad_to_multiple_of": null,
-  "pad_token": "[PAD]",
-  "pad_token_type_id": 0,
-  "padding_side": "right",
-  "sep_token": "[SEP]",
-  "sp_model_kwargs": {},
-  "split_by_punct": false,
-  "stride": 0,
-  "tokenizer_class": "DebertaV2Tokenizer",
-  "truncation_side": "right",
-  "truncation_strategy": "longest_first",
-  "unk_token": "[UNK]",
-  "vocab_type": "spm"
-}