nicolay-r/flan-t5-tsa-thor-base

Model Card for Model ID

Model Details

This model represent a Chain-of-Thought tuned verson Flan-T5 on Target Sentiment Analysis (TSA) task, using training data of RuSentNE-2023 collection.

This model is designed for texts written in English. Since the original collection reprsent non-english texts, the content has been automatically translated into English using [googletrans].

For the given input sentence and mentioned entity in it (target), this model predict author state by answering one of the following classes: [positive, negaitive, neutral]

Model Description

• Developed by: Reforged by nicolay-r, initial credits for implementation to scofield7419
• Model type: Flan-T5
• Language(s) (NLP): English
• License: Apache License 2.0

Model Sources

• Repository: Reasoning-for-Sentiment-Analysis-Framework
• Paper: https://arxiv.org/abs/2404.12342
• Demo: We have a code on Google-Colab for launching the related model

Uses

Direct Use

This sequence of scripts represent a purely torch and transformers based model usage for inference.

This example is also available on GoogleColab

Here are the following three steps for a quick start with model application:

1. Loading model and tokenizer

import torch
from transformers import AutoTokenizer, T5ForConditionalGeneration

# Setup model path.
model_path = "nicolay-r/flan-t5-tsa-thor-base"
# Setup device.
device = "cuda:0"

model = T5ForConditionalGeneration.from_pretrained(model_path, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path)
model.to(device)

2. Setup ask method for generating LLM responses

def ask(prompt):
  inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
  inputs.to(device)
  output = model.generate(**inputs, temperature=1)
  return tokenizer.batch_decode(output, skip_special_tokens=True)[0]

2. Setup Chain-of-Thought

def target_sentiment_extraction(sentence, target):
  # Setup labels.
  labels_list = ['neutral', 'positive', 'negative']
  # Setup Chain-of-Thought
  step1 = f"Given the sentence {sentence}, which specific aspect of {target} is possibly mentioned?"
  aspect = ask(step1)
  step2 = f"{step1}. The mentioned aspect is about {aspect}. Based on the common sense, what is the implicit opinion towards the mentioned aspect of {target}, and why?"
  opinion = ask(step2)
  step3 = f"{step2}. The opinion towards the mentioned aspect of {target} is {opinion}. Based on such opinion, what is the sentiment polarity towards {target}?"
  emotion_state = ask(step3)
  step4 = f"{step3}. The sentiment polarity is {emotion_state}. Based on these contexts, summarize and return the sentiment polarity only, " + "such as: {}.".format(", ".join(labels_list))
  # Return the final response.
  return ask(step4)

Finally, you can infer model results as follows:

# Input sentence.
sentence = "I would support him despite his bad behavior."
# Input target.
target = "him"
# output response
flant5_response = target_sentiment_extraction(sentence, target)
print(f"Author opinion towards `{target}` in `{sentence}` is:\n{flant5_response}")

The response of the model is as follows:

Author opinion towards "him" in "I would support him despite his bad behavior." is: positive

Downstream Use

Please refer to the related section of the Reasoning-for-Sentiment-Analysis Framework

With this example it applies this model in the THoR mode to the validation data of the RuSentNE-2023 competition for evaluation.

python thor_finetune.py -m "nicolay-r/flan-t5-tsa-thor-base" -r "thor" -d "rusentne2023" -z -bs 16 -f "./config/config.yaml"

Following the Google Colab Notebook for implementation reproduction.

Out-of-Scope Use

This model represent a fine-tuned version of the Flan-T5 on RuSentNE-2023 dataset. Since dataset represent three-scale output answers (positive, negative, neutral), the behavior in general might be biased to this particular task.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Please proceed with the code from the related Three-Hop-Reasoning CoT section.

Or following the related section on Google Colab notebook

Training Details

Training Data

We utilize train data which was automatically translated into English using GoogleTransAPI. The initial source of the texts written in Russian, is from the following repository: https://github.com/dialogue-evaluation/RuSentNE-evaluation

The translated version on the dataset in English could be automatically downloaded via the following script: https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/rusentne23_download.py

Training Procedure

This model has been trained using the Three-hop-Reasoning framework, proposed in the paper: https://arxiv.org/abs/2305.11255

For training procedure accomplishing, the reforged version of this framework was used: https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework

Google-colab notebook for reproduction: https://colab.research.google.com/github/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/Reasoning_for_Sentiment_Analysis_Framework.ipynb

Setup: Flan-T5-base, output up to 300 tokens, 16-batch size.

GPU: NVidia-A100, ~4 min/epoch, temperature 1.0, float 32

The overall training process took 5 epochs.

Training Hyperparameters

• Training regime: All the configuration details were highlighted in the related config file

Evaluation

Testing Data, Factors & Metrics

Testing Data

The direct link to the test evaluation data: https://github.com/dialogue-evaluation/RuSentNE-evaluation/blob/main/final_data.csv

Metrics

For the model evaluation, two metrics were used:

1. F1_PN -- F1-measure over positive and negative classes;
2. F1_PN0 -- F1-measure over positive, negative, and neutral classes;

Results

The test evaluation for this model showcases the F1_PN = 62.715

Below is the log of the training process that showcases the final peformance on the RuSentNE-2023 test set after 4 epochs (lines 5-6):

    F1_PN  F1_PN0  default   mode
0  45.523  59.375   59.375  valid
1  62.345  70.260   70.260  valid
2  62.722  70.704   70.704  valid
3  62.721  70.671   70.671  valid
4  62.357  70.247   70.247  valid
5  60.024  68.171   68.171   test
6  60.024  68.171   68.171   test