hagsaeng/MachinLearningBootCamp_QAclassifier_Gemma-2B

Model Card: Question Classification using LoRA with Quantization

Model Overview

This model is a fine-tuned version of google/gemma-2b-it designed to classify text into two categories: QUESTION or NOT_QUESTION. It was fine-tuned on a custom dataset that combines the SQuAD dataset (containing questions) and the GLUE SST-2 dataset (containing general non-question sentences).

Model Architecture

• Base Model: google/gemma-2b-it
• Fine-tuning Method: LoRA (Low-Rank Adaptation) with k-bit quantization (4-bit quantization with NF4).
• Configurations:
  • Quantization: 4-bit quantization using BitsAndBytesConfig
  • Adapter (LoRA) settings:
    • Rank: 64
    • LoRA Alpha: 32
    • Dropout: 0.05
    • Target Modules: q_proj, k_proj, v_proj, o_proj

Dataset

The model was trained using a combination of two datasets:

• SQuAD v1.1 (Question dataset)
• GLUE SST-2 (Non-question dataset)

Each dataset was preprocessed to contain a label:

• QUESTION: For SQuAD questions
• NOT_QUESTION: For non-question sentences from GLUE SST-2.

Data Preprocessing

• A random removal probability (P_remove = 0.3) was applied to remove some of the questions containing a question mark (?), to increase the model's robustness.
• Both datasets were balanced with an equal number of samples (N=100 for training and testing).

Model Performance

• Metrics Evaluated:
  • Accuracy
  • F1 Score
  • Precision
  • Recall
• These metrics were computed on a balanced test dataset containing both question and non-question examples.

How to Use

You can use this model to classify whether a given text is a question or not. Here’s how you can use it:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("your_model_name")
model = AutoModelForSequenceClassification.from_pretrained("your_model_name")

inputs = tokenizer("What is the capital of France?", return_tensors="pt")
outputs = model(**inputs)
predictions = torch.argmax(outputs.logits, axis=1)

label = "QUESTION" if predictions == 1 else "NOT_QUESTION"
print(f"Predicted Label: {label}")

Limitations

• The model was trained on English data only, so it may not perform well on non-English languages.
• Since it is fine-tuned on specific datasets (SQuAD and GLUE SST-2), performance may vary with out-of-domain data.
• The model assumes well-formed input sentences, so performance may degrade with informal or very short text.

Intended Use

This model is intended for text classification tasks where distinguishing between questions and non-questions is needed. Potential use cases include:

• Improving chatbot or virtual assistant interactions.
• Enhancing query detection for search engines.

License

This model follows the same license as google/gemma-2b-it. Please refer to the original license for any usage restrictions.