|
--- |
|
license: apache-2.0 |
|
tags: |
|
- setfit |
|
- sentence-transformers |
|
- text-classification |
|
pipeline_tag: text-classification |
|
--- |
|
|
|
# 😵💫🦙 Alpaca HalluciHunter |
|
|
|
This is a cross-lingual SetFit model [SetFit model](https://github.com/huggingface/setfit) to detect potentially bad instructions from Alpaca (and likely other synthetically generated instruction datasets). |
|
|
|
The model has been fine-tuned with 1,000 labeled examples from the AlpacaCleaned dataset. It leverages a multilingual sentence transformer `paraphrase-multilingual-mpnet-base-v2`, inspired by the findings from the SetFit paper (Section 6. Multilingual experiments.), where they trained models in English that performed well across languages. |
|
|
|
It's a binary classifier with two labels: |
|
|
|
- `ALL GOOD`, a given instruction, input, and output are correct, |
|
- `BAD INSTRUCTION`, there's an issue with the instruction, and/or input and output. |
|
|
|
|
|
This model can greatly speed up the validation of Alpaca Datasets, flagging examples that need to be fixed or simply discarded. |
|
|
|
## Usage |
|
|
|
To use this model for inference, first install the SetFit library: |
|
|
|
```bash |
|
python -m pip install setfit |
|
``` |
|
|
|
Load your Alpaca Dataset: |
|
|
|
```bash |
|
from datasets import Dataset, load_dataset |
|
|
|
import pandas as pd |
|
|
|
# this can be a translation (e.g., Spanish, Camoscio Italian Alpaca, etc.) |
|
dataset = pd.read_json("https://github.com/gururise/AlpacaDataCleaned/raw/main/alpaca_data_cleaned.json") |
|
|
|
dataset["id"] = [i for i in range(len(dataset))] |
|
|
|
ds = Dataset.from_pandas(dataset) |
|
``` |
|
|
|
Create a text field containing the instruction, input and output to use for inference: |
|
|
|
```python |
|
def transform(r): |
|
return { |
|
"text": f"INSTRUCTION:\n{r['instruction']}\nINPUT:\n{r['input']}\nOUTPUT:\n{r['output']}\n" |
|
} |
|
ds = ds.map(transform) |
|
``` |
|
|
|
Load the model: |
|
|
|
```python |
|
from setfit import SetFitModel |
|
|
|
# Download from Hub |
|
model = SetFitModel.from_pretrained("argilla/alpaca-hallucihunter-multilingual ") |
|
``` |
|
|
|
Perform inference and prediction col to your dataset: |
|
```python |
|
labels = ["ALL GOOD", "BAD INSTRUCTION"] |
|
|
|
def get_predictions(texts): |
|
probas = model.predict_proba(texts, as_numpy=True) |
|
for pred in probas: |
|
yield [{"label": label, "score": score} for label, score in zip(labels, pred)] |
|
|
|
ds = ds.map(lambda batch: {"prediction": list(get_predictions(batch["text"]))}, batched=True) |
|
``` |
|
|
|
Load the data into Argilla for exploration and validation. You [need to launch Argilla](https://www.argilla.io/blog/launching-argilla-huggingface-hub): |
|
```python |
|
# Replace api_url with the url to your HF Spaces URL if using Spaces |
|
# Replace api_key if you configured a custom API key |
|
rg.init( |
|
api_url="https://your-agilla-instance.hf.space", |
|
api_key="team.apikey" |
|
) |
|
|
|
rg_dataset = rg.DatasetForTextClassification().from_datasets(ds) |
|
rg.log(records=rg_dataset, name="alpaca_to_clean") |
|
``` |
|
|
|
## Examples |
|
|
|
|
|
|
|
## BibTeX entry and citation info |
|
|
|
```bibtex |
|
@article{https://doi.org/10.48550/arxiv.2209.11055, |
|
doi = {10.48550/ARXIV.2209.11055}, |
|
url = {https://arxiv.org/abs/2209.11055}, |
|
author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren}, |
|
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences}, |
|
title = {Efficient Few-Shot Learning Without Prompts}, |
|
publisher = {arXiv}, |
|
year = {2022}, |
|
copyright = {Creative Commons Attribution 4.0 International} |
|
} |
|
``` |
|
|
