Using the evaluator
with custom pipelines
The evaluator is designed to work with transformer
pipelines out-of-the-box. However, in many cases you might have a model or pipeline that’s not part of the transformer
ecosystem. You can still use evaluator
to easily compute metrics for them. In this guide we show how to do this for a Scikit-Learn pipeline and a Spacy pipeline. Let’s start with the Scikit-Learn case.
Scikit-Learn
First we need to train a model. We’ll train a simple text classifier on the IMDb dataset, so let’s start by downloading the dataset:
from datasets import load_dataset
ds = load_dataset("imdb")
Then we can build a simple TF-IDF preprocessor and Naive Bayes classifier wrapped in a Pipeline
:
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer
text_clf = Pipeline([
('vect', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', MultinomialNB()),
])
text_clf.fit(ds["train"]["text"], ds["train"]["label"])
Following the convention in the TextClassificationPipeline
of transformers
our pipeline should be callable and return a list of dictionaries. In addition we use the task
attribute to check if the pipeline is compatible with the evaluator
. We can write a small wrapper class for that purpose:
class ScikitEvalPipeline:
def __init__(self, pipeline):
self.pipeline = pipeline
self.task = "text-classification"
def __call__(self, input_texts, **kwargs):
return [{"label": p} for p in self.pipeline.predict(input_texts)]
pipe = ScikitEvalPipeline(text_clf)
We can now pass this pipeline
to the evaluator
:
from evaluate import evaluator
task_evaluator = evaluator("text-classification")
task_evaluator.compute(pipe, ds["test"], "accuracy")
>>> {'accuracy': 0.82956}
Implementing that simple wrapper is all that’s needed to use any model from any framework with the evaluator
. In the __call__
you can implement all logic necessary for efficient forward passes through your model.
Spacy
We’ll use the polarity
feature of the spacytextblob
project to get a simple sentiment analyzer. First you’ll need to install the project and download the resources:
pip install spacytextblob python -m textblob.download_corpora python -m spacy download en_core_web_sm
Then we can simply load the nlp
pipeline and add the spacytextblob
pipeline:
import spacy
nlp = spacy.load('en_core_web_sm')
nlp.add_pipe('spacytextblob')
This snippet shows how we can use the polarity
feature added with spacytextblob
to get the sentiment of a text:
texts = ["This movie is horrible", "This movie is awesome"]
results = nlp.pipe(texts)
for txt, res in zip(texts, results):
print(f"{text} | Polarity: {res._.blob.polarity}")
Now we can wrap it in a simple wrapper class like in the Scikit-Learn example before. It just has to return a list of dictionaries with the predicted lables. If the polarity is larger than 0 we’ll predict positive sentiment and negative otherwise:
class SpacyEvalPipeline:
def __init__(self, nlp):
self.nlp = nlp
self.task = "text-classification"
def __call__(self, input_texts, **kwargs):
results =[]
for p in self.nlp.pipe(input_texts):
if p._.blob.polarity>=0:
results.append({"label": 1})
else:
results.append({"label": 0})
return results
pipe = SpacyEvalPipeline(nlp)
That class is compatible with the evaluator
and we can use the same instance from the previous examlpe along with the IMDb test set:
eval.compute(pipe, ds["test"], "accuracy")
>>> {'accuracy': 0.6914}
This will take a little longer than the Scikit-Learn example but after roughly 10-15min you will have the evaluation results!