SetFit AG News

This is a SetFit classifier fine-tuned on the AG News dataset. The model was created following the Outperform OpenAI GPT-3 with SetFit for text-classifiation blog post of Philipp Schmid.

The model achieves an accuracy of 0.87 on the test set and was only trained with 32 total examples (8 per class).

***** Running evaluation *****
model used: sentence-transformers/all-mpnet-base-v2
train dataset: 32 samples
accuracy: 0.8731578947368421

What is SetFit?

"SetFit" (https://arxiv.org/abs/2209.11055) is a new approach that can be used to create high accuracte text-classification models with limited labeled data. SetFit is outperforming GPT-3 in 7 out of 11 tasks, while being 1600x smaller. Check out the blog to learn more: Outperform OpenAI GPT-3 with SetFit for text-classifiation

Inference Endpoints

The model repository also implements a generic custom handler.py as an example for how to use SetFit models with inference-endpoints.

Code: https://huggingface.co/philschmid/setfit-ag-news-endpoint/blob/main/handler.py

Send requests with Pyton

We are going to use requests to send our requests. (make your you have it installed pip install requests)

import json
import requests as r

ENDPOINT_URL=""# url of your endpoint
HF_TOKEN=""

# payload samples
regular_payload = { "inputs": "Coming to The Rescue Got a unique problem? Not to worry: you can find a financial planner for every specialized need"}

# HTTP headers for authorization
headers= {
    "Authorization": f"Bearer {HF_TOKEN}",
    "Content-Type": "application/json"
}

# send request
response = r.post(ENDPOINT_URL, headers=headers, json=paramter_payload)
classified = response.json()

print(classified)
# [ { "label": "World", "score": 0.12341519122860946 }, { "label": "Sports", "score": 0.11741269832494523 }, { "label": "Business", "score": 0.6124446065942992 }, { "label": "Sci/Tech", "score": 0.14672750385214603 } ]

curl example

curl https://YOURDOMAIN.us-east-1.aws.endpoints.huggingface.cloud \
-X POST \
-d '{"inputs": "Coming to The Rescue Got a unique problem? Not to worry: you can find a financial planner for every specialized need"}' \
-H "Authorization: Bearer XXX" \
-H "Content-Type: application/json"