api-inference

Text Generation

Generate text based on a prompt.

If you are interested in a Chat Completion task, which generates a response based on a list of messages, check out the chat-completion task.

Recommended models

• google/gemma-2-2b-it: A text-generation model trained to follow instructions.
• meta-llama/Meta-Llama-3.1-8B-Instruct: Very powerful text generation model trained to follow instructions.
• microsoft/Phi-3-mini-4k-instruct: Small yet powerful text generation model.
• Qwen/Qwen2.5-7B-Instruct: Strong text generation model to follow instructions.

Explore all available models and find the one that suits you best here.

Using the API

import requests

API_URL = "https://api-inference.huggingface.co/models/google/gemma-2-2b-it"
headers = {"Authorization": "Bearer hf_***"}

def query(payload):
	response = requests.post(API_URL, headers=headers, json=payload)
	return response.json()
	
output = query({
	"inputs": "Can you please let us know more details about your ",
})

API specification

Request

Some options can be configured by passing headers to the Inference API. Here are the available headers:

For more information about Inference API headers, check out the parameters guide.

Response

Output type depends on the stream input parameter. If stream is false (default), the response will be a JSON object with the following fields:

If stream is true, generated tokens are returned as a stream, using Server-Sent Events (SSE). For more information about streaming, check out this guide.