Transformers

You are viewing v4.17.0 version. A newer version v4.46.3 is available.

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Speech2Text

Overview

The Speech2Text model was proposed in fairseq S2T: Fast Speech-to-Text Modeling with fairseq by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino. It’s a transformer-based seq2seq (encoder-decoder) model designed for end-to-end Automatic Speech Recognition (ASR) and Speech Translation (ST). It uses a convolutional downsampler to reduce the length of speech inputs by 3/4th before they are fed into the encoder. The model is trained with standard autoregressive cross-entropy loss and generates the transcripts/translations autoregressively. Speech2Text has been fine-tuned on several datasets for ASR and ST: LibriSpeech, CoVoST 2, MuST-C.

This model was contributed by valhalla. The original code can be found here.

Inference

Speech2Text is a speech model that accepts a float tensor of log-mel filter-bank features extracted from the speech signal. It’s a transformer-based seq2seq model, so the transcripts/translations are generated autoregressively. The generate() method can be used for inference.

The Speech2TextFeatureExtractor class is responsible for extracting the log-mel filter-bank features. The Speech2TextProcessor wraps Speech2TextFeatureExtractor and Speech2TextTokenizer into a single instance to both extract the input features and decode the predicted token ids.

The feature extractor depends on torchaudio and the tokenizer depends on sentencepiece so be sure to install those packages before running the examples. You could either install those as extra speech dependencies with pip install transformers"[speech, sentencepiece]" or install the packages seperately with pip install torchaudio sentencepiece. Also torchaudio requires the development version of the libsndfile package which can be installed via a system package manager. On Ubuntu it can be installed as follows: apt install libsndfile1-dev

ASR and Speech Translation

>>> import torch
>>> from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration
>>> from datasets import load_dataset
>>> import soundfile as sf

>>> model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr")
>>> processor = Speech2TextProcessor.from_pretrained("facebook/s2t-small-librispeech-asr")


>>> def map_to_array(batch):
...     speech, _ = sf.read(batch["file"])
...     batch["speech"] = speech
...     return batch


>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.map(map_to_array)

>>> inputs = processor(ds["speech"][0], sampling_rate=16_000, return_tensors="pt")
>>> generated_ids = model.generate(input_ids=inputs["input_features"], attention_mask=inputs["attention_mask"])

>>> transcription = processor.batch_decode(generated_ids)

Multilingual speech translation

For multilingual speech translation models, eos_token_id is used as the decoder_start_token_id and the target language id is forced as the first generated token. To force the target language id as the first generated token, pass the forced_bos_token_id parameter to the generate() method. The following example shows how to transate English speech to French text using the facebook/s2t-medium-mustc-multilingual-st checkpoint.

>>> import torch
>>> from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration
>>> from datasets import load_dataset
>>> import soundfile as sf

>>> model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-medium-mustc-multilingual-st")
>>> processor = Speech2TextProcessor.from_pretrained("facebook/s2t-medium-mustc-multilingual-st")


>>> def map_to_array(batch):
...     speech, _ = sf.read(batch["file"])
...     batch["speech"] = speech
...     return batch


>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.map(map_to_array)

>>> inputs = processor(ds["speech"][0], sampling_rate=16_000, return_tensors="pt")
>>> generated_ids = model.generate(
...     input_ids=inputs["input_features"],
...     attention_mask=inputs["attention_mask"],
...     forced_bos_token_id=processor.tokenizer.lang_code_to_id["fr"],
... )

>>> translation = processor.batch_decode(generated_ids)

See the model hub to look for Speech2Text checkpoints.

Transformers

Speech2Text

Overview

Inference

Speech2TextConfig

class transformers.Speech2TextConfig

Speech2TextTokenizer

class transformers.Speech2TextTokenizer

build_inputs_with_special_tokens

get_special_tokens_mask

create_token_type_ids_from_sequences

save_vocabulary

Speech2TextFeatureExtractor

class transformers.Speech2TextFeatureExtractor

__call__

Speech2TextProcessor

class transformers.Speech2TextProcessor

__call__

from_pretrained

save_pretrained

batch_decode

decode

as_target_processor

Speech2TextModel

class transformers.Speech2TextModel

forward

Speech2TextForConditionalGeneration

class transformers.Speech2TextForConditionalGeneration

forward

TFSpeech2TextModel

class transformers.TFSpeech2TextModel

call

TFSpeech2TextForConditionalGeneration

class transformers.TFSpeech2TextForConditionalGeneration

call

call

call