T5¶

DISCLAIMER: This model is still a work in progress, if you see something strange, file a Github Issue

Overview¶

The T5 model was presented in Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu in Here the abstract:

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.

The Authors’ code can be found here .

Training¶

T5 is an encoder-decoder model and converts all NLP problems into a text-to-text format. It is trained using teacher forcing. This means that for training we always need an input sequence and a target sequence. The input sequence is fed to the model using input_ids. The target sequence is shifted to the right, i.e. prepended by a start-sequence token and fed to the decoder using the decoder_input_ids. In teacher-forcing style, the target sequence is then appended by the EOS token and corresponds to the lm_labels. The PAD token is hereby used as the start-sequence token. T5 can be trained / fine-tuned both in a supervised and unsupervised fashion.

Unsupervised denoising training

In this setup spans of the input sequence are masked by so-called sentinel tokens (a.k.a unique mask tokens) and the output sequence is formed as a concatenation of the same sentinel tokens and the real masked tokens. Each sentinel token represents a unique mask token for this sentence and should start with <extra_id_1>, <extra_id_2>, … up to <extra_id_100>. As a default 100 sentinel tokens are available in T5Tokenizer. E.g. the sentence “The cute dog walks in the park” with the masks put on “cute dog” and “the” should be processed as follows:

input_ids = tokenizer.encode('The <extra_id_1> walks in <extra_id_2> park', return_tensors='pt')
lm_labels = tokenizer.encode('<extra_id_1> cute dog <extra_id_2> the <extra_id_3> </s>', return_tensors='pt')
# the forward function automatically creates the correct decoder_input_ids
model(input_ids=input_ids, lm_labels=lm_labels)

Supervised training

In this setup the input sequence and output sequence are standard sequence to sequence input output mapping. In translation, e.g. the input sequence “The house is wonderful.” and output sequence “Das Haus ist wunderbar.” should be processed as follows:

input_ids = tokenizer.encode('translate English to German: The house is wonderful. </s>', return_tensors='pt')
lm_labels = tokenizer.encode('Das Haus ist wunderbar. </s>', return_tensors='pt')
# the forward function automatically creates the correct decoder_input_ids
model(input_ids=input_ids, lm_labels=lm_labels)

Tips¶

T5 is an encoder-decoder model pre-trained on a multi-task mixture of unsupervised and supervised tasks and for which each task is converted into a text-to-text format. T5 works well on a variety of tasks out-of-the-box by prepending a different prefix to the input corresponding to each task, e.g.: for translation: translate English to German: …, summarize: …. For more information about which prefix to use, it is easiest to look into Appendix D of the paper .
For sequence to sequence generation, it is recommended to use T5ForConditionalGeneration.generate(). The method takes care of feeding the encoded input via cross-attention layers to the decoder and auto-regressively generates the decoder output.
T5 uses relative scalar embeddings. Encoder input padding can be done on the left and on the right.

The original code can be found here.

T5Config¶

class transformers.T5Config(vocab_size=32128, n_positions=512, d_model=512, d_kv=64, d_ff=2048, num_layers=6, num_heads=8, relative_attention_num_buckets=32, dropout_rate=0.1, layer_norm_epsilon=1e-06, initializer_factor=1.0, is_encoder_decoder=True, pad_token_id=0, eos_token_id=1, **kwargs)[source]¶

T5Config is the configuration class to store the configuration of a T5Model.

Parameters

vocab_size_or_config_json_file – Vocabulary size of inputs_ids in T5Model.
d_model – Size of the encoder layers and the pooler layer. d_model can also accesed via the property hidden_size.
num_layers – Number of hidden layers in the Transformer encoder. num_layers can also be accessed via the property num_hidden_layers.
num_heads – Number of attention heads for each attention layer in the Transformer encoder. num_heads can also be accessed via the property num_attention_heads.
intermediate_size – The size of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.
hidden_act – The non-linear activation function (function or string) in the encoder and pooler. If string, “gelu”, “relu”, “swish” and “gelu_new” are supported.
hidden_dropout_prob – The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
attention_probs_dropout_prob – The dropout ratio for the attention probabilities.
n_positions – The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). n_positions can also be accessed via the property `max_position_embeddings’.
type_vocab_size – The vocabulary size of the token_type_ids passed into T5Model.
initializer_factor – A factor for initializing all weight matrices (should be kept to 1.0, used for initialization testing).
layer_norm_eps – The epsilon used by LayerNorm.

T5Tokenizer¶

class transformers.T5Tokenizer(vocab_file, eos_token='</s>', unk_token='<unk>', pad_token='<pad>', extra_ids=100, additional_special_tokens=None, **kwargs)[source]¶

Constructs an XLNet tokenizer. Based on SentencePiece .

This tokenizer inherits from PreTrainedTokenizer which contains most of the methods. Users should refer to the superclass for more information regarding methods.

Parameters

vocab_file (string) – SentencePiece file (generally has a .spm extension) that contains the vocabulary necessary to instantiate a tokenizer.
eos_token (string, optional, defaults to “</s>”) –
The end of sequence token.

Note

When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the sep_token.
unk_token (string, optional, defaults to “<unk>”) – The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead.
pad_token (string, optional, defaults to “<pad>”) – The token used for padding, for example when batching sequences of different lengths.
extra_ids (List[str], optional, defaults to 100) – Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are accessible as “<extra_id_{%d}>” where “{%d}” is a number between 0 and extra_ids-1. Extra tokens are indexed from the end of the vocabulary up to beginnning (“<extra_id_0>” is the last token in the vocabulary like in T5 preprocessing see: https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)
additional_special_tokens (List[str], optional, defaults to None) – Additional special tokens used by the tokenizer.

build_inputs_with_special_tokens(token_ids_0: List, token_ids_1: Optional[List] = None) → List¶: Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. This implementation does not add special tokens.

get_special_tokens_mask(token_ids_0: List, token_ids_1: Optional[List] = None, already_has_special_tokens: bool = False) → List[int]¶

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer prepare_for_model or encode_plus methods.

Parameters

token_ids_0 – list of ids (must not contain special tokens)
token_ids_1 – Optional list of ids (must not contain special tokens), necessary when fetching sequence ids for sequence pairs
already_has_special_tokens – (default False) Set to True if the token list is already formated with special tokens for the model

Returns

1 for a special token, 0 for a sequence token.

Return type

A list of integers in the range [0, 1]

save_vocabulary(save_directory)[source]¶: Save the sentencepiece vocabulary (copy original file) and special tokens file to a directory.

T5Model¶

class transformers.T5Model(config)[source]¶

The bare T5 Model transformer outputting raw hidden-stateswithout any specific head on top. The T5 model was proposed in `Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer`_ by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. It’s an encoder decoder transformer pre-trained in a text-to-text denoising generative setting.

This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters: config (T5Config) – Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

forward(input_ids=None, attention_mask=None, encoder_outputs=None, decoder_input_ids=None, decoder_attention_mask=None, decoder_past_key_value_states=None, use_cache=True, inputs_embeds=None, decoder_inputs_embeds=None, head_mask=None)[source]¶

The T5Model forward method, overrides the __call__() special method.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Parameters

input_ids (torch.LongTensor of shape (batch_size, sequence_length)) – Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using transformers.T5Tokenizer. See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.convert_tokens_to_ids() for details. To know more on how to prepare input_ids for pre-training take a look at T5 Training .
attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) – Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for tokens that are NOT MASKED, 0 for MASKED tokens.
encoder_outputs (tuple(tuple(torch.FloatTensor), optional, defaults to None) – Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions) last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.
decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional, defaults to None) –
Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation. If decoder_past_key_value_states is used, optionally only the last decoder_input_ids have to be input (see decoder_past_key_value_states). To know more on how to prepare decoder_input_ids for pre-training take a look at T5 Training .
decoder_attention_mask (torch.BoolTensor of shape (batch_size, tgt_seq_len), optional, defaults to None) – Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default.
decoder_past_key_value_states (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) – Contains pre-computed key and value hidden-states of the attention blocks. Can be used to speed up decoding. If decoder_past_key_value_states are used, the user can optionally input only the last decoder_input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all decoder_input_ids of shape (batch_size, sequence_length).
use_cache (bool, optional, defaults to True) – If use_cache is True, decoder_past_key_value_states are returned and can be used to speed up decoding (see decoder_past_key_value_states).
inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
decoder_inputs_embeds (torch.FloatTensor of shape (batch_size, target_sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded representation. If decoder_past_key_value_states is used, optionally only the last decoder_inputs_embeds have to be input (see decoder_past_key_value_states). This is useful if you want more control over how to convert decoder_input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
head_mask – (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional, defaults to None): Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]: 1 indicates the head is not masked, 0 indicates the head is masked.

Returns

tuple(torch.FloatTensor) comprising various elements depending on the configuration (T5Config) and inputs. last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)):

Sequence of hidden-states at the output of the last layer of the model. If decoder_past_key_value_states is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.

decoder_past_key_value_states (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length, embed_size_per_head), optional, returned when use_cache=True):

Contains pre-computed key and value hidden-states of the attention blocks. Can be used to speed up sequential decoding (see decoder_past_key_value_states input). Note that when using decoder_past_key_value_states, the model only outputs the last hidden-state of the sequence of shape (batch_size, 1, config.vocab_size).

hidden_states (tuple(torch.FloatTensor), optional, returned when config.output_hidden_states=True):

Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

Hidden-states of the model at the output of each layer plus the initial embedding outputs.

attentions (tuple(torch.FloatTensor), optional, returned when config.output_attentions=True):

Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Examples:

from transformers import T5Tokenizer, T5Model

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5Model.from_pretrained('t5-small')
input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="pt")  # Batch size 1
outputs = model(input_ids=input_ids, decoder_input_ids=input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

get_input_embeddings()[source]¶

Returns the model’s input embeddings.

Returns: A torch module mapping vocabulary to hidden states.
Return type: nn.Module

set_input_embeddings(new_embeddings)[source]¶

Set model’s input embeddings

Parameters: value (nn.Module) – A module mapping vocabulary to hidden states.

T5ForConditionalGeneration¶

class transformers.T5ForConditionalGeneration(config)[source]¶

T5 Model with a language modeling head on top. The T5 model was proposed in `Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer`_ by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. It’s an encoder decoder transformer pre-trained in a text-to-text denoising generative setting.

This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters: config (T5Config) – Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

forward(input_ids=None, attention_mask=None, encoder_outputs=None, decoder_input_ids=None, decoder_attention_mask=None, decoder_past_key_value_states=None, use_cache=True, lm_labels=None, inputs_embeds=None, decoder_inputs_embeds=None, head_mask=None)[source]¶

The T5ForConditionalGeneration forward method, overrides the __call__() special method.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Parameters

input_ids (torch.LongTensor of shape (batch_size, sequence_length)) –
Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using transformers.T5Tokenizer. See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.convert_tokens_to_ids() for details. To know more on how to prepare input_ids for pre-training take a look at T5 Training .
attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) – Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for tokens that are NOT MASKED, 0 for MASKED tokens.
encoder_outputs (tuple(tuple(torch.FloatTensor), optional, defaults to None) – Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions) last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.
decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional, defaults to None) –
Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation. If decoder_past_key_value_states is used, optionally only the last decoder_input_ids have to be input (see decoder_past_key_value_states). To know more on how to prepare decoder_input_ids for pre-training take a look at T5 Training .
decoder_attention_mask (torch.BoolTensor of shape (batch_size, tgt_seq_len), optional, defaults to None) – Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default.
decoder_past_key_value_states (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) – Contains pre-computed key and value hidden-states of the attention blocks. Can be used to speed up decoding. If decoder_past_key_value_states are used, the user can optionally input only the last decoder_input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all decoder_input_ids of shape (batch_size, sequence_length).
use_cache (bool, optional, defaults to True) – If use_cache is True, decoder_past_key_value_states are returned and can be used to speed up decoding (see decoder_past_key_value_states).
inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
decoder_inputs_embeds (torch.FloatTensor of shape (batch_size, target_sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded representation. If decoder_past_key_value_states is used, optionally only the last decoder_inputs_embeds have to be input (see decoder_past_key_value_states). This is useful if you want more control over how to convert decoder_input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
head_mask – (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional, defaults to None): Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]: 1 indicates the head is not masked, 0 indicates the head is masked.
lm_labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for computing the sequence classification/regression loss. Indices should be in [-100, 0, ..., config.vocab_size - 1]. All labels set to -100 are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size]

Returns

tuple(torch.FloatTensor) comprising various elements depending on the configuration (T5Config) and inputs. loss (torch.FloatTensor of shape (1,), optional, returned when lm_label is provided):

Classification loss (cross entropy).

prediction_scores (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). If past_key_value_states is used only the last prediction_scores of the sequences of shape (batch_size, 1, hidden_size) is output.
decoder_past_key_value_states (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length, embed_size_per_head), optional, returned when use_cache=True):: Contains pre-computed key and value hidden-states of the attention blocks. Can be used to speed up sequential decoding (see decoder_past_key_value_states input). Note that when using decoder_past_key_value_states, the model only outputs the last prediction_score of the sequence of shape (batch_size, 1, config.vocab_size).
hidden_states (tuple(torch.FloatTensor), optional, returned when config.output_hidden_states=True):: Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size). Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when config.output_attentions=True):: Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights after the attention softmax, used to compute the weighted average in the self-attention.

Examples:

from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5ForConditionalGeneration.from_pretrained('t5-small')
input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="pt")  # Batch size 1
outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, lm_labels=input_ids)
loss, prediction_scores = outputs[:2]

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5ForConditionalGeneration.from_pretrained('t5-small')
input_ids = tokenizer.encode("summarize: Hello, my dog is cute", return_tensors="pt")  # Batch size 1
outputs = model.generate(input_ids)

get_input_embeddings()[source]¶

Returns the model’s input embeddings.

Returns: A torch module mapping vocabulary to hidden states.
Return type: nn.Module

get_output_embeddings()[source]¶

Returns the model’s output embeddings.

Returns: A torch module mapping hidden states to vocabulary.
Return type: nn.Module

set_input_embeddings(new_embeddings)[source]¶

Set model’s input embeddings

Parameters: value (nn.Module) – A module mapping vocabulary to hidden states.

TFT5Model¶

class transformers.TFT5Model(*args, **kwargs)[source]¶

The bare T5 Model transformer outputting raw hidden-stateswithout any specific head on top. The T5 model was proposed in `Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer`_ by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. It’s an encoder decoder transformer pre-trained in a text-to-text denoising generative setting.

This model is a tf.keras.Model tf.keras.Model sub-class. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

Note on the model inputs:

TF 2.0 models accepts two formats as inputs:

having all inputs as keyword arguments (like PyTorch models), or

having all inputs as a list, tuple or dict in the first positional arguments.

This second option is usefull when using tf.keras.Model.fit() method which currently requires having all the tensors in the first argument of the model call function: model(inputs).

If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument :

a single Tensor with inputs only and nothing else: `model(inputs_ids)
a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([inputs, attention_mask]) or model([inputs, attention_mask, token_type_ids])
a dictionary with one or several input Tensors associaed to the input names given in the docstring:
model({‘inputs’: inputs, ‘token_type_ids’: token_type_ids})

Parameters: config (T5Config) – Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

call(inputs, **kwargs)[source]¶

The TFT5Model forward method, overrides the __call__() special method.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Parameters

are usually used as a dict (inputs) –
inputs (tf.Tensor of shape (batch_size, sequence_length)) –
Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on the right or the left. Indices can be obtained using transformers.T5Tokenizer. To know more on how to prepare inputs for pre-training take a look at T5 Training . See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.convert_tokens_to_ids() for details.
decoder_input_ids (tf.Tensor of shape (batch_size, target_sequence_length), optional, defaults to None) – Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation. If decoder_past_key_value_states is used, optionally only the last decoder_input_ids have to be input (see decoder_past_key_value_states).
attention_mask (tf.Tensor of shape (batch_size, sequence_length), optional, defaults to None) – Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for tokens that are NOT MASKED, 0 for MASKED tokens.
encoder_outputs (tuple(tuple(tf.FloatTensor), optional, defaults to None) – Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions) last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.
decoder_attention_mask (tf.Tensor of shape (batch_size, tgt_seq_len), optional, defaults to None) – Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default.
decoder_past_key_value_states (tuple(tuple(tf.Tensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) – Contains pre-computed key and value hidden-states of the attention blocks. Can be used to speed up decoding. If decoder_past_key_value_states are used, the user can optionally input only the last decoder_input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, 1)
use_cache (bool, optional, defaults to True) – If use_cache is True, decoder_past_key_value_states are returned and can be used to speed up decoding (see decoder_past_key_value_states).
inputs_embeds (tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing inputs you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert inputs indices into associated vectors than the model’s internal embedding lookup matrix.
decoder_inputs_embeds (tf.Tensor of shape (batch_size, target_sequence_length, hidden_size), optional, defaults to None) –
Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert decoder_input_ids indices into associated vectors than the model’s internal embedding lookup matrix. To know more on how to prepare decoder_input_ids for pre-training take a look at T5 Training .
head_mask – (tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional, defaults to None): Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]: 1 indicates the head is not masked, 0 indicates the head is masked.

Returns

tuple(tf.Tensor) comprising various elements depending on the configuration (T5Config) and inputs. last_hidden_state (tf.Tensor of shape (batch_size, sequence_length, hidden_size)):

Sequence of hidden-states at the output of the last layer of the model. If decoder_past_key_value_states is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.

decoder_past_key_value_states (tuple(tuple(tf.Tensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length, embed_size_per_head), optional, returned when use_cache=True):

Contains pre-computed key and value hidden-states of the attention blocks. Can be used to speed up sequential decoding (see decoder_past_key_value_states input). Note that when using decoder_past_key_value_states, the model only outputs the last hidden-state of the sequence of shape (batch_size, 1, config.vocab_size).

hidden_states (tuple(tf.Tensor), optional, returned when config.output_hidden_states=True):

Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

Hidden-states of the model at the output of each layer plus the initial embedding outputs.

attentions (tuple(tf.Tensor), optional, returned when config.output_attentions=True):

Tuple of tf.Tensor (one for each layer) of shape: (batch_size, num_heads, sequence_length, sequence_length).

Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Examples:

from transformers import T5Tokenizer, TFT5Model

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = TFT5Model.from_pretrained('t5-small')
inputs = tokenizer.encode("Hello, my dog is cute", return_tensors="tf")  # Batch size 1
outputs = model(inputs, decoder_input_ids=inputs)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

get_input_embeddings()[source]¶

Returns the model’s input embeddings.

Returns: A torch module mapping vocabulary to hidden states.
Return type: tf.keras.layers.Layer

get_output_embeddings()[source]¶

Returns the model’s output embeddings.

Returns: A torch module mapping hidden states to vocabulary.
Return type: tf.keras.layers.Layer

TFT5ForConditionalGeneration¶

class transformers.TFT5ForConditionalGeneration(*args, **kwargs)[source]¶

T5 Model with a language modeling head on top. The T5 model was proposed in `Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer`_ by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. It’s an encoder decoder transformer pre-trained in a text-to-text denoising generative setting.

This model is a tf.keras.Model tf.keras.Model sub-class. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

Note on the model inputs:

TF 2.0 models accepts two formats as inputs:

having all inputs as keyword arguments (like PyTorch models), or

having all inputs as a list, tuple or dict in the first positional arguments.

This second option is usefull when using tf.keras.Model.fit() method which currently requires having all the tensors in the first argument of the model call function: model(inputs).

If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument :

a single Tensor with inputs only and nothing else: `model(inputs_ids)
a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([inputs, attention_mask]) or model([inputs, attention_mask, token_type_ids])
a dictionary with one or several input Tensors associaed to the input names given in the docstring:
model({‘inputs’: inputs, ‘token_type_ids’: token_type_ids})

Parameters: config (T5Config) – Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

call(inputs, **kwargs)[source]¶

The TFT5ForConditionalGeneration forward method, overrides the __call__() special method.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Parameters

are usually used as a dict (inputs) –
inputs (tf.Tensor of shape (batch_size, sequence_length)) –
Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on the right or the left. Indices can be obtained using transformers.T5Tokenizer. To know more on how to prepare inputs for pre-training take a look at T5 Training . See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.convert_tokens_to_ids() for details.
decoder_input_ids (tf.Tensor of shape (batch_size, target_sequence_length), optional, defaults to None) – Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation. If decoder_past_key_value_states is used, optionally only the last decoder_input_ids have to be input (see decoder_past_key_value_states).
attention_mask (tf.Tensor of shape (batch_size, sequence_length), optional, defaults to None) – Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for tokens that are NOT MASKED, 0 for MASKED tokens.
encoder_outputs (tuple(tuple(tf.FloatTensor), optional, defaults to None) – Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions) last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.
decoder_attention_mask (tf.Tensor of shape (batch_size, tgt_seq_len), optional, defaults to None) – Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default.
decoder_past_key_value_states (tuple(tuple(tf.Tensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) – Contains pre-computed key and value hidden-states of the attention blocks. Can be used to speed up decoding. If decoder_past_key_value_states are used, the user can optionally input only the last decoder_input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, 1)
use_cache (bool, optional, defaults to True) – If use_cache is True, decoder_past_key_value_states are returned and can be used to speed up decoding (see decoder_past_key_value_states).
inputs_embeds (tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing inputs you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert inputs indices into associated vectors than the model’s internal embedding lookup matrix.
decoder_inputs_embeds (tf.Tensor of shape (batch_size, target_sequence_length, hidden_size), optional, defaults to None) –
Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert decoder_input_ids indices into associated vectors than the model’s internal embedding lookup matrix. To know more on how to prepare decoder_input_ids for pre-training take a look at T5 Training .
head_mask – (tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional, defaults to None): Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]: 1 indicates the head is not masked, 0 indicates the head is masked.

Returns

tuple(tf.Tensor) comprising various elements depending on the configuration (T5Config) and inputs. loss (tf.Tensor of shape (1,), optional, returned when lm_label is provided):

Classification loss (cross entropy).

prediction_scores (tf.Tensor of shape (batch_size, sequence_length, config.vocab_size))

Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

decoder_past_key_value_states (tuple(tuple(tf.Tensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length, embed_size_per_head), optional, returned when use_cache=True):

Contains pre-computed key and value hidden-states of the attention blocks. Can be used to speed up sequential decoding (see decoder_past_key_value_states input). Note that when using decoder_past_key_value_states, the model only outputs the last prediction_score of the sequence of shape (batch_size, 1, config.vocab_size).

hidden_states (tuple(tf.Tensor), optional, returned when config.output_hidden_states=True):

Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

Hidden-states of the model at the output of each layer plus the initial embedding outputs.

attentions (tuple(tf.Tensor), optional, returned when config.output_attentions=True):

Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

Attentions weights after the attention softmax, used to compute the weighted average in the self-attention.

Examples:

from transformers import T5Tokenizer, TFT5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = TFT5ForConditionalGeneration.from_pretrained('t5-small')
inputs = tokenizer.encode("Hello, my dog is cute", return_tensors="tf")  # Batch size 1
outputs = model(inputs, decoder_input_ids=inputs)
prediction_scores = outputs[0]

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = TFT5ForConditionalGeneration.from_pretrained('t5-small')
inputs = tokenizer.encode("summarize: Hello, my dog is cute", return_tensors="tf")  # Batch size 1
model.generate(inputs)

get_input_embeddings()[source]¶

Returns the model’s input embeddings.

Returns: A torch module mapping vocabulary to hidden states.
Return type: tf.keras.layers.Layer

get_output_embeddings()[source]¶

Returns the model’s output embeddings.

Returns: A torch module mapping hidden states to vocabulary.
Return type: tf.keras.layers.Layer