Longformer

DISCLAIMER: This model is still a work in progress, if you see something strange, file a Github Issue

Overview

The Longformer model was presented in Longformer: The Long-Document Transformer by Iz Beltagy, Matthew E. Peters, Arman Cohan. Here the abstract:

Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer’s attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA.

The Authors’ code can be found here .

Longformer Self Attention

Longformer self attention employs self attention on both a “local” context and a “global” context. Most tokens only attend “locally” to each other meaning that each token attends to its \(\frac{1}{2} w\) previous tokens and \(\frac{1}{2} w\) succeding tokens with \(w\) being the window length as defined in config.attention_window. Note that config.attention_window can be of type list to define a different \(w\) for each layer. A selecetd few tokens attend “globally” to all other tokens, as it is conventionally done for all tokens in e.g. BertSelfAttention.

Note that “locally” and “globally” attending tokens are projected by different query, key and value matrices. Also note that every “locally” attending token not only attends to tokens within its window \(w\), but also to all “globally” attending tokens so that global attention is symmetric.

The user can define which tokens attend “locally” and which tokens attend “globally” by setting the tensor global_attention_mask at run-time appropriately. Longformer employs the following logic for global_attention_mask: 0 - the token attends “locally”, 1 - token attends “globally”. For more information please also refer to forward() method.

Using Longformer self attention, the memory and time complexity of the query-key matmul operation, which usually represents the memory and time bottleneck, can be reduced from \(\mathcal{O}(n_s \times n_s)\) to \(\mathcal{O}(n_s \times w)\), with \(n_s\) being the sequence length and \(w\) being the average window size. It is assumed that the number of “globally” attending tokens is insignificant as compared to the number of “locally” attending tokens.

For more information, please refer to the official paper .

Training

LongformerForMaskedLM is trained the exact same way, RobertaForMaskedLM is trained and should be used as follows:

input_ids = tokenizer.encode('This is a sentence from [MASK] training data', return_tensors='pt')
mlm_labels = tokenizer.encode('This is a sentence from the training data', return_tensors='pt')

loss = model(input_ids, labels=input_ids, masked_lm_labels=mlm_labels)[0]

LongformerConfig

class transformers.LongformerConfig(attention_window: Union[List[int], int] = 512, sep_token_id: int = 2, **kwargs)

This is the configuration class to store the configuration of a LongformerModel. It is used to instantiate an Longformer model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the RoBERTa roberta-base architecture with a sequence length 4,096.

The LongformerConfig class directly inherits RobertaConfig. It reuses the same defaults. Please check the parent class for more information.

Parameters

attention_window (int or List[int], optional, defaults to 512) – Size of an attention window around each token. If int, use the same size for all layers. To specify a different window size for each layer, use a List[int] where len(attention_window) == num_hidden_layers.

Example:

from transformers import LongformerConfig, LongformerModel

# Initializing a Longformer configuration
configuration = LongformerConfig()

# Initializing a model from the configuration
model = LongformerModel(configuration)

# Accessing the model configuration
configuration = model.config

LongformerTokenizer

class transformers.LongformerTokenizer(vocab_file, merges_file, errors='replace', bos_token='<s>', eos_token='</s>', sep_token='</s>', cls_token='<s>', unk_token='<unk>', pad_token='<pad>', mask_token='<mask>', **kwargs)

LongformerModel

class transformers.LongformerModel(config)

The bare Longformer Model outputting raw hidden-states without any specific head on top.

This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters

config (LongformerConfig) – Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

This class overrides RobertaModel to provide the ability to process long sequences following the selfattention approach described in `Longformer: the Long-Document Transformer`_by Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer selfattention combines a local (sliding window) and global attention to extend to long documents without the O(n^2) increase in memory and compute.

The selfattention module LongformerSelfAttention implemented here supports the combination of local and global attention but it lacks support for autoregressive attention and dilated attention. Autoregressive and dilated attention are more relevant for autoregressive language modeling than finetuning on downstream tasks. Future release will add support for autoregressive attention, but the support for dilated attention requires a custom CUDA kernel to be memory and compute efficient.

config_class

alias of transformers.configuration_longformer.LongformerConfig

forward(input_ids=None, attention_mask=None, global_attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, masked_lm_labels=None)

The LongformerModel forward method, overrides the __call__() special method.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Parameters

• input_ids (torch.LongTensor of shape (batch_size, sequence_length)) –

  Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using transformers.LonmgformerTokenizer. See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.encode_plus() for details.

  What are input IDs?

• attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for tokens that are NOT MASKED, 0 for MASKED tokens.

  What are attention masks?

• global_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) – Mask to decide the attention given on each token, local attention or global attenion. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the <s> token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the Longformer paper https://arxiv.org/abs/2004.05150 for more details. Mask values selected in [0, 1]: 0 for local attention (a sliding window attention), 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).

• token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token

  What are token type IDs?

• position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].

  What are position IDs?

• inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.

Returns

masked_lm_loss (optional, returned when masked_lm_labels is provided) torch.FloatTensor of shape (1,):

Masked language modeling loss.

prediction_scores (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size))

Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

hidden_states (tuple(torch.FloatTensor), optional, returned when config.output_hidden_states=True):

Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

Hidden-states of the model at the output of each layer plus the initial embedding outputs.

attentions (tuple(torch.FloatTensor), optional, returned when config.output_attentions=True):

Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Return type

tuple(torch.FloatTensor) comprising various elements depending on the configuration (RobertaConfig) and inputs

Examples:

import torch
from transformers import LongformerModel, LongformerTokenizer

model = LongformerModel.from_pretrained('allenai/longformer-base-4096')
tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')

SAMPLE_TEXT = ' '.join(['Hello world! '] * 1000)  # long input document
input_ids = torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0)  # batch of size 1

# Attention mask values -- 0: no attention, 1: local attention, 2: global attention
attention_mask = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
attention_mask[:, [1, 4, 21,]] = 2  # Set global attention based on the task. For example,
                                    # classification: the <s> token
                                    # QA: question tokens
                                    # LM: potentially on the beginning of sentences and paragraphs
sequence_output, pooled_output = model(input_ids, attention_mask=attention_mask)

LongformerForMaskedLM

class transformers.LongformerForMaskedLM(config)

Longformer Model with a language modeling head on top.

This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters

config (LongformerConfig) – Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

config_class

alias of transformers.configuration_longformer.LongformerConfig

forward(input_ids=None, attention_mask=None, global_attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, masked_lm_labels=None)

The LongformerForMaskedLM forward method, overrides the __call__() special method.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Parameters

• input_ids (torch.LongTensor of shape (batch_size, sequence_length)) –

  Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using transformers.LonmgformerTokenizer. See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.encode_plus() for details.

  What are input IDs?

• attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for tokens that are NOT MASKED, 0 for MASKED tokens.

  What are attention masks?

• global_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) – Mask to decide the attention given on each token, local attention or global attenion. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the <s> token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the Longformer paper https://arxiv.org/abs/2004.05150 for more details. Mask values selected in [0, 1]: 0 for local attention (a sliding window attention), 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).

• token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token

  What are token type IDs?

• position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].

  What are position IDs?

• inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.

• masked_lm_labels (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) – Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size]

Returns

masked_lm_loss (optional, returned when masked_lm_labels is provided) torch.FloatTensor of shape (1,):

Masked language modeling loss.

prediction_scores (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size))

Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

hidden_states (tuple(torch.FloatTensor), optional, returned when config.output_hidden_states=True):

Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

Hidden-states of the model at the output of each layer plus the initial embedding outputs.

attentions (tuple(torch.FloatTensor), optional, returned when config.output_attentions=True):

Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Return type

tuple(torch.FloatTensor) comprising various elements depending on the configuration (RobertaConfig) and inputs

Examples:

import torch
from transformers import LongformerForMaskedLM, LongformerTokenizer

model = LongformerForMaskedLM.from_pretrained('allenai/longformer-base-4096')
tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')

SAMPLE_TEXT = ' '.join(['Hello world! '] * 1000)  # long input document
input_ids = torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0)  # batch of size 1

attention_mask = None  # default is local attention everywhere, which is a good choice for MaskedLM
                       # check ``LongformerModel.forward`` for more details how to set `attention_mask`
loss, prediction_scores = model(input_ids, attention_mask=attention_mask, masked_lm_labels=input_ids)

LongformerForQuestionAnswering

class transformers.LongformerForQuestionAnswering(config)

Longformer Model with a span classification head on top for extractive question-answering tasks like SQuAD / TriviaQA (a linear layers on top of the hidden-states output to compute span start logits and span end logits).

This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters

config (LongformerConfig) – Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

config_class

alias of transformers.configuration_longformer.LongformerConfig

forward(input_ids, attention_mask=None, global_attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, start_positions=None, end_positions=None)

The LongformerForQuestionAnswering forward method, overrides the __call__() special method.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Parameters

• input_ids (torch.LongTensor of shape (batch_size, sequence_length)) –

  Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using transformers.LonmgformerTokenizer. See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.encode_plus() for details.

  What are input IDs?

• attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for tokens that are NOT MASKED, 0 for MASKED tokens.

  What are attention masks?

• global_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) – Mask to decide the attention given on each token, local attention or global attenion. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the <s> token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the Longformer paper https://arxiv.org/abs/2004.05150 for more details. Mask values selected in [0, 1]: 0 for local attention (a sliding window attention), 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).

• token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token

  What are token type IDs?

• position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].

  What are position IDs?

• inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.

• start_positions (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence are not taken into account for computing the loss.

• end_positions (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence are not taken into account for computing the loss.

Returns

loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided):

Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.

start_scores (torch.FloatTensor of shape (batch_size, sequence_length,)):

Span-start scores (before SoftMax).

end_scores (torch.FloatTensor of shape (batch_size, sequence_length,)):

Span-end scores (before SoftMax).

hidden_states (tuple(torch.FloatTensor), optional, returned when config.output_hidden_states=True):

Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size). Hidden-states of the model at the output of each layer plus the initial embedding outputs.

attentions (tuple(torch.FloatTensor), optional, returned when config.output_attentions=True):

Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Return type

tuple(torch.FloatTensor) comprising various elements depending on the configuration (LongformerConfig) and inputs

Examples:

from transformers import LongformerTokenizer, LongformerForQuestionAnswering
import torch

tokenizer = LongformerTokenizer.from_pretrained("allenai/longformer-large-4096-finetuned-triviaqa")
model = LongformerForQuestionAnswering.from_pretrained("allenai/longformer-large-4096-finetuned-triviaqa")

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
encoding = tokenizer.encode_plus(question, text, return_tensors="pt")
input_ids = encoding["input_ids"]

# default is local attention everywhere
# the forward method will automatically set global attention on question tokens
attention_mask = encoding["attention_mask"]

start_scores, end_scores = model(input_ids, attention_mask=attention_mask)
all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())

answer_tokens = all_tokens[torch.argmax(start_scores) :torch.argmax(end_scores)+1]
answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens)) # remove space prepending space token

LongformerForMultipleChoice

class transformers.LongformerForMultipleChoice(config)

Longformer Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.

This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters

config (LongformerConfig) – Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

config_class

alias of transformers.configuration_longformer.LongformerConfig

forward(input_ids=None, token_type_ids=None, attention_mask=None, global_attention_mask=None, labels=None, position_ids=None, inputs_embeds=None)

The LongformerForMultipleChoice forward method, overrides the __call__() special method.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Parameters

• input_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length)) –

  Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using transformers.LonmgformerTokenizer. See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.encode_plus() for details.

  What are input IDs?

• attention_mask (torch.FloatTensor of shape (batch_size, num_choices, sequence_length), optional, defaults to None) –

  Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for tokens that are NOT MASKED, 0 for MASKED tokens.

  What are attention masks?

• global_attention_mask (torch.FloatTensor of shape (batch_size, num_choices, sequence_length), optional, defaults to None) – Mask to decide the attention given on each token, local attention or global attenion. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the <s> token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the Longformer paper https://arxiv.org/abs/2004.05150 for more details. Mask values selected in [0, 1]: 0 for local attention (a sliding window attention), 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).

• token_type_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional, defaults to None) –

  Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token

  What are token type IDs?

• position_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional, defaults to None) –

  Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].

  What are position IDs?

• inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.

• labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices] where num_choices is the size of the second dimension of the input tensors. (see input_ids above)

Returns

loss (torch.FloatTensor` of shape ``(1,)`, optional, returned when labels is provided):

Classification loss.

classification_scores (torch.FloatTensor of shape (batch_size, num_choices)):

num_choices is the second dimension of the input tensors. (see input_ids above).

Classification scores (before SoftMax).

hidden_states (tuple(torch.FloatTensor), optional, returned when config.output_hidden_states=True):

Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

Hidden-states of the model at the output of each layer plus the initial embedding outputs.

attentions (tuple(torch.FloatTensor), optional, returned when config.output_attentions=True):

Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Return type

tuple(torch.FloatTensor) comprising various elements depending on the configuration (RobertaConfig) and inputs

Examples:

from transformers import LongformerTokenizer, LongformerForMultipleChoice
import torch

tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')
model = LongformerForMultipleChoice.from_pretrained('allenai/longformer-base-4096')
# context = "The dog is cute" | choice = "the dog" / "the cat"
choices = [("The dog is cute", "the dog"), ("The dog is cute", "the cat")]
input_ids = torch.tensor([tokenizer.encode(s[0], s[1], add_special_tokens=True) for s in choices]).unsqueeze(0)  # Batch size 1, 2 choices
labels = torch.tensor(1).unsqueeze(0)  # Batch size 1

# global attention is automatically put on "the dog" and "the cat"
outputs = model(input_ids, labels=labels)
loss, classification_scores = outputs[:2]

LongformerForTokenClassification

class transformers.LongformerForTokenClassification(config)

Longformer Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks.

This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters

config (LongformerConfig) – Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

config_class

alias of transformers.configuration_longformer.LongformerConfig

forward(input_ids=None, attention_mask=None, global_attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, labels=None)

The LongformerForTokenClassification forward method, overrides the __call__() special method.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Parameters

• input_ids (torch.LongTensor of shape (batch_size, sequence_length)) –

  Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using transformers.LonmgformerTokenizer. See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.encode_plus() for details.

  What are input IDs?

• attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for tokens that are NOT MASKED, 0 for MASKED tokens.

  What are attention masks?

• global_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) – Mask to decide the attention given on each token, local attention or global attenion. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the <s> token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the Longformer paper https://arxiv.org/abs/2004.05150 for more details. Mask values selected in [0, 1]: 0 for local attention (a sliding window attention), 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).

• token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token

  What are token type IDs?

• position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –

  Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].

  What are position IDs?

• inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.

• labels (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) – Labels for computing the token classification loss. Indices should be in [0, ..., config.num_labels - 1].

Returns

loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) :

Classification loss.

scores (torch.FloatTensor of shape (batch_size, sequence_length, config.num_labels))

Classification scores (before SoftMax).

hidden_states (tuple(torch.FloatTensor), optional, returned when config.output_hidden_states=True):

Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

Hidden-states of the model at the output of each layer plus the initial embedding outputs.

attentions (tuple(torch.FloatTensor), optional, returned when config.output_attentions=True):

Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Return type

tuple(torch.FloatTensor) comprising various elements depending on the configuration (LongformerConfig) and inputs

Examples:

from transformers import LongformerTokenizer, LongformerForTokenClassification
import torch

tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')
model = LongformerForTokenClassification.from_pretrained('allenai/longformer-base-4096')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)
loss, scores = outputs[:2]