|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
pipeline_tag: text2text-generation |
|
--- |
|
# MANTa-LM (base) |
|
|
|
Pretrained MANTa-LM architecture as introduced in the paper [MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling](https://aclanthology.org/2022.findings-emnlp.207.pdf). |
|
<center><img src="https://github.com/NathanGodey/nathangodey.github.io/raw/main/img/posts/full_difftok_schema.png" width="600"></center> |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
The MANTa tokenizer aims at mimicking the combination of a subword tokenizer and an embedding matrix in a classical language model in a differentiable way. |
|
This trainable tokenizer is thus added as the first layer of an encoder-decoder model and trained using the language modeling objective. |
|
|
|
Our results show that MANTa-LM only slightly degrades the performance of a T5 equivalent on the GLUE benchmark while being **much more robust** to artificial and user-generated noise. |
|
|
|
|
|
### Model Sources |
|
|
|
- **Paper:** [MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling](https://aclanthology.org/2022.findings-emnlp.207.pdf) (EMNLP 2022 Findings) |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("google/byt5-base") |
|
manta_model = AutoModelForSeq2SeqLM.from_pretrained("almanach/manta-lm-base", trust_remote_code=True) |
|
|
|
tokens = tokenizer("The name of the capital of France is <extra_id_0> and it is a very big city.", return_tensors="pt") |
|
output = manta_model.generate(**tokens, decoder_start_token_id=0) |
|
|
|
print(tokenizer.batch_decode(output)) |
|
``` |
|
|
|
### Recommendations |
|
|
|
We recommend using a smaller learning rate for the tokenizer module during fine-tuning (byte embeddings, frontier predictor, pooler). |
|
|
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
This model was trained on the C4 dataset. |
|
|
|
### Training Procedure |
|
|
|
The training objective is the same as ByT5, but most hyperparameters are taken from T5. |
|
|
|
|
|
## Citation |
|
|
|
**BibTeX:** |
|
|
|
``` |
|
@inproceedings{godey-etal-2022-manta, |
|
title = "{MANT}a: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling", |
|
author = "Godey, Nathan and |
|
Castagn{\'e}, Roman and |
|
de la Clergerie, {\'E}ric and |
|
Sagot, Beno{\^\i}t", |
|
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022", |
|
month = dec, |
|
year = "2022", |
|
address = "Abu Dhabi, United Arab Emirates", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://aclanthology.org/2022.findings-emnlp.207", |
|
pages = "2859--2870", |
|
} |
|
|
|
``` |
|
|
|
## Model Card Authors |
|
|
|
[Nathan Godey](https://nathangodey.github.io/) |
|
[Roman Castagné](https://romancast.github.io/) |
|
|
|
|
