# Fast-Inference with Ctranslate2

Speedup inference by 2x-8x using int8 inference in C++

quantized version of Helsinki-NLP/opus-mt-en-de

pip install hf-hub-ctranslate2>=1.0.0 ctranslate2>=3.13.0

Converted using

ct2-transformers-converter --model Helsinki-NLP/opus-mt-en-de --output_dir /home/michael/tmp-ct2fast-opus-mt-en-de --force --copy_files README.md generation_config.json tokenizer_config.json vocab.json source.spm .gitattributes target.spm --quantization float16

Checkpoint compatible to ctranslate2 and hf-hub-ctranslate2

compute_type=int8_float16 for device="cuda"
compute_type=int8 for device="cpu"

from hf_hub_ctranslate2 import TranslatorCT2fromHfHub, GeneratorCT2fromHfHub
from transformers import AutoTokenizer

model_name = "michaelfeil/ct2fast-opus-mt-en-de"
# use either TranslatorCT2fromHfHub or GeneratorCT2fromHfHub here, depending on model.
model = TranslatorCT2fromHfHub(
        # load in int8 on CUDA
        model_name_or_path=model_name, 
        device="cuda",
        compute_type="int8_float16",
        tokenizer=AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de")
)
outputs = model.generate(
    text=["How do you call a fast Flan-ingo?", "User: How are you doing?"],
)
print(outputs)

Licence and other remarks:

This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.

Original description

Model Details

Model Description:

Developed by: Language Technology Research Group at the University of Helsinki
Model Type: Translation
Language(s):
- Source Language: English
- Target Language: German
License: CC-BY-4.0
Resources for more information:
- GitHub Repo

Uses

Direct Use

This model can be used for translation and text-to-text generation.

Risks, Limitations and Biases

CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)).

Further details about the dataset for this model can be found in the OPUS readme: en-de

Training Data

Preprocessing

pre-processing: normalization + SentencePiece
dataset: opus
download original weights: opus-2020-02-26.zip
test set translations: opus-2020-02-26.test.txt

Evaluation

Results

test set scores: opus-2020-02-26.eval.txt

Benchmarks

testset	BLEU	chr-F
newssyscomb2009.en.de	23.5	0.540
news-test2008.en.de	23.5	0.529
newstest2009.en.de	22.3	0.530
newstest2010.en.de	24.9	0.544
newstest2011.en.de	22.5	0.524
newstest2012.en.de	23.0	0.525
newstest2013.en.de	26.9	0.553
newstest2015-ende.en.de	31.1	0.594
newstest2016-ende.en.de	37.0	0.636
newstest2017-ende.en.de	29.9	0.586
newstest2018-ende.en.de	45.2	0.690
newstest2019-ende.en.de	40.9	0.654
Tatoeba.en.de	47.3	0.664

Citation Information

@InProceedings{TiedemannThottingal:EAMT2020,
  author = {J{\"o}rg Tiedemann and Santhosh Thottingal},
  title = {{OPUS-MT} — {B}uilding open translation services for the {W}orld},
  booktitle = {Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT)},
  year = {2020},
  address = {Lisbon, Portugal}
 }

How to Get Started With the Model

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de")

model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-en-de")

michaelfeil
/

ct2fast-opus-mt-en-de