cartesinus
commited on
Commit
•
d3e042e
1
Parent(s):
aa189ad
Update README.md
Browse files
README.md
CHANGED
@@ -43,9 +43,37 @@ It achieves the following results on the evaluation set:
|
|
43 |
|
44 |
More information needed
|
45 |
|
46 |
-
##
|
47 |
-
|
48 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
49 |
|
50 |
## Training and evaluation data
|
51 |
|
|
|
43 |
|
44 |
More information needed
|
45 |
|
46 |
+
## How to use
|
47 |
+
|
48 |
+
First please make sure to install `pip install transformers`. First download model:
|
49 |
+
|
50 |
+
```python
|
51 |
+
from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
|
52 |
+
import torch
|
53 |
+
|
54 |
+
def translate(input_text, lang):
|
55 |
+
input_ids = tokenizer(input_text, return_tensors="pt")
|
56 |
+
generated_tokens = model.generate(**input_ids, forced_bos_token_id=tokenizer.get_lang_id(lang))
|
57 |
+
return tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
|
58 |
+
|
59 |
+
model_name = "cartesinus/iva_mt_wslot-m2m100_418M-0.1.0-en-sv"
|
60 |
+
tokenizer = M2M100Tokenizer.from_pretrained(model_name, src_lang="en", tgt_lang="sv")
|
61 |
+
model = M2M100ForConditionalGeneration.from_pretrained(model_name)
|
62 |
+
```
|
63 |
+
|
64 |
+
Then you can translate either plain text like this:
|
65 |
+
```python
|
66 |
+
print(translate("set the temperature on my thermostat", "sv"))
|
67 |
+
```
|
68 |
+
or you can translate with slot annotations that will be restored in tgt language:
|
69 |
+
```python
|
70 |
+
print(translate("wake me up at <a>nine am<a> on <b>friday<b>", "sv"))
|
71 |
+
```
|
72 |
+
Limitations of translation with slot transfer:
|
73 |
+
1) Annotated words must be placed between semi-xml tags like this "this is \<a\>example\<a\>"
|
74 |
+
2) There is no closing tag for example "\<\a\>" in above example - this is done on purpose to omit problems with backslash escape
|
75 |
+
3) If the sentence consists of more than one slot then simply use the next alphabet letter. For example "this is \<a\>example\<a\> with more than \<b\>one\<b\> slot"
|
76 |
+
4) Please do not add space before the first or last annotated word because this particular model was trained this way and it most probably will lower its results
|
77 |
|
78 |
## Training and evaluation data
|
79 |
|