jordimas commited on
Commit
eaa8760
1 Parent(s): 66caf50

Add description

Browse files
Files changed (3) hide show
  1. README.md +58 -3
  2. config.json +2 -0
  3. model_description.txt +9 -0
README.md CHANGED
@@ -1,3 +1,58 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - ca
4
+ - fr
5
+
6
+ tags:
7
+ - translation
8
+
9
+ library_name: opennmt
10
+ license: mit
11
+ metrics:
12
+ - bleu
13
+
14
+ inference: false
15
+ ---
16
+
17
+ ### Introduction
18
+
19
+ Catalan - French translation model for OpenNMT. These are the same models that we have in production at https://www.softcatala.org/traductor/.
20
+ The models are quantified for low latency.
21
+
22
+ ### Usage
23
+
24
+ Install the necessary dependencies:
25
+
26
+
27
+ ```bash
28
+ pip3 install ctranslate2 pyonmttok
29
+ ```
30
+
31
+
32
+ Simple tokenization & translation using Python:
33
+
34
+
35
+ ```python
36
+ import ctranslate2
37
+ import pyonmttok
38
+ from huggingface_hub import snapshot_download
39
+ model_dir = snapshot_download(repo_id="softcatala/translate-cat-fra", revision="main")
40
+
41
+ tokenizer=pyonmttok.Tokenizer(mode="none", sp_model_path = model_dir + "/sp_m.model")
42
+ tokenized=tokenizer.tokenize("Hola amics")
43
+
44
+ translator = ctranslate2.Translator(model_dir)
45
+ translated = translator.translate_batch([tokenized[0]])
46
+ print(tokenizer.detokenize(translated[0][0]['tokens']))
47
+ ```
48
+
49
+ ## Benchmarks
50
+
51
+ | testset | BLEU |
52
+ |---------------------------------------|-------|
53
+ | test dataset (from train/dev/test) | 41.4 |
54
+ | Flores200 dataset | 35.4 |
55
+
56
+ ## Additional information
57
+ * https://github.com/Softcatala/nmt-models
58
+ * https://github.com/Softcatala/parallel-catalan-corpus
config.json ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ {
2
+ }
model_description.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ Model description: cat-fra
2
+ Date: 2022-11-14
3
+ TF version 2.10.0, OpenNMT version 2.29.1, CTranslate2 version 2.24.0
4
+ Test data set
5
+ BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0 = 41.4 67.8/47.5/35.9/27.7 (BP = 0.978 ratio = 0.979 hyp_len = 155890 ref_len = 159288)
6
+ chrF2|nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0 = 65.1
7
+ Flores data set
8
+ BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0 = 35.4 65.0/42.4/29.8/21.5 (BP = 0.970 ratio = 0.971 hyp_len = 27511 ref_len = 28343)
9
+ chrF2|nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0 = 61.3