stefan-it commited on
Commit
476b649
1 Parent(s): 05934a6

readme: add benchmark results

Browse files
Files changed (1) hide show
  1. README.md +35 -0
README.md CHANGED
@@ -25,6 +25,41 @@ Version 1 of the Journaux-LM was pretrained on the following publicly available
25
 
26
  In total, the pretraining corpus has a size of 408GB.
27
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
28
  # Changelog
29
 
30
  * 02.11.2024: Initial version of the model. More details are coming very soon!
 
25
 
26
  In total, the pretraining corpus has a size of 408GB.
27
 
28
+ ## Benchmarks (Named Entity Recognition)
29
+
30
+ We compare our Zeitungs-LM directly to the French Europeana BERT model (as Zeitungs-LM is supposed to be the successor of it) on various downstream tasks from the [hmBench](https://github.com/stefan-it/hmBench) repository, which is focussed on Named Entity Recognition.
31
+
32
+ We report averaged micro F1-Score over 5 runs with different seeds and use the best hyper-parameter configuration on the development set of each dataset to report the final test score.
33
+
34
+ ### Development Set
35
+
36
+ The results on the development set can be seen in the following table:
37
+
38
+ | Model \ Dataset | [AjMC][1] | [ICDAR][2] | [LeTemps][3] | [NewsEye][4] | [HIPE-2020][5] | Avg. |
39
+ |:--------------------|:----------|:-----------|:-------------|:-------------|:---------------|:----------|
40
+ | [Europeana BERT][6] | 85.7 | 77.63 | 67.14 | 82.68 | 85.98 | 79.83 |
41
+ | Journaux-LM v1 | 86.25 | 78.51 | 67.76 | 84.07 | 88.17 | 80.95 |
42
+
43
+ Our Journaux-LM leads to a performance boost of 1.12% compared to the German Europeana BERT model.
44
+
45
+ ### Test Set
46
+
47
+ The final results on the test set can be seen here:
48
+
49
+ | Model \ Dataset | [AjMC][1] | [ICDAR][2] | [LeTemps][3] | [NewsEye][4] | [HIPE-2020][5] | Avg. |
50
+ |:--------------------|:----------|:-----------|:-------------|:-------------|:---------------|:----------|
51
+ | [Europeana BERT][6] | 81.06 | 78.17 | 67.22 | 73.51 | 81.00 | 76.19 |
52
+ | Journaux-LM v1 | 83.41 | 77.73 | 67.11 | 74.48 | 83.14 | 77.17 |
53
+
54
+ Our Journaux-LM beats the French Europeana BERT model by 0.98%.
55
+
56
+ [1]: https://github.com/hipe-eval/HIPE-2022-data/blob/main/documentation/README-ajmc.md
57
+ [2]: https://github.com/stefan-it/historic-domain-adaptation-icdar
58
+ [3]: https://github.com/hipe-eval/HIPE-2022-data/blob/main/documentation/README-letemps.md
59
+ [4]: https://github.com/hipe-eval/HIPE-2022-data/blob/main/documentation/README-newseye.md
60
+ [5]: https://github.com/hipe-eval/HIPE-2022-data/blob/main/documentation/README-hipe2020.md
61
+ [6]: https://huggingface.co/dbmdz/bert-base-french-europeana-cased
62
+
63
  # Changelog
64
 
65
  * 02.11.2024: Initial version of the model. More details are coming very soon!