readme: add benchmark results
Browse files
README.md
CHANGED
@@ -25,6 +25,41 @@ Version 1 of the Journaux-LM was pretrained on the following publicly available
|
|
25 |
|
26 |
In total, the pretraining corpus has a size of 408GB.
|
27 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
28 |
# Changelog
|
29 |
|
30 |
* 02.11.2024: Initial version of the model. More details are coming very soon!
|
|
|
25 |
|
26 |
In total, the pretraining corpus has a size of 408GB.
|
27 |
|
28 |
+
## Benchmarks (Named Entity Recognition)
|
29 |
+
|
30 |
+
We compare our Zeitungs-LM directly to the French Europeana BERT model (as Zeitungs-LM is supposed to be the successor of it) on various downstream tasks from the [hmBench](https://github.com/stefan-it/hmBench) repository, which is focussed on Named Entity Recognition.
|
31 |
+
|
32 |
+
We report averaged micro F1-Score over 5 runs with different seeds and use the best hyper-parameter configuration on the development set of each dataset to report the final test score.
|
33 |
+
|
34 |
+
### Development Set
|
35 |
+
|
36 |
+
The results on the development set can be seen in the following table:
|
37 |
+
|
38 |
+
| Model \ Dataset | [AjMC][1] | [ICDAR][2] | [LeTemps][3] | [NewsEye][4] | [HIPE-2020][5] | Avg. |
|
39 |
+
|:--------------------|:----------|:-----------|:-------------|:-------------|:---------------|:----------|
|
40 |
+
| [Europeana BERT][6] | 85.7 | 77.63 | 67.14 | 82.68 | 85.98 | 79.83 |
|
41 |
+
| Journaux-LM v1 | 86.25 | 78.51 | 67.76 | 84.07 | 88.17 | 80.95 |
|
42 |
+
|
43 |
+
Our Journaux-LM leads to a performance boost of 1.12% compared to the German Europeana BERT model.
|
44 |
+
|
45 |
+
### Test Set
|
46 |
+
|
47 |
+
The final results on the test set can be seen here:
|
48 |
+
|
49 |
+
| Model \ Dataset | [AjMC][1] | [ICDAR][2] | [LeTemps][3] | [NewsEye][4] | [HIPE-2020][5] | Avg. |
|
50 |
+
|:--------------------|:----------|:-----------|:-------------|:-------------|:---------------|:----------|
|
51 |
+
| [Europeana BERT][6] | 81.06 | 78.17 | 67.22 | 73.51 | 81.00 | 76.19 |
|
52 |
+
| Journaux-LM v1 | 83.41 | 77.73 | 67.11 | 74.48 | 83.14 | 77.17 |
|
53 |
+
|
54 |
+
Our Journaux-LM beats the French Europeana BERT model by 0.98%.
|
55 |
+
|
56 |
+
[1]: https://github.com/hipe-eval/HIPE-2022-data/blob/main/documentation/README-ajmc.md
|
57 |
+
[2]: https://github.com/stefan-it/historic-domain-adaptation-icdar
|
58 |
+
[3]: https://github.com/hipe-eval/HIPE-2022-data/blob/main/documentation/README-letemps.md
|
59 |
+
[4]: https://github.com/hipe-eval/HIPE-2022-data/blob/main/documentation/README-newseye.md
|
60 |
+
[5]: https://github.com/hipe-eval/HIPE-2022-data/blob/main/documentation/README-hipe2020.md
|
61 |
+
[6]: https://huggingface.co/dbmdz/bert-base-french-europeana-cased
|
62 |
+
|
63 |
# Changelog
|
64 |
|
65 |
* 02.11.2024: Initial version of the model. More details are coming very soon!
|