language: id
tags:
- roberta
license: mit
datasets:
- indonli
widget:
- text: >-
Amir Sjarifoeddin Harahap lahir di Kota Medan, Sumatera Utara, 27 April
1907. Ia meninggal di Surakarta, Jawa Tengah, pada 19 Desember 1948 dalam
usia 41 tahun. </s></s> Amir Sjarifoeddin Harahap masih hidup.
Indo-roberta-indonli
Indo-roberta-indonli is natural language inference classifier based on Indo-roberta model. It was trained on the trained on IndoNLI dataset. The model used was Indo-roberta and was transfer-learned to a natural inference classifier model. The model are tested using the validation, test_layer and test_expert dataset given in the github repository. The results are shown below.
Result
| Dataset | Accuracy | F1 | Precision | Recall |
|---|---|---|---|---|
| Test Lay | 0.74329 | 0.74075 | 0.74283 | 0.74133 |
| Test Expert | 0.6115 | 0.60543 | 0.63924 | 0.61742 |
Model
The model was trained on with 5 epochs, batch size 16, learning rate 2e-5 and weight decay 0.01. Achieved different metrics as shown below.
| Epoch | Training Loss | Validation Loss | Accuracy | F1 | Precision | Recall |
|---|---|---|---|---|---|---|
| 1 | 0.942500 | 0.658559 | 0.737369 | 0.735552 | 0.735488 | 0.736679 |
| 2 | 0.649200 | 0.645290 | 0.761493 | 0.759593 | 0.762784 | 0.759642 |
| 3 | 0.437100 | 0.667163 | 0.766045 | 0.763979 | 0.765740 | 0.763792 |
| 4 | 0.282000 | 0.786683 | 0.764679 | 0.761802 | 0.762011 | 0.761684 |
| 5 | 0.193500 | 0.925717 | 0.765134 | 0.763127 | 0.763560 | 0.763489 |
How to Use
As NLI Classifier
from transformers import pipeline
pretrained_name = "StevenLimcorn/indonesian-roberta-indonli"
nlp = pipeline(
"zero-shot-classification",
model=pretrained_name,
tokenizer=pretrained_name
)
nlp("Amir Sjarifoeddin Harahap lahir di Kota Medan, Sumatera Utara, 27 April 1907. Ia meninggal di Surakarta, Jawa Tengah, pada 19 Desember 1948 dalam usia 41 tahun. </s></s> Amir Sjarifoeddin Harahap masih hidup.")
Disclaimer
Do consider the biases which come from both the pre-trained RoBERTa model and the INDONLI dataset that may be carried over into the results of this model.
Author
Indonesian RoBERTa Base IndoNLI was trained and evaluated by Steven Limcorn. All computation and development are done on Google Colaboratory using their free GPU access.
Reference
The dataset we used is by IndoNLI.
@inproceedings{indonli,
title = "IndoNLI: A Natural Language Inference Dataset for Indonesian",
author = "Mahendra, Rahmad and Aji, Alham Fikri and Louvan, Samuel and Rahman, Fahrurrozi and Vania, Clara",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2021",
publisher = "Association for Computational Linguistics",
}