File size: 2,675 Bytes
5a68661 c1bb504 465ab0c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 |
---
datasets:
- raalst/squad_v2_dutch
language:
- nl
---
The used dataset raalst/squad_v2_dutch was kindly provided by Henryk
it contains train and validation.
I declared 20% of Train to function as Test
when using raalst/squad_v2_dutch, be sure to clean up quotes and double quotes in the contexts
def cleanup(mylist):
for item in mylist:
if '"' in item["context"]:
item["context"] = item["context"].replace('"','\\"')
if "'" in item["context"]:
item["context"] = item["context"].replace("'","\\'")
The pretrained model was pdelobelle/robbert-v2-dutch-base, a dutch RoBERTa model
results obtained in training are :
{'exact': 61.75389109958193,
'f1': 66.89717170237417,
'total': 19853,
'HasAns_exact': 48.967182330322814,
'HasAns_f1': 58.09796564493008,
'HasAns_total': 11183,
'NoAns_exact': 78.24682814302192,
'NoAns_f1': 78.24682814302192,
'NoAns_total': 8670,
'best_exact': 61.75389109958193,
'best_exact_thresh': 0.0,
'best_f1': 66.89717170237276,
'best_f1_thresh': 0.0}
settings (until I figured out how to report them properly):
DatasetDict({
train: Dataset({
features: ['id', 'title', 'context', 'question', 'answers'],
num_rows: 79412
})
test: Dataset({
features: ['id', 'title', 'context', 'question', 'answers'],
num_rows: 19853
})
validation: Dataset({
features: ['id', 'title', 'context', 'question', 'answers'],
num_rows: 9669
})
})
tokenizer = AutoTokenizer.from_pretrained("pdelobelle/robbert-v2-dutch-base")
from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer
model = AutoModelForQuestionAnswering.from_pretrained("pdelobelle/robbert-v2-dutch-base")
training_args = TrainingArguments(
output_dir="./qa_model",
evaluation_strategy="epoch",
learning_rate=2e-5,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
num_train_epochs=3,
weight_decay=0.01,
push_to_hub=False,
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_squad["train"],
eval_dataset=tokenized_squad["validation"],
tokenizer=tokenizer,
data_collator=data_collator,
)
trainer.train()
[15198/15198 2:57:03, Epoch 3/3]
Epoch Training Loss Validation Loss
1 1.380700 1.177431
2 1.093000 1.052601
3 0.849700 1.143632
TrainOutput(global_step=15198, training_loss=1.1917077029499668, metrics={'train_runtime': 10623.9565,
'train_samples_per_second': 22.886, 'train_steps_per_second': 1.431, 'total_flos': 4.764955396486349e+16,
'train_loss': 1.1917077029499668, 'epoch': 3.0})
Trained on Ubuntu with 1080Ti
|