Hugging Face's logo

Beomseok-LEE
/
NLU-Speech-MASSIVE-finetune

Safetensors
mt5
Model card Files Community
NLU-Speech-MASSIVE-finetune / sbatch-2857510-02-05.18-47.log
Beomseok-LEE's picture
Beomseok-LEE
Upload folder using huggingface_hub
8048f3a verified
raw
history blame contribute delete
No virus
99.8 kB
Using the `WANDB_DISABLED` environment variable is deprecated and will be removed in v5. Use the --report_to flag to control the integrations used for logging result (for instance --report_to none).
02/05/2024 18:47:23 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: False, 16-bits training: False
02/05/2024 18:47:23 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
_n_gpu=1,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
bf16=False,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=None,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_predict=True,
do_train=False,
eval_accumulation_steps=None,
eval_delay=0,
eval_steps=None,
evaluation_strategy=no,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
generation_config=None,
generation_max_length=None,
generation_num_beams=2,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=5e-05,
length_column_name=input_length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/beegfs/scratch/user/blee/project_3/models/NLU.mt5-base.task_type-1.fine_tune.gpu_a100-40g+.node-1x1.bsz-64.epochs-22.metric-ema.metric_lang-all/checkpoint-30407/eval/cascaded_SLU/runs/Feb05_18-47-22_chasma-02,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=500,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=linear,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=3.0,
optim=adamw_torch,
optim_args=None,
output_dir=/beegfs/scratch/user/blee/project_3/models/NLU.mt5-base.task_type-1.fine_tune.gpu_a100-40g+.node-1x1.bsz-64.epochs-22.metric-ema.metric_lang-all/checkpoint-30407/eval/cascaded_SLU,
overwrite_output_dir=False,
past_index=-1,
per_device_eval_batch_size=32,
per_device_train_batch_size=8,
predict_with_generate=True,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=[],
resume_from_checkpoint=None,
run_name=/beegfs/scratch/user/blee/project_3/models/NLU.mt5-base.task_type-1.fine_tune.gpu_a100-40g+.node-1x1.bsz-64.epochs-22.metric-ema.metric_lang-all/checkpoint-30407/eval/cascaded_SLU,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=steps,
save_total_limit=None,
seed=42,
skip_memory_metrics=True,
sortish_sampler=False,
split_batches=False,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
)
Loading Dataset Infos from /beegfs/scratch/user/blee/hugging-face/models/modules/datasets_modules/datasets/speech_massive_cascaded/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293
02/05/2024 18:47:23 - INFO - datasets.info - Loading Dataset Infos from /beegfs/scratch/user/blee/hugging-face/models/modules/datasets_modules/datasets/speech_massive_cascaded/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293
Overwrite dataset info from restored data version if exists.
02/05/2024 18:47:23 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists.
Loading Dataset info from /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293
02/05/2024 18:47:23 - INFO - datasets.info - Loading Dataset info from /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293
Found cached dataset speech_massive_cascaded (/beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293)
02/05/2024 18:47:23 - INFO - datasets.builder - Found cached dataset speech_massive_cascaded (/beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293)
Loading Dataset info from /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293
02/05/2024 18:47:23 - INFO - datasets.info - Loading Dataset info from /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293
[INFO|configuration_utils.py:737] 2024-02-05 18:47:23,241 >> loading configuration file /beegfs/scratch/user/blee/project_3/models/NLU.mt5-base.task_type-1.fine_tune.gpu_a100-40g+.node-1x1.bsz-64.epochs-22.metric-ema.metric_lang-all/checkpoint-30407/config.json
[INFO|configuration_utils.py:802] 2024-02-05 18:47:23,251 >> Model config MT5Config {
"_name_or_path": "/beegfs/scratch/user/blee/project_3/models/NLU.mt5-base.task_type-1.fine_tune.gpu_a100-40g+.node-1x1.bsz-64.epochs-22.metric-ema.metric_lang-all/checkpoint-30407",
"architectures": [
"MT5ForConditionalGeneration"
],
"classifier_dropout": 0.0,
"d_ff": 2048,
"d_kv": 64,
"d_model": 768,
"decoder_start_token_id": 0,
"dense_act_fn": "gelu_new",
"dropout": 0.2,
"dropout_rate": 0.1,
"eos_token_id": 1,
"feed_forward_proj": "gated-gelu",
"initializer_factor": 1.0,
"is_encoder_decoder": true,
"is_gated_act": true,
"layer_norm_epsilon": 1e-06,
"model_type": "mt5",
"num_decoder_layers": 12,
"num_heads": 12,
"num_layers": 12,
"output_past": true,
"pad_token_id": 0,
"relative_attention_max_distance": 128,
"relative_attention_num_buckets": 32,
"tie_word_embeddings": false,
"tokenizer_class": "T5Tokenizer",
"torch_dtype": "float32",
"transformers_version": "4.37.0.dev0",
"use_cache": true,
"vocab_size": 250112
}
[INFO|tokenization_utils_base.py:2024] 2024-02-05 18:47:23,254 >> loading file spiece.model
[INFO|tokenization_utils_base.py:2024] 2024-02-05 18:47:23,254 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2024] 2024-02-05 18:47:23,254 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2024] 2024-02-05 18:47:23,255 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2024] 2024-02-05 18:47:23,255 >> loading file tokenizer_config.json
[INFO|modeling_utils.py:3373] 2024-02-05 18:47:23,703 >> loading weights file /beegfs/scratch/user/blee/project_3/models/NLU.mt5-base.task_type-1.fine_tune.gpu_a100-40g+.node-1x1.bsz-64.epochs-22.metric-ema.metric_lang-all/checkpoint-30407/model.safetensors
[INFO|configuration_utils.py:826] 2024-02-05 18:47:23,890 >> Generate config GenerationConfig {
"decoder_start_token_id": 0,
"eos_token_id": 1,
"pad_token_id": 0
}
[INFO|modeling_utils.py:4224] 2024-02-05 18:47:28,850 >> All model checkpoint weights were used when initializing MT5ForConditionalGeneration.
[INFO|modeling_utils.py:4232] 2024-02-05 18:47:28,857 >> All the weights of MT5ForConditionalGeneration were initialized from the model checkpoint at /beegfs/scratch/user/blee/project_3/models/NLU.mt5-base.task_type-1.fine_tune.gpu_a100-40g+.node-1x1.bsz-64.epochs-22.metric-ema.metric_lang-all/checkpoint-30407.
If your task is similar to the task the model of the checkpoint was trained on, you can already use MT5ForConditionalGeneration for predictions without further training.
[INFO|configuration_utils.py:779] 2024-02-05 18:47:28,863 >> loading configuration file /beegfs/scratch/user/blee/project_3/models/NLU.mt5-base.task_type-1.fine_tune.gpu_a100-40g+.node-1x1.bsz-64.epochs-22.metric-ema.metric_lang-all/checkpoint-30407/generation_config.json
[INFO|configuration_utils.py:826] 2024-02-05 18:47:28,864 >> Generate config GenerationConfig {
"decoder_start_token_id": 0,
"eos_token_id": 1,
"pad_token_id": 0
}
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-a28263cfb71413f6.arrow
02/05/2024 18:47:29 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-a28263cfb71413f6.arrow
Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 10526.21 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 10282.67 examples/s]
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-0f6b9ba1cc4e5fb1.arrow
02/05/2024 18:47:29 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-0f6b9ba1cc4e5fb1.arrow
Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 21768.02 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 20740.13 examples/s]
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-cd10216b4341f1a2.arrow
02/05/2024 18:47:29 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-cd10216b4341f1a2.arrow
Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 21336.19 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 20262.24 examples/s]
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-cf09cb968cef4c56.arrow
02/05/2024 18:47:29 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-cf09cb968cef4c56.arrow
Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 20894.52 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 19789.16 examples/s]
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-b372c4d6e9ad447f.arrow
02/05/2024 18:47:29 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-b372c4d6e9ad447f.arrow
Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 10819.03 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 10482.08 examples/s]
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-5e85733ab0d7983c.arrow
02/05/2024 18:47:30 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-5e85733ab0d7983c.arrow
Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 19433.65 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 18477.60 examples/s]
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-3498127f38d0e88c.arrow
02/05/2024 18:47:30 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-3498127f38d0e88c.arrow
Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 21220.04 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 20249.02 examples/s]
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-5031b1ae09c119f0.arrow
02/05/2024 18:47:30 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-5031b1ae09c119f0.arrow
Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 22030.41 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 21147.80 examples/s]
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-afad46b8cc76fbde.arrow
02/05/2024 18:47:30 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-afad46b8cc76fbde.arrow
Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 21270.91 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 20297.85 examples/s]
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-44bbdcb1f95b0505.arrow
02/05/2024 18:47:31 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-44bbdcb1f95b0505.arrow
Running tokenizer on prediction dataset: 34%|███▎ | 1000/2974 [00:00<00:00, 5169.56 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 10278.78 examples/s]
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-649defd19aa00c44.arrow
02/05/2024 18:47:31 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-649defd19aa00c44.arrow
Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 22109.21 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 21308.60 examples/s]
Running tokenizer on prediction dataset: 0%| | 0/2974 [00:00<?, ? examples/s]Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-bc58b34f0f2e6d55.arrow
02/05/2024 18:47:31 - INFO - datasets.arrow_dataset - Caching processed dataset at /beegfs/scratch/user/blee/hugging-face/models/datasets/speech_massive_cascaded/multilingual-test/1.0.0/f36c9e4210ec02a91ee05c9fa785d90aec211ba2025363c65b643c68e109b293/cache-bc58b34f0f2e6d55.arrow
Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 19544.58 examples/s] Running tokenizer on prediction dataset: 100%|██████████| 2974/2974 [00:00<00:00, 18677.47 examples/s]
02/05/2024 18:47:52 - WARNING - accelerate.utils.other - Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
02/05/2024 18:47:53 - INFO - __main__ - *** Predict ***
02/05/2024 18:47:53 - INFO - __main__ - *** test_ar_SA ***
[INFO|trainer.py:718] 2024-02-05 18:47:53,119 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:47:53,129 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:47:53,129 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:47:53,130 >> Batch size = 32
[WARNING|logging.py:314] 2024-02-05 18:47:53,134 >> You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:18, 5.04it/s] 3%|▎ | 3/93 [00:00<00:22, 3.94it/s] 4%|▍ | 4/93 [00:01<00:29, 3.00it/s] 5%|▌ | 5/93 [00:01<00:34, 2.51it/s] 6%|▋ | 6/93 [00:02<00:33, 2.61it/s] 8%|▊ | 7/93 [00:02<00:35, 2.44it/s] 9%|▊ | 8/93 [00:02<00:33, 2.56it/s] 10%|▉ | 9/93 [00:03<00:33, 2.52it/s] 11%|█ | 10/93 [00:03<00:32, 2.56it/s] 12%|█▏ | 11/93 [00:04<00:32, 2.52it/s] 13%|█▎ | 12/93 [00:04<00:31, 2.60it/s] 14%|█▍ | 13/93 [00:04<00:30, 2.65it/s] 15%|█▌ | 14/93 [00:05<00:30, 2.56it/s] 16%|█▌ | 15/93 [00:05<00:31, 2.45it/s] 17%|█▋ | 16/93 [00:06<00:34, 2.24it/s] 18%|█▊ | 17/93 [00:06<00:34, 2.21it/s] 19%|█▉ | 18/93 [00:07<00:32, 2.29it/s] 20%|██ | 19/93 [00:07<00:30, 2.42it/s] 22%|██▏ | 20/93 [00:07<00:29, 2.48it/s] 23%|██▎ | 21/93 [00:08<00:28, 2.53it/s] 24%|██▎ | 22/93 [00:08<00:31, 2.23it/s] 25%|██▍ | 23/93 [00:09<00:30, 2.28it/s] 26%|██▌ | 24/93 [00:09<00:30, 2.23it/s] 27%|██▋ | 25/93 [00:10<00:29, 2.30it/s] 28%|██▊ | 26/93 [00:10<00:29, 2.24it/s] 29%|██▉ | 27/93 [00:10<00:29, 2.24it/s] 30%|███ | 28/93 [00:13<01:00, 1.08it/s] 31%|███ | 29/93 [00:13<00:49, 1.29it/s] 32%|███▏ | 30/93 [00:13<00:41, 1.52it/s] 33%|███▎ | 31/93 [00:14<00:37, 1.67it/s] 34%|███▍ | 32/93 [00:14<00:33, 1.81it/s] 35%|███▌ | 33/93 [00:15<00:30, 1.95it/s] 37%|███▋ | 34/93 [00:15<00:27, 2.13it/s] 38%|███▊ | 35/93 [00:15<00:24, 2.35it/s] 39%|███▊ | 36/93 [00:16<00:23, 2.39it/s] 40%|███▉ | 37/93 [00:16<00:24, 2.33it/s] 41%|████ | 38/93 [00:17<00:25, 2.18it/s] 42%|████▏ | 39/93 [00:17<00:24, 2.20it/s] 43%|████▎ | 40/93 [00:18<00:22, 2.31it/s] 44%|████▍ | 41/93 [00:18<00:22, 2.33it/s] 45%|████▌ | 42/93 [00:18<00:21, 2.40it/s] 46%|████▌ | 43/93 [00:19<00:19, 2.54it/s] 47%|████▋ | 44/93 [00:19<00:18, 2.68it/s] 48%|████▊ | 45/93 [00:19<00:18, 2.59it/s] 49%|████▉ | 46/93 [00:20<00:19, 2.47it/s] 51%|█████ | 47/93 [00:20<00:19, 2.34it/s] 52%|█████▏ | 48/93 [00:21<00:20, 2.22it/s] 53%|█████▎ | 49/93 [00:21<00:18, 2.34it/s] 54%|█████▍ | 50/93 [00:22<00:18, 2.38it/s] 55%|█████▍ | 51/93 [00:22<00:20, 2.06it/s] 56%|█████▌ | 52/93 [00:23<00:18, 2.16it/s] 57%|█████▋ | 53/93 [00:23<00:17, 2.29it/s] 58%|█████▊ | 54/93 [00:24<00:17, 2.27it/s] 59%|█████▉ | 55/93 [00:24<00:16, 2.25it/s] 60%|██████ | 56/93 [00:24<00:15, 2.34it/s] 61%|██████▏ | 57/93 [00:25<00:15, 2.40it/s] 62%|██████▏ | 58/93 [00:25<00:14, 2.36it/s] 63%|██████▎ | 59/93 [00:26<00:13, 2.53it/s] 65%|██████▍ | 60/93 [00:26<00:12, 2.68it/s] 66%|██████▌ | 61/93 [00:26<00:13, 2.45it/s] 67%|██████▋ | 62/93 [00:27<00:12, 2.52it/s] 68%|██████▊ | 63/93 [00:27<00:11, 2.69it/s] 69%|██████▉ | 64/93 [00:27<00:10, 2.72it/s] 70%|██████▉ | 65/93 [00:28<00:10, 2.68it/s] 71%|███████ | 66/93 [00:28<00:10, 2.62it/s] 72%|███████▏ | 67/93 [00:29<00:09, 2.64it/s] 73%|███████▎ | 68/93 [00:29<00:09, 2.64it/s] 74%|███████▍ | 69/93 [00:29<00:09, 2.47it/s] 75%|███████▌ | 70/93 [00:30<00:09, 2.40it/s] 76%|███████▋ | 71/93 [00:30<00:09, 2.26it/s] 77%|███████▋ | 72/93 [00:31<00:09, 2.24it/s] 78%|███████▊ | 73/93 [00:31<00:08, 2.35it/s] 80%|███████▉ | 74/93 [00:32<00:08, 2.35it/s] 81%|████████ | 75/93 [00:32<00:07, 2.39it/s] 82%|████████▏ | 76/93 [00:32<00:06, 2.49it/s] 83%|████████▎ | 77/93 [00:33<00:06, 2.61it/s] 84%|████████▍ | 78/93 [00:33<00:05, 2.64it/s] 85%|████████▍ | 79/93 [00:34<00:05, 2.49it/s] 86%|████████▌ | 80/93 [00:34<00:05, 2.42it/s] 87%|████████▋ | 81/93 [00:34<00:05, 2.38it/s] 88%|████████▊ | 82/93 [00:35<00:04, 2.53it/s] 89%|████████▉ | 83/93 [00:35<00:03, 2.58it/s] 90%|█████████ | 84/93 [00:35<00:03, 2.60it/s] 91%|█████████▏| 85/93 [00:36<00:03, 2.32it/s] 92%|█████████▏| 86/93 [00:37<00:03, 2.19it/s] 94%|█████████▎| 87/93 [00:37<00:02, 2.29it/s] 95%|█████████▍| 88/93 [00:37<00:02, 2.45it/s] 96%|█████████▌| 89/93 [00:38<00:01, 2.42it/s] 97%|█████████▋| 90/93 [00:38<00:01, 2.35it/s] 98%|█████████▊| 91/93 [00:39<00:00, 2.15it/s] 99%|█████████▉| 92/93 [00:39<00:00, 2.12it/s] 100%|██████████| 93/93 [00:40<00:00, 2.29it/s] 100%|██████████| 93/93 [00:40<00:00, 2.31it/s]
***** predict_test_ar_SA metrics *****
predict_ex_match_acc = 0.4455
predict_ex_match_acc_stderr = 0.0091
predict_intent_acc = 0.7061
predict_intent_acc_stderr = 0.0084
predict_loss = 0.5896
predict_runtime = 0:00:41.25
predict_samples = 2974
predict_samples_per_second = 72.09
predict_slot_micro_f1 = 0.5994
predict_slot_micro_f1_stderr = 0.0039
predict_steps_per_second = 2.254
02/05/2024 18:48:34 - INFO - __main__ - *** test_de_DE ***
[INFO|trainer.py:718] 2024-02-05 18:48:34,602 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:48:34,605 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:48:34,605 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:48:34,605 >> Batch size = 32
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:18, 4.97it/s] 3%|▎ | 3/93 [00:00<00:23, 3.84it/s] 4%|▍ | 4/93 [00:01<00:31, 2.84it/s] 5%|▌ | 5/93 [00:01<00:31, 2.80it/s] 6%|▋ | 6/93 [00:01<00:30, 2.88it/s] 8%|▊ | 7/93 [00:02<00:32, 2.66it/s] 9%|▊ | 8/93 [00:02<00:31, 2.69it/s] 10%|▉ | 9/93 [00:03<00:32, 2.56it/s] 11%|█ | 10/93 [00:03<00:30, 2.73it/s] 12%|█▏ | 11/93 [00:03<00:30, 2.68it/s] 13%|█▎ | 12/93 [00:04<00:31, 2.55it/s] 14%|█▍ | 13/93 [00:04<00:30, 2.64it/s] 15%|█▌ | 14/93 [00:05<00:29, 2.66it/s] 16%|█▌ | 15/93 [00:05<00:33, 2.35it/s] 17%|█▋ | 16/93 [00:06<00:34, 2.21it/s] 18%|█▊ | 17/93 [00:06<00:34, 2.18it/s] 19%|█▉ | 18/93 [00:06<00:32, 2.28it/s] 20%|██ | 19/93 [00:07<00:32, 2.26it/s] 22%|██▏ | 20/93 [00:07<00:31, 2.31it/s] 23%|██▎ | 21/93 [00:08<00:30, 2.34it/s] 24%|██▎ | 22/93 [00:08<00:34, 2.08it/s] 25%|██▍ | 23/93 [00:09<00:33, 2.06it/s] 26%|██▌ | 24/93 [00:09<00:31, 2.16it/s] 27%|██▋ | 25/93 [00:10<00:30, 2.23it/s] 28%|██▊ | 26/93 [00:10<00:29, 2.27it/s] 29%|██▉ | 27/93 [00:10<00:27, 2.37it/s] 30%|███ | 28/93 [00:13<00:59, 1.08it/s] 31%|███ | 29/93 [00:13<00:48, 1.33it/s] 32%|███▏ | 30/93 [00:13<00:42, 1.49it/s] 33%|███▎ | 31/93 [00:14<00:35, 1.73it/s] 34%|███▍ | 32/93 [00:14<00:33, 1.84it/s] 35%|███▌ | 33/93 [00:15<00:32, 1.85it/s] 37%|███▋ | 34/93 [00:15<00:29, 2.03it/s] 38%|███▊ | 35/93 [00:16<00:28, 2.03it/s] 39%|███▊ | 36/93 [00:16<00:26, 2.14it/s] 40%|███▉ | 37/93 [00:16<00:25, 2.20it/s] 41%|████ | 38/93 [00:17<00:24, 2.27it/s] 42%|████▏ | 39/93 [00:17<00:24, 2.17it/s] 43%|████▎ | 40/93 [00:18<00:23, 2.28it/s] 44%|████▍ | 41/93 [00:18<00:21, 2.38it/s] 45%|████▌ | 42/93 [00:19<00:21, 2.37it/s] 46%|████▌ | 43/93 [00:19<00:20, 2.40it/s] 47%|████▋ | 44/93 [00:19<00:19, 2.48it/s] 48%|████▊ | 45/93 [00:20<00:19, 2.45it/s] 49%|████▉ | 46/93 [00:20<00:18, 2.54it/s] 51%|█████ | 47/93 [00:20<00:17, 2.63it/s] 52%|█████▏ | 48/93 [00:21<00:17, 2.55it/s] 53%|█████▎ | 49/93 [00:21<00:17, 2.53it/s] 54%|█████▍ | 50/93 [00:22<00:18, 2.29it/s] 55%|█████▍ | 51/93 [00:22<00:18, 2.26it/s] 56%|█████▌ | 52/93 [00:23<00:20, 2.00it/s] 57%|█████▋ | 53/93 [00:23<00:18, 2.12it/s] 58%|█████▊ | 54/93 [00:24<00:17, 2.23it/s] 59%|█████▉ | 55/93 [00:24<00:16, 2.35it/s] 60%|██████ | 56/93 [00:24<00:14, 2.48it/s] 61%|██████▏ | 57/93 [00:25<00:15, 2.40it/s] 62%|██████▏ | 58/93 [00:25<00:15, 2.33it/s] 63%|██████▎ | 59/93 [00:26<00:14, 2.38it/s] 65%|██████▍ | 60/93 [00:26<00:13, 2.43it/s] 66%|██████▌ | 61/93 [00:27<00:14, 2.25it/s] 67%|██████▋ | 62/93 [00:27<00:13, 2.24it/s] 68%|██████▊ | 63/93 [00:28<00:13, 2.30it/s] 69%|██████▉ | 64/93 [00:28<00:13, 2.17it/s] 70%|██████▉ | 65/93 [00:28<00:11, 2.37it/s] 71%|███████ | 66/93 [00:29<00:11, 2.42it/s] 72%|███████▏ | 67/93 [00:29<00:10, 2.46it/s] 73%|███████▎ | 68/93 [00:30<00:10, 2.42it/s] 74%|███████▍ | 69/93 [00:30<00:10, 2.25it/s] 75%|███████▌ | 70/93 [00:31<00:10, 2.18it/s] 76%|███████▋ | 71/93 [00:31<00:10, 2.12it/s] 77%|███████▋ | 72/93 [00:32<00:10, 2.02it/s] 78%|███████▊ | 73/93 [00:32<00:09, 2.11it/s] 80%|███████▉ | 74/93 [00:32<00:08, 2.15it/s] 81%|████████ | 75/93 [00:33<00:08, 2.15it/s] 82%|████████▏ | 76/93 [00:33<00:07, 2.13it/s] 83%|████████▎ | 77/93 [00:34<00:07, 2.20it/s] 84%|████████▍ | 78/93 [00:35<00:08, 1.78it/s] 85%|████████▍ | 79/93 [00:35<00:07, 1.90it/s] 86%|████████▌ | 80/93 [00:36<00:06, 1.99it/s] 87%|████████▋ | 81/93 [00:36<00:05, 2.15it/s] 88%|████████▊ | 82/93 [00:36<00:04, 2.21it/s] 89%|████████▉ | 83/93 [00:37<00:04, 2.30it/s] 90%|█████████ | 84/93 [00:37<00:03, 2.34it/s] 91%|█████████▏| 85/93 [00:38<00:03, 2.38it/s] 92%|█████████▏| 86/93 [00:38<00:02, 2.38it/s] 94%|█████████▎| 87/93 [00:38<00:02, 2.38it/s] 95%|█████████▍| 88/93 [00:39<00:02, 2.46it/s] 96%|█████████▌| 89/93 [00:39<00:01, 2.42it/s] 97%|█████████▋| 90/93 [00:40<00:01, 2.48it/s] 98%|█████████▊| 91/93 [00:40<00:00, 2.25it/s] 99%|█████████▉| 92/93 [00:41<00:00, 2.36it/s] 100%|██████████| 93/93 [00:41<00:00, 2.45it/s] 100%|██████████| 93/93 [00:41<00:00, 2.23it/s]
***** predict_test_de_DE metrics *****
predict_ex_match_acc = 0.5662
predict_ex_match_acc_stderr = 0.0091
predict_intent_acc = 0.8366
predict_intent_acc_stderr = 0.0068
predict_loss = 0.4835
predict_runtime = 0:00:42.06
predict_samples = 2974
predict_samples_per_second = 70.707
predict_slot_micro_f1 = 0.6886
predict_slot_micro_f1_stderr = 0.0033
predict_steps_per_second = 2.211
02/05/2024 18:49:16 - INFO - __main__ - *** test_es_ES ***
[INFO|trainer.py:718] 2024-02-05 18:49:16,902 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:49:16,904 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:49:16,904 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:49:16,905 >> Batch size = 32
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:19, 4.65it/s] 3%|▎ | 3/93 [00:00<00:28, 3.18it/s] 4%|▍ | 4/93 [00:01<00:35, 2.51it/s] 5%|▌ | 5/93 [00:01<00:37, 2.33it/s] 6%|▋ | 6/93 [00:02<00:37, 2.35it/s] 8%|▊ | 7/93 [00:02<00:37, 2.30it/s] 9%|▊ | 8/93 [00:03<00:35, 2.41it/s] 10%|▉ | 9/93 [00:03<00:35, 2.39it/s] 11%|█ | 10/93 [00:03<00:34, 2.41it/s] 12%|█▏ | 11/93 [00:04<00:35, 2.32it/s] 13%|█▎ | 12/93 [00:04<00:37, 2.16it/s] 14%|█▍ | 13/93 [00:05<00:36, 2.19it/s] 15%|█▌ | 14/93 [00:05<00:35, 2.21it/s] 16%|█▌ | 15/93 [00:06<00:38, 2.05it/s] 17%|█▋ | 16/93 [00:07<00:39, 1.96it/s] 18%|█▊ | 17/93 [00:07<00:37, 2.04it/s] 19%|█▉ | 18/93 [00:07<00:35, 2.12it/s] 20%|██ | 19/93 [00:08<00:35, 2.11it/s] 22%|██▏ | 20/93 [00:08<00:33, 2.20it/s] 23%|██▎ | 21/93 [00:09<00:32, 2.22it/s] 24%|██▎ | 22/93 [00:09<00:37, 1.89it/s] 25%|██▍ | 23/93 [00:10<00:37, 1.89it/s] 26%|██▌ | 24/93 [00:10<00:35, 1.95it/s] 27%|██▋ | 25/93 [00:11<00:33, 2.05it/s] 28%|██▊ | 26/93 [00:11<00:32, 2.08it/s] 29%|██▉ | 27/93 [00:12<00:31, 2.10it/s] 30%|███ | 28/93 [00:14<01:02, 1.03it/s] 31%|███ | 29/93 [00:14<00:51, 1.23it/s] 32%|███▏ | 30/93 [00:15<00:45, 1.40it/s] 33%|███▎ | 31/93 [00:15<00:38, 1.61it/s] 34%|███▍ | 32/93 [00:16<00:37, 1.64it/s] 35%|███▌ | 33/93 [00:17<00:38, 1.54it/s] 37%|███▋ | 34/93 [00:17<00:34, 1.71it/s] 38%|███▊ | 35/93 [00:17<00:32, 1.81it/s] 39%|███▊ | 36/93 [00:18<00:31, 1.83it/s] 40%|███▉ | 37/93 [00:19<00:32, 1.72it/s] 41%|████ | 38/93 [00:19<00:31, 1.73it/s] 42%|████▏ | 39/93 [00:20<00:29, 1.83it/s] 43%|████▎ | 40/93 [00:20<00:28, 1.89it/s] 44%|████▍ | 41/93 [00:21<00:25, 2.02it/s] 45%|████▌ | 42/93 [00:21<00:24, 2.08it/s] 46%|████▌ | 43/93 [00:21<00:22, 2.20it/s] 47%|████▋ | 44/93 [00:22<00:21, 2.30it/s] 48%|████▊ | 45/93 [00:22<00:21, 2.20it/s] 49%|████▉ | 46/93 [00:23<00:23, 2.01it/s] 51%|█████ | 47/93 [00:23<00:21, 2.17it/s] 52%|█████▏ | 48/93 [00:24<00:24, 1.85it/s] 53%|█████▎ | 49/93 [00:24<00:21, 2.02it/s] 54%|█████▍ | 50/93 [00:25<00:21, 1.98it/s] 55%|█████▍ | 51/93 [00:25<00:20, 2.08it/s] 56%|█████▌ | 52/93 [00:26<00:22, 1.81it/s] 57%|█████▋ | 53/93 [00:27<00:20, 1.96it/s] 58%|█████▊ | 54/93 [00:27<00:19, 2.03it/s] 59%|█████▉ | 55/93 [00:27<00:17, 2.12it/s] 60%|██████ | 56/93 [00:28<00:17, 2.15it/s] 61%|██████▏ | 57/93 [00:28<00:17, 2.01it/s] 62%|██████▏ | 58/93 [00:29<00:16, 2.07it/s] 63%|██████▎ | 59/93 [00:29<00:17, 1.98it/s] 65%|██████▍ | 60/93 [00:30<00:15, 2.14it/s] 66%|██████▌ | 61/93 [00:30<00:15, 2.10it/s] 67%|██████▋ | 62/93 [00:31<00:15, 2.01it/s] 68%|██████▊ | 63/93 [00:31<00:13, 2.18it/s] 69%|██████▉ | 64/93 [00:32<00:12, 2.28it/s] 70%|██████▉ | 65/93 [00:32<00:11, 2.34it/s] 71%|███████ | 66/93 [00:32<00:11, 2.40it/s] 72%|███████▏ | 67/93 [00:33<00:13, 1.96it/s] 73%|███████▎ | 68/93 [00:34<00:13, 1.89it/s] 74%|███████▍ | 69/93 [00:34<00:12, 1.88it/s] 75%|███████▌ | 70/93 [00:35<00:12, 1.90it/s] 76%|███████▋ | 71/93 [00:35<00:12, 1.82it/s] 77%|███████▋ | 72/93 [00:36<00:10, 1.94it/s] 78%|███████▊ | 73/93 [00:36<00:09, 2.06it/s] 80%|███████▉ | 74/93 [00:37<00:09, 2.03it/s] 81%|████████ | 75/93 [00:37<00:09, 1.96it/s] 82%|████████▏ | 76/93 [00:38<00:09, 1.85it/s] 83%|████████▎ | 77/93 [00:38<00:08, 1.82it/s] 84%|████████▍ | 78/93 [00:40<00:14, 1.01it/s] 85%|████████▍ | 79/93 [00:41<00:11, 1.22it/s] 86%|████████▌ | 80/93 [00:41<00:09, 1.35it/s] 87%|████████▋ | 81/93 [00:42<00:07, 1.52it/s] 88%|████████▊ | 82/93 [00:42<00:06, 1.68it/s] 89%|████████▉ | 83/93 [00:43<00:05, 1.83it/s] 90%|█████████ | 84/93 [00:43<00:04, 1.91it/s] 91%|█████████▏| 85/93 [00:44<00:04, 1.92it/s] 92%|█████████▏| 86/93 [00:44<00:03, 1.92it/s] 94%|█████████▎| 87/93 [00:45<00:02, 2.02it/s] 95%|█████████▍| 88/93 [00:45<00:02, 2.18it/s] 96%|█████████▌| 89/93 [00:46<00:01, 2.07it/s] 97%|█████████▋| 90/93 [00:46<00:01, 2.16it/s] 98%|█████████▊| 91/93 [00:47<00:00, 2.03it/s] 99%|█████████▉| 92/93 [00:47<00:00, 2.03it/s] 100%|██████████| 93/93 [00:47<00:00, 2.21it/s] 100%|██████████| 93/93 [00:48<00:00, 1.92it/s]
***** predict_test_es_ES metrics *****
predict_ex_match_acc = 0.6009
predict_ex_match_acc_stderr = 0.009
predict_intent_acc = 0.8521
predict_intent_acc_stderr = 0.0065
predict_loss = 0.2917
predict_runtime = 0:00:48.87
predict_samples = 2974
predict_samples_per_second = 60.848
predict_slot_micro_f1 = 0.7136
predict_slot_micro_f1_stderr = 0.0031
predict_steps_per_second = 1.903
02/05/2024 18:50:06 - INFO - __main__ - *** test_fr_FR ***
[INFO|trainer.py:718] 2024-02-05 18:50:06,022 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:50:06,024 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:50:06,025 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:50:06,025 >> Batch size = 32
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:19, 4.63it/s] 3%|▎ | 3/93 [00:00<00:29, 3.02it/s] 4%|▍ | 4/93 [00:01<00:35, 2.49it/s] 5%|▌ | 5/93 [00:01<00:40, 2.20it/s] 6%|▋ | 6/93 [00:02<00:39, 2.20it/s] 8%|▊ | 7/93 [00:02<00:40, 2.14it/s] 9%|▊ | 8/93 [00:03<00:38, 2.20it/s] 10%|▉ | 9/93 [00:03<00:38, 2.20it/s] 11%|█ | 10/93 [00:04<00:36, 2.26it/s] 12%|█▏ | 11/93 [00:04<00:38, 2.12it/s] 13%|█▎ | 12/93 [00:05<00:38, 2.09it/s] 14%|█▍ | 13/93 [00:05<00:38, 2.07it/s] 15%|█▌ | 14/93 [00:06<00:39, 2.02it/s] 16%|█▌ | 15/93 [00:06<00:40, 1.92it/s] 17%|█▋ | 16/93 [00:07<00:41, 1.86it/s] 18%|█▊ | 17/93 [00:08<00:42, 1.79it/s] 19%|█▉ | 18/93 [00:08<00:39, 1.91it/s] 20%|██ | 19/93 [00:08<00:36, 2.00it/s] 22%|██▏ | 20/93 [00:09<00:36, 1.99it/s] 23%|██▎ | 21/93 [00:09<00:34, 2.08it/s] 24%|██▎ | 22/93 [00:10<00:36, 1.92it/s] 25%|██▍ | 23/93 [00:11<00:37, 1.86it/s] 26%|██▌ | 24/93 [00:11<00:38, 1.81it/s] 27%|██▋ | 25/93 [00:12<00:36, 1.85it/s] 28%|██▊ | 26/93 [00:12<00:34, 1.94it/s] 29%|██▉ | 27/93 [00:13<00:33, 1.98it/s] 30%|███ | 28/93 [00:15<01:04, 1.01it/s] 31%|███ | 29/93 [00:15<00:52, 1.21it/s] 32%|███▏ | 30/93 [00:16<00:45, 1.38it/s] 33%|███▎ | 31/93 [00:16<00:39, 1.58it/s] 34%|███▍ | 32/93 [00:17<00:36, 1.68it/s] 35%|███▌ | 33/93 [00:17<00:33, 1.81it/s] 37%|███▋ | 34/93 [00:17<00:29, 1.97it/s] 38%|███▊ | 35/93 [00:18<00:31, 1.84it/s] 39%|███▊ | 36/93 [00:19<00:30, 1.88it/s] 40%|███▉ | 37/93 [00:19<00:29, 1.87it/s] 41%|████ | 38/93 [00:20<00:30, 1.81it/s] 42%|████▏ | 39/93 [00:20<00:29, 1.85it/s] 43%|████▎ | 40/93 [00:21<00:28, 1.88it/s] 44%|████▍ | 41/93 [00:21<00:27, 1.91it/s] 45%|████▌ | 42/93 [00:22<00:25, 1.97it/s] 46%|████▌ | 43/93 [00:22<00:25, 1.95it/s] 47%|████▋ | 44/93 [00:23<00:23, 2.07it/s] 48%|████▊ | 45/93 [00:23<00:23, 2.06it/s] 49%|████▉ | 46/93 [00:24<00:22, 2.12it/s] 51%|█████ | 47/93 [00:24<00:21, 2.14it/s] 52%|█████▏ | 48/93 [00:25<00:23, 1.89it/s] 53%|█████▎ | 49/93 [00:25<00:23, 1.87it/s] 54%|█████▍ | 50/93 [00:26<00:22, 1.88it/s] 55%|█████▍ | 51/93 [00:26<00:21, 1.94it/s] 56%|█████▌ | 52/93 [00:27<00:24, 1.67it/s] 57%|█████▋ | 53/93 [00:27<00:21, 1.84it/s] 58%|█████▊ | 54/93 [00:28<00:20, 1.94it/s] 59%|█████▉ | 55/93 [00:28<00:19, 1.96it/s] 60%|██████ | 56/93 [00:29<00:18, 1.98it/s] 61%|██████▏ | 57/93 [00:29<00:17, 2.01it/s] 62%|██████▏ | 58/93 [00:30<00:17, 2.00it/s] 63%|██████▎ | 59/93 [00:30<00:16, 2.04it/s] 65%|██████▍ | 60/93 [00:31<00:15, 2.14it/s] 66%|██████▌ | 61/93 [00:31<00:16, 1.98it/s] 67%|██████▋ | 62/93 [00:32<00:15, 1.98it/s] 68%|██████▊ | 63/93 [00:32<00:14, 2.12it/s] 69%|██████▉ | 64/93 [00:33<00:13, 2.23it/s] 70%|██████▉ | 65/93 [00:33<00:12, 2.29it/s] 71%|███████ | 66/93 [00:34<00:11, 2.28it/s] 72%|███████▏ | 67/93 [00:34<00:11, 2.23it/s] 73%|███████▎ | 68/93 [00:35<00:12, 1.94it/s] 74%|███████▍ | 69/93 [00:35<00:12, 1.89it/s] 75%|███████▌ | 70/93 [00:36<00:12, 1.86it/s] 76%|███████▋ | 71/93 [00:38<00:21, 1.02it/s] 77%|███████▋ | 72/93 [00:38<00:17, 1.18it/s] 78%|███████▊ | 73/93 [00:39<00:14, 1.39it/s] 80%|███████▉ | 74/93 [00:39<00:12, 1.52it/s] 81%|████████ | 75/93 [00:40<00:11, 1.57it/s] 82%|████████▏ | 76/93 [00:40<00:10, 1.60it/s] 83%|████████▎ | 77/93 [00:41<00:09, 1.68it/s] 84%|████████▍ | 78/93 [00:42<00:09, 1.65it/s] 85%|████████▍ | 79/93 [00:42<00:07, 1.78it/s] 86%|████████▌ | 80/93 [00:43<00:07, 1.82it/s] 87%|████████▋ | 81/93 [00:43<00:06, 1.91it/s] 88%|████████▊ | 82/93 [00:43<00:05, 2.00it/s] 89%|████████▉ | 83/93 [00:44<00:04, 2.12it/s] 90%|█████████ | 84/93 [00:44<00:04, 2.14it/s] 91%|█████████▏| 85/93 [00:45<00:03, 2.06it/s] 92%|█████████▏| 86/93 [00:45<00:03, 2.09it/s] 94%|█████████▎| 87/93 [00:46<00:02, 2.14it/s] 95%|█████████▍| 88/93 [00:46<00:02, 2.20it/s] 96%|█████████▌| 89/93 [00:47<00:01, 2.11it/s] 97%|█████████▋| 90/93 [00:47<00:01, 2.13it/s] 98%|█████████▊| 91/93 [00:48<00:01, 1.96it/s] 99%|█████████▉| 92/93 [00:48<00:00, 1.86it/s] 100%|██████████| 93/93 [00:49<00:00, 2.05it/s] 100%|██████████| 93/93 [00:49<00:00, 1.88it/s]
***** predict_test_fr_FR metrics *****
predict_ex_match_acc = 0.459
predict_ex_match_acc_stderr = 0.0091
predict_intent_acc = 0.8366
predict_intent_acc_stderr = 0.0068
predict_loss = 0.675
predict_runtime = 0:00:50.06
predict_samples = 2974
predict_samples_per_second = 59.402
predict_slot_micro_f1 = 0.5132
predict_slot_micro_f1_stderr = 0.0034
predict_steps_per_second = 1.858
02/05/2024 18:50:56 - INFO - __main__ - *** test_hu_HU ***
[INFO|trainer.py:718] 2024-02-05 18:50:56,344 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:50:56,347 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:50:56,347 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:50:56,347 >> Batch size = 32
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:15, 5.76it/s] 3%|▎ | 3/93 [00:00<00:22, 4.06it/s] 4%|▍ | 4/93 [00:01<00:33, 2.67it/s] 5%|▌ | 5/93 [00:01<00:33, 2.66it/s] 6%|▋ | 6/93 [00:01<00:31, 2.77it/s] 8%|▊ | 7/93 [00:02<00:32, 2.65it/s] 9%|▊ | 8/93 [00:02<00:31, 2.69it/s] 10%|▉ | 9/93 [00:03<00:31, 2.67it/s] 11%|█ | 10/93 [00:03<00:31, 2.61it/s] 12%|█▏ | 11/93 [00:03<00:30, 2.66it/s] 13%|█▎ | 12/93 [00:04<00:31, 2.57it/s] 14%|█▍ | 13/93 [00:04<00:32, 2.45it/s] 15%|█▌ | 14/93 [00:05<00:32, 2.43it/s] 16%|█▌ | 15/93 [00:05<00:31, 2.50it/s] 17%|█▋ | 16/93 [00:06<00:35, 2.16it/s] 18%|█▊ | 17/93 [00:06<00:35, 2.16it/s] 19%|█▉ | 18/93 [00:07<00:33, 2.22it/s] 20%|██ | 19/93 [00:07<00:31, 2.38it/s] 22%|██▏ | 20/93 [00:07<00:29, 2.50it/s] 23%|██▎ | 21/93 [00:08<00:29, 2.48it/s] 24%|██▎ | 22/93 [00:08<00:28, 2.50it/s] 25%|██▍ | 23/93 [00:09<00:29, 2.36it/s] 26%|██▌ | 24/93 [00:09<00:30, 2.27it/s] 27%|██▋ | 25/93 [00:09<00:29, 2.28it/s] 28%|██▊ | 26/93 [00:10<00:28, 2.38it/s] 29%|██▉ | 27/93 [00:10<00:26, 2.48it/s] 30%|███ | 28/93 [00:12<00:59, 1.09it/s] 31%|███ | 29/93 [00:13<00:48, 1.33it/s] 32%|███▏ | 30/93 [00:13<00:42, 1.47it/s] 33%|███▎ | 31/93 [00:14<00:36, 1.71it/s] 34%|███▍ | 32/93 [00:14<00:34, 1.76it/s] 35%|███▌ | 33/93 [00:15<00:33, 1.81it/s] 37%|███▋ | 34/93 [00:15<00:30, 1.93it/s] 38%|███▊ | 35/93 [00:15<00:28, 2.06it/s] 39%|███▊ | 36/93 [00:16<00:26, 2.14it/s] 40%|███▉ | 37/93 [00:16<00:25, 2.22it/s] 41%|████ | 38/93 [00:17<00:25, 2.18it/s] 42%|████▏ | 39/93 [00:17<00:25, 2.14it/s] 43%|████▎ | 40/93 [00:18<00:23, 2.24it/s] 44%|████▍ | 41/93 [00:18<00:22, 2.33it/s] 45%|████▌ | 42/93 [00:18<00:21, 2.40it/s] 46%|████▌ | 43/93 [00:19<00:20, 2.49it/s] 47%|████▋ | 44/93 [00:19<00:21, 2.25it/s] 48%|████▊ | 45/93 [00:20<00:21, 2.26it/s] 49%|████▉ | 46/93 [00:20<00:21, 2.22it/s] 51%|█████ | 47/93 [00:21<00:19, 2.33it/s] 52%|█████▏ | 48/93 [00:21<00:19, 2.35it/s] 53%|█████▎ | 49/93 [00:21<00:17, 2.45it/s] 54%|█████▍ | 50/93 [00:22<00:17, 2.49it/s] 55%|█████▍ | 51/93 [00:22<00:16, 2.51it/s] 56%|█████▌ | 52/93 [00:23<00:16, 2.47it/s] 57%|█████▋ | 53/93 [00:23<00:16, 2.42it/s] 58%|█████▊ | 54/93 [00:23<00:16, 2.35it/s] 59%|█████▉ | 55/93 [00:24<00:15, 2.43it/s] 60%|██████ | 56/93 [00:24<00:14, 2.52it/s] 61%|██████▏ | 57/93 [00:25<00:14, 2.47it/s] 62%|██████▏ | 58/93 [00:25<00:13, 2.54it/s] 63%|██████▎ | 59/93 [00:25<00:12, 2.62it/s] 65%|██████▍ | 60/93 [00:26<00:12, 2.68it/s] 66%|██████▌ | 61/93 [00:26<00:13, 2.33it/s] 67%|██████▋ | 62/93 [00:27<00:12, 2.41it/s] 68%|██████▊ | 63/93 [00:27<00:12, 2.46it/s] 69%|██████▉ | 64/93 [00:28<00:13, 2.16it/s] 70%|██████▉ | 65/93 [00:28<00:11, 2.34it/s] 71%|███████ | 66/93 [00:28<00:11, 2.38it/s] 72%|███████▏ | 67/93 [00:29<00:10, 2.45it/s] 73%|███████▎ | 68/93 [00:29<00:11, 2.26it/s] 74%|███████▍ | 69/93 [00:30<00:11, 2.13it/s] 75%|███████▌ | 70/93 [00:30<00:10, 2.21it/s] 76%|███████▋ | 71/93 [00:31<00:11, 1.99it/s] 77%|███████▋ | 72/93 [00:31<00:09, 2.12it/s] 78%|███████▊ | 73/93 [00:32<00:08, 2.23it/s] 80%|███████▉ | 74/93 [00:32<00:08, 2.25it/s] 81%|████████ | 75/93 [00:33<00:08, 2.23it/s] 82%|████████▏ | 76/93 [00:33<00:07, 2.30it/s] 83%|████████▎ | 77/93 [00:33<00:06, 2.31it/s] 84%|████████▍ | 78/93 [00:34<00:06, 2.26it/s] 85%|████████▍ | 79/93 [00:34<00:06, 2.16it/s] 86%|████████▌ | 80/93 [00:35<00:06, 2.12it/s] 87%|████████▋ | 81/93 [00:35<00:05, 2.28it/s] 88%|████████▊ | 82/93 [00:36<00:04, 2.22it/s] 89%|████████▉ | 83/93 [00:36<00:04, 2.22it/s] 90%|█████████ | 84/93 [00:37<00:03, 2.32it/s] 91%|█████████▏| 85/93 [00:37<00:03, 2.19it/s] 92%|█████████▏| 86/93 [00:38<00:03, 1.98it/s] 94%|█████████▎| 87/93 [00:38<00:02, 2.09it/s] 95%|█████████▍| 88/93 [00:38<00:02, 2.26it/s] 96%|█████████▌| 89/93 [00:39<00:01, 2.18it/s] 97%|█████████▋| 90/93 [00:39<00:01, 2.30it/s] 98%|█████████▊| 91/93 [00:40<00:00, 2.17it/s] 99%|█████████▉| 92/93 [00:40<00:00, 2.18it/s] 100%|██████████| 93/93 [00:41<00:00, 2.39it/s] 100%|██████████| 93/93 [00:41<00:00, 2.25it/s]
***** predict_test_hu_HU metrics *****
predict_ex_match_acc = 0.273
predict_ex_match_acc_stderr = 0.0082
predict_intent_acc = 0.5454
predict_intent_acc_stderr = 0.0091
predict_loss = 0.7601
predict_runtime = 0:00:41.75
predict_samples = 2974
predict_samples_per_second = 71.228
predict_slot_micro_f1 = 0.5011
predict_slot_micro_f1_stderr = 0.0038
predict_steps_per_second = 2.227
02/05/2024 18:51:38 - INFO - __main__ - *** test_ko_KR ***
[INFO|trainer.py:718] 2024-02-05 18:51:38,346 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:51:38,349 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:51:38,349 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:51:38,350 >> Batch size = 32
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:18, 5.00it/s] 3%|▎ | 3/93 [00:00<00:21, 4.24it/s] 4%|▍ | 4/93 [00:01<00:25, 3.45it/s] 5%|▌ | 5/93 [00:01<00:26, 3.35it/s] 6%|▋ | 6/93 [00:01<00:25, 3.38it/s] 8%|▊ | 7/93 [00:02<00:26, 3.21it/s] 9%|▊ | 8/93 [00:02<00:27, 3.14it/s] 10%|▉ | 9/93 [00:02<00:27, 3.07it/s] 11%|█ | 10/93 [00:03<00:27, 3.06it/s] 12%|█▏ | 11/93 [00:03<00:27, 3.03it/s] 13%|█▎ | 12/93 [00:03<00:28, 2.85it/s] 14%|█▍ | 13/93 [00:04<00:29, 2.72it/s] 15%|█▌ | 14/93 [00:04<00:28, 2.81it/s] 16%|█▌ | 15/93 [00:04<00:28, 2.72it/s] 17%|█▋ | 16/93 [00:05<00:27, 2.76it/s] 18%|█▊ | 17/93 [00:05<00:27, 2.77it/s] 19%|█▉ | 18/93 [00:05<00:27, 2.69it/s] 20%|██ | 19/93 [00:06<00:27, 2.71it/s] 22%|██▏ | 20/93 [00:06<00:27, 2.69it/s] 23%|██▎ | 21/93 [00:07<00:25, 2.78it/s] 24%|██▎ | 22/93 [00:07<00:25, 2.78it/s] 25%|██▍ | 23/93 [00:07<00:25, 2.73it/s] 26%|██▌ | 24/93 [00:08<00:24, 2.85it/s] 27%|██▋ | 25/93 [00:08<00:24, 2.77it/s] 28%|██▊ | 26/93 [00:08<00:25, 2.65it/s] 29%|██▉ | 27/93 [00:09<00:24, 2.74it/s] 30%|███ | 28/93 [00:09<00:24, 2.63it/s] 31%|███ | 29/93 [00:10<00:23, 2.74it/s] 32%|███▏ | 30/93 [00:10<00:23, 2.74it/s] 33%|███▎ | 31/93 [00:10<00:22, 2.80it/s] 34%|███▍ | 32/93 [00:11<00:22, 2.65it/s] 35%|███▌ | 33/93 [00:11<00:22, 2.67it/s] 37%|███▋ | 34/93 [00:11<00:21, 2.77it/s] 38%|███▊ | 35/93 [00:12<00:23, 2.47it/s] 39%|███▊ | 36/93 [00:12<00:21, 2.60it/s] 40%|███▉ | 37/93 [00:13<00:20, 2.67it/s] 41%|████ | 38/93 [00:13<00:21, 2.61it/s] 42%|████▏ | 39/93 [00:13<00:21, 2.57it/s] 43%|████▎ | 40/93 [00:14<00:20, 2.61it/s] 44%|████▍ | 41/93 [00:14<00:19, 2.67it/s] 45%|████▌ | 42/93 [00:14<00:18, 2.77it/s] 46%|████▌ | 43/93 [00:15<00:18, 2.70it/s] 47%|████▋ | 44/93 [00:15<00:17, 2.73it/s] 48%|████▊ | 45/93 [00:15<00:17, 2.81it/s] 49%|████▉ | 46/93 [00:16<00:16, 2.89it/s] 51%|█████ | 47/93 [00:16<00:15, 2.88it/s] 52%|█████▏ | 48/93 [00:17<00:16, 2.79it/s] 53%|█████▎ | 49/93 [00:17<00:15, 2.89it/s] 54%|█████▍ | 50/93 [00:17<00:15, 2.82it/s] 55%|█████▍ | 51/93 [00:18<00:14, 2.84it/s] 56%|█████▌ | 52/93 [00:18<00:15, 2.69it/s] 57%|█████▋ | 53/93 [00:18<00:15, 2.64it/s] 58%|█████▊ | 54/93 [00:19<00:14, 2.72it/s] 59%|█████▉ | 55/93 [00:19<00:13, 2.82it/s] 60%|██████ | 56/93 [00:19<00:13, 2.77it/s] 61%|██████▏ | 57/93 [00:20<00:13, 2.67it/s] 62%|██████▏ | 58/93 [00:20<00:12, 2.73it/s] 63%|██████▎ | 59/93 [00:21<00:12, 2.76it/s] 65%|██████▍ | 60/93 [00:21<00:11, 2.83it/s] 66%|██████▌ | 61/93 [00:21<00:11, 2.69it/s] 67%|██████▋ | 62/93 [00:22<00:11, 2.78it/s] 68%|██████▊ | 63/93 [00:22<00:10, 2.89it/s] 69%|██████▉ | 64/93 [00:22<00:10, 2.86it/s] 70%|██████▉ | 65/93 [00:23<00:09, 2.90it/s] 71%|███████ | 66/93 [00:23<00:09, 2.98it/s] 72%|███████▏ | 67/93 [00:23<00:09, 2.70it/s] 73%|███████▎ | 68/93 [00:24<00:09, 2.64it/s] 74%|███████▍ | 69/93 [00:24<00:08, 2.83it/s] 75%|███████▌ | 70/93 [00:24<00:08, 2.74it/s] 76%|███████▋ | 71/93 [00:25<00:08, 2.61it/s] 77%|███████▋ | 72/93 [00:25<00:08, 2.59it/s] 78%|███████▊ | 73/93 [00:26<00:07, 2.69it/s] 80%|███████▉ | 74/93 [00:26<00:07, 2.54it/s] 81%|████████ | 75/93 [00:26<00:07, 2.52it/s] 82%|████████▏ | 76/93 [00:27<00:06, 2.60it/s] 83%|████████▎ | 77/93 [00:27<00:05, 2.70it/s] 84%|████████▍ | 78/93 [00:28<00:05, 2.64it/s] 85%|████████▍ | 79/93 [00:28<00:05, 2.67it/s] 86%|████████▌ | 80/93 [00:28<00:04, 2.75it/s] 87%|████████▋ | 81/93 [00:29<00:04, 2.81it/s] 88%|████████▊ | 82/93 [00:29<00:04, 2.69it/s] 89%|████████▉ | 83/93 [00:29<00:03, 2.68it/s] 90%|█████████ | 84/93 [00:30<00:03, 2.74it/s] 91%|█████████▏| 85/93 [00:30<00:03, 2.63it/s] 92%|█████████▏| 86/93 [00:31<00:02, 2.63it/s] 94%|█████████▎| 87/93 [00:31<00:02, 2.79it/s] 95%|█████████▍| 88/93 [00:31<00:01, 2.86it/s] 96%|█████████▌| 89/93 [00:32<00:01, 2.83it/s] 97%|█████████▋| 90/93 [00:32<00:01, 2.76it/s] 98%|█████████▊| 91/93 [00:32<00:00, 2.77it/s] 99%|█████████▉| 92/93 [00:33<00:00, 2.75it/s] 100%|██████████| 93/93 [00:33<00:00, 2.75it/s] 100%|██████████| 93/93 [00:33<00:00, 2.76it/s]
***** predict_test_ko_KR metrics *****
predict_ex_match_acc = 0.0333
predict_ex_match_acc_stderr = 0.0033
predict_intent_acc = 0.0568
predict_intent_acc_stderr = 0.0042
predict_loss = 1.5014
predict_runtime = 0:00:34.05
predict_samples = 2974
predict_samples_per_second = 87.32
predict_slot_micro_f1 = 0.1925
predict_slot_micro_f1_stderr = 0.0033
predict_steps_per_second = 2.731
02/05/2024 18:52:12 - INFO - __main__ - *** test_nl_NL ***
[INFO|trainer.py:718] 2024-02-05 18:52:12,643 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:52:12,646 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:52:12,646 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:52:12,646 >> Batch size = 32
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:19, 4.55it/s] 3%|▎ | 3/93 [00:00<00:31, 2.84it/s] 4%|▍ | 4/93 [00:01<00:33, 2.66it/s] 5%|▌ | 5/93 [00:01<00:33, 2.65it/s] 6%|▋ | 6/93 [00:02<00:33, 2.58it/s] 8%|▊ | 7/93 [00:02<00:35, 2.42it/s] 9%|▊ | 8/93 [00:03<00:35, 2.41it/s] 10%|▉ | 9/93 [00:03<00:35, 2.34it/s] 11%|█ | 10/93 [00:03<00:36, 2.27it/s] 12%|█▏ | 11/93 [00:04<00:36, 2.22it/s] 13%|█▎ | 12/93 [00:04<00:36, 2.19it/s] 14%|█▍ | 13/93 [00:05<00:36, 2.20it/s] 15%|█▌ | 14/93 [00:05<00:34, 2.26it/s] 16%|█▌ | 15/93 [00:06<00:38, 2.00it/s] 17%|█▋ | 16/93 [00:07<00:40, 1.92it/s] 18%|█▊ | 17/93 [00:07<00:40, 1.90it/s] 19%|█▉ | 18/93 [00:07<00:37, 1.99it/s] 20%|██ | 19/93 [00:08<00:35, 2.07it/s] 22%|██▏ | 20/93 [00:08<00:34, 2.12it/s] 23%|██▎ | 21/93 [00:09<00:32, 2.20it/s] 24%|██▎ | 22/93 [00:09<00:36, 1.97it/s] 25%|██▍ | 23/93 [00:10<00:34, 2.04it/s] 26%|██▌ | 24/93 [00:10<00:34, 2.01it/s] 27%|██▋ | 25/93 [00:11<00:33, 2.05it/s] 28%|██▊ | 26/93 [00:11<00:31, 2.12it/s] 29%|██▉ | 27/93 [00:12<00:30, 2.20it/s] 30%|███ | 28/93 [00:14<01:01, 1.06it/s] 31%|███ | 29/93 [00:14<00:49, 1.30it/s] 32%|███▏ | 30/93 [00:15<00:42, 1.48it/s] 33%|███▎ | 31/93 [00:15<00:37, 1.66it/s] 34%|███▍ | 32/93 [00:16<00:34, 1.75it/s] 35%|███▌ | 33/93 [00:16<00:34, 1.76it/s] 37%|███▋ | 34/93 [00:16<00:30, 1.94it/s] 38%|███▊ | 35/93 [00:17<00:29, 1.96it/s] 39%|███▊ | 36/93 [00:17<00:27, 2.06it/s] 40%|███▉ | 37/93 [00:18<00:26, 2.10it/s] 41%|████ | 38/93 [00:19<00:30, 1.79it/s] 42%|████▏ | 39/93 [00:19<00:30, 1.80it/s] 43%|████▎ | 40/93 [00:20<00:27, 1.95it/s] 44%|████▍ | 41/93 [00:20<00:24, 2.11it/s] 45%|████▌ | 42/93 [00:20<00:23, 2.18it/s] 46%|████▌ | 43/93 [00:21<00:21, 2.29it/s] 47%|████▋ | 44/93 [00:21<00:20, 2.35it/s] 48%|████▊ | 45/93 [00:22<00:20, 2.30it/s] 49%|████▉ | 46/93 [00:22<00:20, 2.31it/s] 51%|█████ | 47/93 [00:22<00:19, 2.41it/s] 52%|█████▏ | 48/93 [00:23<00:18, 2.39it/s] 53%|█████▎ | 49/93 [00:23<00:18, 2.41it/s] 54%|█████▍ | 50/93 [00:24<00:17, 2.44it/s] 55%|█████▍ | 51/93 [00:24<00:19, 2.20it/s] 56%|█████▌ | 52/93 [00:25<00:19, 2.12it/s] 57%|█████▋ | 53/93 [00:25<00:19, 2.09it/s] 58%|█████▊ | 54/93 [00:26<00:18, 2.16it/s] 59%|█████▉ | 55/93 [00:26<00:16, 2.31it/s] 60%|██████ | 56/93 [00:26<00:15, 2.37it/s] 61%|██████▏ | 57/93 [00:27<00:15, 2.35it/s] 62%|██████▏ | 58/93 [00:27<00:15, 2.26it/s] 63%|██████▎ | 59/93 [00:28<00:14, 2.29it/s] 65%|██████▍ | 60/93 [00:28<00:14, 2.35it/s] 66%|██████▌ | 61/93 [00:29<00:14, 2.17it/s] 67%|██████▋ | 62/93 [00:29<00:14, 2.15it/s] 68%|██████▊ | 63/93 [00:30<00:13, 2.29it/s] 69%|██████▉ | 64/93 [00:30<00:13, 2.17it/s] 70%|██████▉ | 65/93 [00:30<00:11, 2.34it/s] 71%|███████ | 66/93 [00:31<00:11, 2.35it/s] 72%|███████▏ | 67/93 [00:31<00:10, 2.39it/s] 73%|███████▎ | 68/93 [00:32<00:10, 2.29it/s] 74%|███████▍ | 69/93 [00:32<00:11, 2.12it/s] 75%|███████▌ | 70/93 [00:33<00:10, 2.15it/s] 76%|███████▋ | 71/93 [00:33<00:11, 1.95it/s] 77%|███████▋ | 72/93 [00:34<00:10, 1.93it/s] 78%|███████▊ | 73/93 [00:34<00:09, 2.07it/s] 80%|███████▉ | 74/93 [00:35<00:09, 1.99it/s] 81%|████████ | 75/93 [00:35<00:09, 1.98it/s] 82%|████████▏ | 76/93 [00:36<00:07, 2.13it/s] 83%|████████▎ | 77/93 [00:36<00:07, 2.16it/s] 84%|████████▍ | 78/93 [00:37<00:07, 1.99it/s] 85%|████████▍ | 79/93 [00:37<00:07, 1.97it/s] 86%|████████▌ | 80/93 [00:38<00:06, 2.03it/s] 87%|████████▋ | 81/93 [00:38<00:05, 2.14it/s] 88%|████████▊ | 82/93 [00:39<00:05, 2.19it/s] 89%|████████▉ | 83/93 [00:39<00:04, 2.16it/s] 90%|█████████ | 84/93 [00:39<00:04, 2.21it/s] 91%|█████████▏| 85/93 [00:40<00:03, 2.28it/s] 92%|█████████▏| 86/93 [00:40<00:03, 2.15it/s] 94%|█████████▎| 87/93 [00:41<00:02, 2.23it/s] 95%|█████████▍| 88/93 [00:41<00:02, 2.32it/s] 96%|█████████▌| 89/93 [00:42<00:01, 2.20it/s] 97%|█████████▋| 90/93 [00:42<00:01, 2.16it/s] 98%|█████████▊| 91/93 [00:43<00:00, 2.06it/s] 99%|█████████▉| 92/93 [00:43<00:00, 2.10it/s] 100%|██████████| 93/93 [00:44<00:00, 2.18it/s] 100%|██████████| 93/93 [00:44<00:00, 2.10it/s]
***** predict_test_nl_NL metrics *****
predict_ex_match_acc = 0.5928
predict_ex_match_acc_stderr = 0.009
predict_intent_acc = 0.8578
predict_intent_acc_stderr = 0.0064
predict_loss = 0.473
predict_runtime = 0:00:44.89
predict_samples = 2974
predict_samples_per_second = 66.244
predict_slot_micro_f1 = 0.6919
predict_slot_micro_f1_stderr = 0.0032
predict_steps_per_second = 2.072
02/05/2024 18:52:57 - INFO - __main__ - *** test_pl_PL ***
[INFO|trainer.py:718] 2024-02-05 18:52:57,776 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:52:57,778 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:52:57,778 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:52:57,779 >> Batch size = 32
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:16, 5.55it/s] 3%|▎ | 3/93 [00:00<00:25, 3.51it/s] 4%|▍ | 4/93 [00:01<00:30, 2.96it/s] 5%|▌ | 5/93 [00:01<00:31, 2.75it/s] 6%|▋ | 6/93 [00:02<00:32, 2.70it/s] 8%|▊ | 7/93 [00:02<00:33, 2.57it/s] 9%|▊ | 8/93 [00:03<00:37, 2.24it/s] 10%|▉ | 9/93 [00:03<00:39, 2.12it/s] 11%|█ | 10/93 [00:03<00:37, 2.22it/s] 12%|█▏ | 11/93 [00:04<00:37, 2.16it/s] 13%|█▎ | 12/93 [00:04<00:34, 2.34it/s] 14%|█▍ | 13/93 [00:05<00:33, 2.37it/s] 15%|█▌ | 14/93 [00:05<00:32, 2.47it/s] 16%|█▌ | 15/93 [00:06<00:32, 2.38it/s] 17%|█▋ | 16/93 [00:06<00:32, 2.37it/s] 18%|█▊ | 17/93 [00:06<00:32, 2.37it/s] 19%|█▉ | 18/93 [00:07<00:31, 2.36it/s] 20%|██ | 19/93 [00:07<00:31, 2.32it/s] 22%|██▏ | 20/93 [00:08<00:33, 2.17it/s] 23%|██▎ | 21/93 [00:08<00:32, 2.23it/s] 24%|██▎ | 22/93 [00:09<00:30, 2.35it/s] 25%|██▍ | 23/93 [00:09<00:29, 2.39it/s] 26%|██▌ | 24/93 [00:09<00:28, 2.45it/s] 27%|██▋ | 25/93 [00:10<00:27, 2.50it/s] 28%|██▊ | 26/93 [00:10<00:26, 2.52it/s] 29%|██▉ | 27/93 [00:11<00:26, 2.53it/s] 30%|███ | 28/93 [00:11<00:26, 2.43it/s] 31%|███ | 29/93 [00:11<00:27, 2.32it/s] 32%|███▏ | 30/93 [00:12<00:26, 2.41it/s] 33%|███▎ | 31/93 [00:12<00:25, 2.42it/s] 34%|███▍ | 32/93 [00:13<00:24, 2.47it/s] 35%|███▌ | 33/93 [00:13<00:24, 2.44it/s] 37%|███▋ | 34/93 [00:13<00:22, 2.59it/s] 38%|███▊ | 35/93 [00:14<00:23, 2.49it/s] 39%|███▊ | 36/93 [00:14<00:25, 2.27it/s] 40%|███▉ | 37/93 [00:15<00:26, 2.14it/s] 41%|████ | 38/93 [00:15<00:25, 2.17it/s] 42%|████▏ | 39/93 [00:16<00:24, 2.23it/s] 43%|████▎ | 40/93 [00:16<00:24, 2.15it/s] 44%|████▍ | 41/93 [00:18<00:49, 1.05it/s] 45%|████▌ | 42/93 [00:19<00:41, 1.23it/s] 46%|████▌ | 43/93 [00:19<00:33, 1.50it/s] 47%|████▋ | 44/93 [00:19<00:28, 1.73it/s] 48%|████▊ | 45/93 [00:20<00:26, 1.82it/s] 49%|████▉ | 46/93 [00:20<00:24, 1.93it/s] 51%|█████ | 47/93 [00:21<00:21, 2.12it/s] 52%|█████▏ | 48/93 [00:21<00:20, 2.22it/s] 53%|█████▎ | 49/93 [00:22<00:19, 2.22it/s] 54%|█████▍ | 50/93 [00:22<00:18, 2.28it/s] 55%|█████▍ | 51/93 [00:22<00:18, 2.30it/s] 56%|█████▌ | 52/93 [00:23<00:17, 2.37it/s] 57%|█████▋ | 53/93 [00:23<00:17, 2.35it/s] 58%|█████▊ | 54/93 [00:24<00:16, 2.44it/s] 59%|█████▉ | 55/93 [00:24<00:15, 2.47it/s] 60%|██████ | 56/93 [00:25<00:16, 2.29it/s] 61%|██████▏ | 57/93 [00:25<00:15, 2.30it/s] 62%|██████▏ | 58/93 [00:25<00:14, 2.35it/s] 63%|██████▎ | 59/93 [00:26<00:14, 2.42it/s] 65%|██████▍ | 60/93 [00:26<00:13, 2.37it/s] 66%|██████▌ | 61/93 [00:27<00:14, 2.26it/s] 67%|██████▋ | 62/93 [00:27<00:13, 2.27it/s] 68%|██████▊ | 63/93 [00:28<00:12, 2.36it/s] 69%|██████▉ | 64/93 [00:28<00:11, 2.50it/s] 70%|██████▉ | 65/93 [00:28<00:11, 2.35it/s] 71%|███████ | 66/93 [00:29<00:12, 2.19it/s] 72%|███████▏ | 67/93 [00:29<00:11, 2.26it/s] 73%|███████▎ | 68/93 [00:30<00:10, 2.34it/s] 74%|███████▍ | 69/93 [00:30<00:10, 2.26it/s] 75%|███████▌ | 70/93 [00:32<00:21, 1.08it/s] 76%|███████▋ | 71/93 [00:33<00:17, 1.26it/s] 77%|███████▋ | 72/93 [00:33<00:14, 1.45it/s] 78%|███████▊ | 73/93 [00:34<00:11, 1.67it/s] 80%|███████▉ | 74/93 [00:34<00:09, 1.91it/s] 81%|████████ | 75/93 [00:34<00:08, 2.08it/s] 82%|████████▏ | 76/93 [00:35<00:08, 2.09it/s] 83%|████████▎ | 77/93 [00:35<00:08, 1.93it/s] 84%|████████▍ | 78/93 [00:36<00:07, 2.07it/s] 85%|████████▍ | 79/93 [00:36<00:06, 2.12it/s] 86%|████████▌ | 80/93 [00:37<00:06, 2.14it/s] 87%|████████▋ | 81/93 [00:37<00:05, 2.01it/s] 88%|████████▊ | 82/93 [00:38<00:05, 2.13it/s] 89%|████████▉ | 83/93 [00:38<00:04, 2.24it/s] 90%|█████████ | 84/93 [00:38<00:03, 2.46it/s] 91%|█████████▏| 85/93 [00:39<00:03, 2.38it/s] 92%|█████████▏| 86/93 [00:39<00:02, 2.35it/s] 94%|█████████▎| 87/93 [00:40<00:02, 2.27it/s] 95%|█████████▍| 88/93 [00:40<00:02, 2.31it/s] 96%|█████████▌| 89/93 [00:41<00:01, 2.40it/s] 97%|█████████▋| 90/93 [00:41<00:01, 2.23it/s] 98%|█████████▊| 91/93 [00:42<00:00, 2.16it/s] 99%|█████████▉| 92/93 [00:42<00:00, 2.15it/s] 100%|██████████| 93/93 [00:42<00:00, 2.29it/s] 100%|██████████| 93/93 [00:43<00:00, 2.16it/s]
***** predict_test_pl_PL metrics *****
predict_ex_match_acc = 0.5128
predict_ex_match_acc_stderr = 0.0092
predict_intent_acc = 0.7993
predict_intent_acc_stderr = 0.0073
predict_loss = 0.3891
predict_runtime = 0:00:43.56
predict_samples = 2974
predict_samples_per_second = 68.259
predict_slot_micro_f1 = 0.6636
predict_slot_micro_f1_stderr = 0.0036
predict_steps_per_second = 2.135
02/05/2024 18:53:41 - INFO - __main__ - *** test_pt_PT ***
[INFO|trainer.py:718] 2024-02-05 18:53:41,587 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:53:41,590 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:53:41,590 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:53:41,590 >> Batch size = 32
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:18, 4.89it/s] 3%|▎ | 3/93 [00:00<00:31, 2.88it/s] 4%|▍ | 4/93 [00:01<00:33, 2.69it/s] 5%|▌ | 5/93 [00:01<00:35, 2.50it/s] 6%|▋ | 6/93 [00:02<00:41, 2.09it/s] 8%|▊ | 7/93 [00:03<00:44, 1.94it/s] 9%|▊ | 8/93 [00:03<00:43, 1.95it/s] 10%|▉ | 9/93 [00:04<00:42, 2.00it/s] 11%|█ | 10/93 [00:04<00:41, 2.01it/s] 12%|█▏ | 11/93 [00:05<00:44, 1.85it/s] 13%|█▎ | 12/93 [00:05<00:40, 2.01it/s] 14%|█▍ | 13/93 [00:06<00:39, 2.02it/s] 15%|█▌ | 14/93 [00:06<00:36, 2.16it/s] 16%|█▌ | 15/93 [00:06<00:37, 2.08it/s] 17%|█▋ | 16/93 [00:07<00:37, 2.04it/s] 18%|█▊ | 17/93 [00:07<00:36, 2.11it/s] 19%|█▉ | 18/93 [00:08<00:34, 2.18it/s] 20%|██ | 19/93 [00:08<00:35, 2.11it/s] 22%|██▏ | 20/93 [00:09<00:36, 2.01it/s] 23%|██▎ | 21/93 [00:09<00:34, 2.08it/s] 24%|██▎ | 22/93 [00:10<00:32, 2.18it/s] 25%|██▍ | 23/93 [00:10<00:31, 2.19it/s] 26%|██▌ | 24/93 [00:11<00:31, 2.19it/s] 27%|██▋ | 25/93 [00:11<00:31, 2.19it/s] 28%|██▊ | 26/93 [00:11<00:29, 2.29it/s] 29%|██▉ | 27/93 [00:12<00:33, 1.99it/s] 30%|███ | 28/93 [00:13<00:34, 1.91it/s] 31%|███ | 29/93 [00:13<00:32, 1.99it/s] 32%|███▏ | 30/93 [00:14<00:29, 2.16it/s] 33%|███▎ | 31/93 [00:14<00:27, 2.24it/s] 34%|███▍ | 32/93 [00:14<00:27, 2.22it/s] 35%|███▌ | 33/93 [00:15<00:29, 2.03it/s] 37%|███▋ | 34/93 [00:16<00:30, 1.93it/s] 38%|███▊ | 35/93 [00:18<00:57, 1.00it/s] 39%|███▊ | 36/93 [00:18<00:48, 1.17it/s] 40%|███▉ | 37/93 [00:19<00:40, 1.37it/s] 41%|████ | 38/93 [00:19<00:35, 1.53it/s] 42%|████▏ | 39/93 [00:20<00:32, 1.68it/s] 43%|████▎ | 40/93 [00:20<00:30, 1.74it/s] 44%|████▍ | 41/93 [00:21<00:28, 1.81it/s] 45%|████▌ | 42/93 [00:21<00:27, 1.83it/s] 46%|████▌ | 43/93 [00:22<00:26, 1.91it/s] 47%|████▋ | 44/93 [00:22<00:26, 1.86it/s] 48%|████▊ | 45/93 [00:23<00:26, 1.84it/s] 49%|████▉ | 46/93 [00:23<00:26, 1.79it/s] 51%|█████ | 47/93 [00:24<00:23, 1.96it/s] 52%|█████▏ | 48/93 [00:24<00:21, 2.05it/s] 53%|█████▎ | 49/93 [00:25<00:21, 2.07it/s] 54%|█████▍ | 50/93 [00:25<00:20, 2.07it/s] 55%|█████▍ | 51/93 [00:26<00:20, 2.04it/s] 56%|█████▌ | 52/93 [00:26<00:24, 1.70it/s] 57%|█████▋ | 53/93 [00:27<00:25, 1.60it/s] 58%|█████▊ | 54/93 [00:28<00:23, 1.66it/s] 59%|█████▉ | 55/93 [00:28<00:20, 1.84it/s] 60%|██████ | 56/93 [00:29<00:19, 1.89it/s] 61%|██████▏ | 57/93 [00:29<00:18, 1.97it/s] 62%|██████▏ | 58/93 [00:30<00:17, 1.96it/s] 63%|██████▎ | 59/93 [00:30<00:16, 2.09it/s] 65%|██████▍ | 60/93 [00:30<00:15, 2.16it/s] 66%|██████▌ | 61/93 [00:31<00:16, 1.94it/s] 67%|██████▋ | 62/93 [00:32<00:15, 1.97it/s] 68%|██████▊ | 63/93 [00:32<00:16, 1.86it/s] 69%|██████▉ | 64/93 [00:33<00:14, 1.94it/s] 70%|██████▉ | 65/93 [00:33<00:13, 2.04it/s] 71%|███████ | 66/93 [00:34<00:12, 2.10it/s] 72%|███████▏ | 67/93 [00:34<00:12, 2.07it/s] 73%|███████▎ | 68/93 [00:34<00:11, 2.08it/s] 74%|███████▍ | 69/93 [00:35<00:11, 2.14it/s] 75%|███████▌ | 70/93 [00:35<00:10, 2.12it/s] 76%|███████▋ | 71/93 [00:36<00:10, 2.15it/s] 77%|███████▋ | 72/93 [00:36<00:09, 2.24it/s] 78%|███████▊ | 73/93 [00:37<00:08, 2.26it/s] 80%|███████▉ | 74/93 [00:37<00:08, 2.22it/s] 81%|████████ | 75/93 [00:38<00:07, 2.31it/s] 82%|████████▏ | 76/93 [00:38<00:07, 2.17it/s] 83%|████████▎ | 77/93 [00:39<00:07, 2.15it/s] 84%|████████▍ | 78/93 [00:39<00:07, 1.99it/s] 85%|████████▍ | 79/93 [00:40<00:06, 2.00it/s] 86%|████████▌ | 80/93 [00:40<00:06, 2.05it/s] 87%|████████▋ | 81/93 [00:41<00:05, 2.13it/s] 88%|████████▊ | 82/93 [00:41<00:05, 1.97it/s] 89%|████████▉ | 83/93 [00:42<00:04, 2.05it/s] 90%|█████████ | 84/93 [00:42<00:04, 2.12it/s] 91%|█████████▏| 85/93 [00:42<00:03, 2.23it/s] 92%|█████████▏| 86/93 [00:43<00:02, 2.38it/s] 94%|█████████▎| 87/93 [00:43<00:02, 2.28it/s] 95%|█████████▍| 88/93 [00:44<00:02, 2.24it/s] 96%|█████████▌| 89/93 [00:44<00:01, 2.21it/s] 97%|█████████▋| 90/93 [00:45<00:01, 2.05it/s] 98%|█████████▊| 91/93 [00:45<00:00, 2.07it/s] 99%|█████████▉| 92/93 [00:46<00:00, 2.06it/s] 100%|██████████| 93/93 [00:46<00:00, 2.19it/s] 100%|██████████| 93/93 [00:46<00:00, 1.98it/s]
***** predict_test_pt_PT metrics *****
predict_ex_match_acc = 0.498
predict_ex_match_acc_stderr = 0.0092
predict_intent_acc = 0.8268
predict_intent_acc_stderr = 0.0069
predict_loss = 0.5365
predict_runtime = 0:00:47.52
predict_samples = 2974
predict_samples_per_second = 62.577
predict_slot_micro_f1 = 0.5705
predict_slot_micro_f1_stderr = 0.0034
predict_steps_per_second = 1.957
02/05/2024 18:54:29 - INFO - __main__ - *** test_ru_RU ***
[INFO|trainer.py:718] 2024-02-05 18:54:29,364 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:54:29,366 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:54:29,367 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:54:29,367 >> Batch size = 32
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:17, 5.16it/s] 3%|▎ | 3/93 [00:00<00:25, 3.55it/s] 4%|▍ | 4/93 [00:01<00:32, 2.77it/s] 5%|▌ | 5/93 [00:01<00:31, 2.76it/s] 6%|▋ | 6/93 [00:01<00:30, 2.81it/s] 8%|▊ | 7/93 [00:02<00:34, 2.46it/s] 9%|▊ | 8/93 [00:02<00:33, 2.52it/s] 10%|▉ | 9/93 [00:03<00:32, 2.60it/s] 11%|█ | 10/93 [00:03<00:31, 2.62it/s] 12%|█▏ | 11/93 [00:04<00:33, 2.42it/s] 13%|█▎ | 12/93 [00:04<00:33, 2.38it/s] 14%|█▍ | 13/93 [00:04<00:31, 2.51it/s] 15%|█▌ | 14/93 [00:05<00:31, 2.53it/s] 16%|█▌ | 15/93 [00:05<00:34, 2.29it/s] 17%|█▋ | 16/93 [00:06<00:36, 2.10it/s] 18%|█▊ | 17/93 [00:06<00:37, 2.04it/s] 19%|█▉ | 18/93 [00:07<00:35, 2.14it/s] 20%|██ | 19/93 [00:07<00:32, 2.27it/s] 22%|██▏ | 20/93 [00:08<00:31, 2.33it/s] 23%|██▎ | 21/93 [00:08<00:29, 2.48it/s] 24%|██▎ | 22/93 [00:08<00:28, 2.52it/s] 25%|██▍ | 23/93 [00:09<00:29, 2.36it/s] 26%|██▌ | 24/93 [00:09<00:28, 2.44it/s] 27%|██▋ | 25/93 [00:10<00:30, 2.26it/s] 28%|██▊ | 26/93 [00:10<00:28, 2.33it/s] 29%|██▉ | 27/93 [00:11<00:29, 2.24it/s] 30%|███ | 28/93 [00:13<01:01, 1.06it/s] 31%|███ | 29/93 [00:13<00:49, 1.30it/s] 32%|███▏ | 30/93 [00:13<00:41, 1.51it/s] 33%|███▎ | 31/93 [00:14<00:36, 1.70it/s] 34%|███▍ | 32/93 [00:15<00:37, 1.61it/s] 35%|███▌ | 33/93 [00:15<00:33, 1.79it/s] 37%|███▋ | 34/93 [00:15<00:31, 1.86it/s] 38%|███▊ | 35/93 [00:16<00:30, 1.92it/s] 39%|███▊ | 36/93 [00:16<00:27, 2.06it/s] 40%|███▉ | 37/93 [00:17<00:27, 2.02it/s] 41%|████ | 38/93 [00:17<00:27, 1.97it/s] 42%|████▏ | 39/93 [00:18<00:27, 1.97it/s] 43%|████▎ | 40/93 [00:18<00:25, 2.11it/s] 44%|████▍ | 41/93 [00:19<00:23, 2.20it/s] 45%|████▌ | 42/93 [00:19<00:21, 2.36it/s] 46%|████▌ | 43/93 [00:19<00:19, 2.52it/s] 47%|████▋ | 44/93 [00:20<00:19, 2.55it/s] 48%|████▊ | 45/93 [00:20<00:19, 2.51it/s] 49%|████▉ | 46/93 [00:21<00:19, 2.46it/s] 51%|█████ | 47/93 [00:21<00:17, 2.57it/s] 52%|█████▏ | 48/93 [00:22<00:19, 2.25it/s] 53%|█████▎ | 49/93 [00:22<00:18, 2.41it/s] 54%|█████▍ | 50/93 [00:22<00:18, 2.28it/s] 55%|█████▍ | 51/93 [00:23<00:19, 2.11it/s] 56%|█████▌ | 52/93 [00:24<00:21, 1.95it/s] 57%|█████▋ | 53/93 [00:24<00:18, 2.11it/s] 58%|█████▊ | 54/93 [00:24<00:18, 2.16it/s] 59%|█████▉ | 55/93 [00:25<00:16, 2.32it/s] 60%|██████ | 56/93 [00:25<00:15, 2.33it/s] 61%|██████▏ | 57/93 [00:26<00:15, 2.38it/s] 62%|██████▏ | 58/93 [00:26<00:15, 2.29it/s] 63%|██████▎ | 59/93 [00:26<00:14, 2.33it/s] 65%|██████▍ | 60/93 [00:27<00:13, 2.44it/s] 66%|██████▌ | 61/93 [00:27<00:13, 2.33it/s] 67%|██████▋ | 62/93 [00:28<00:13, 2.36it/s] 68%|██████▊ | 63/93 [00:28<00:12, 2.44it/s] 69%|██████▉ | 64/93 [00:28<00:11, 2.42it/s] 70%|██████▉ | 65/93 [00:29<00:11, 2.47it/s] 71%|███████ | 66/93 [00:29<00:11, 2.44it/s] 72%|███████▏ | 67/93 [00:30<00:10, 2.39it/s] 73%|███████▎ | 68/93 [00:30<00:10, 2.29it/s] 74%|███████▍ | 69/93 [00:31<00:10, 2.21it/s] 75%|███████▌ | 70/93 [00:31<00:10, 2.19it/s] 76%|███████▋ | 71/93 [00:32<00:10, 2.06it/s] 77%|███████▋ | 72/93 [00:32<00:09, 2.18it/s] 78%|███████▊ | 73/93 [00:32<00:08, 2.33it/s] 80%|███████▉ | 74/93 [00:33<00:08, 2.30it/s] 81%|████████ | 75/93 [00:33<00:08, 2.25it/s] 82%|████████▏ | 76/93 [00:34<00:07, 2.20it/s] 83%|████████▎ | 77/93 [00:34<00:07, 2.15it/s] 84%|████████▍ | 78/93 [00:35<00:07, 2.01it/s] 85%|████████▍ | 79/93 [00:35<00:07, 1.98it/s] 86%|████████▌ | 80/93 [00:36<00:06, 1.99it/s] 87%|████████▋ | 81/93 [00:36<00:05, 2.17it/s] 88%|████████▊ | 82/93 [00:37<00:04, 2.26it/s] 89%|████████▉ | 83/93 [00:37<00:04, 2.23it/s] 90%|█████████ | 84/93 [00:38<00:03, 2.32it/s] 91%|█████████▏| 85/93 [00:38<00:03, 2.34it/s] 92%|█████████▏| 86/93 [00:38<00:03, 2.24it/s] 94%|█████████▎| 87/93 [00:39<00:02, 2.25it/s] 95%|█████████▍| 88/93 [00:39<00:02, 2.29it/s] 96%|█████████▌| 89/93 [00:40<00:01, 2.29it/s] 97%|█████████▋| 90/93 [00:40<00:01, 2.30it/s] 98%|█████████▊| 91/93 [00:41<00:00, 2.11it/s] 99%|█████████▉| 92/93 [00:41<00:00, 2.14it/s] 100%|██████████| 93/93 [00:42<00:00, 2.31it/s] 100%|██████████| 93/93 [00:42<00:00, 2.20it/s]
***** predict_test_ru_RU metrics *****
predict_ex_match_acc = 0.6113
predict_ex_match_acc_stderr = 0.0089
predict_intent_acc = 0.8557
predict_intent_acc_stderr = 0.0064
predict_loss = 0.3015
predict_runtime = 0:00:42.75
predict_samples = 2974
predict_samples_per_second = 69.553
predict_slot_micro_f1 = 0.7306
predict_slot_micro_f1_stderr = 0.0034
predict_steps_per_second = 2.175
02/05/2024 18:55:12 - INFO - __main__ - *** test_tr_TR ***
[INFO|trainer.py:718] 2024-02-05 18:55:12,367 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:55:12,369 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:55:12,370 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:55:12,370 >> Batch size = 32
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:18, 4.93it/s] 3%|▎ | 3/93 [00:00<00:27, 3.31it/s] 4%|▍ | 4/93 [00:01<00:30, 2.93it/s] 5%|▌ | 5/93 [00:01<00:32, 2.74it/s] 6%|▋ | 6/93 [00:01<00:30, 2.84it/s] 8%|▊ | 7/93 [00:02<00:33, 2.57it/s] 9%|▊ | 8/93 [00:02<00:31, 2.67it/s] 10%|▉ | 9/93 [00:03<00:35, 2.35it/s] 11%|█ | 10/93 [00:03<00:34, 2.41it/s] 12%|█▏ | 11/93 [00:04<00:35, 2.32it/s] 13%|█▎ | 12/93 [00:04<00:33, 2.42it/s] 14%|█▍ | 13/93 [00:04<00:31, 2.50it/s] 15%|█▌ | 14/93 [00:05<00:32, 2.46it/s] 16%|█▌ | 15/93 [00:05<00:35, 2.23it/s] 17%|█▋ | 16/93 [00:06<00:38, 2.02it/s] 18%|█▊ | 17/93 [00:07<00:38, 1.95it/s] 19%|█▉ | 18/93 [00:07<00:38, 1.94it/s] 20%|██ | 19/93 [00:07<00:34, 2.16it/s] 22%|██▏ | 20/93 [00:08<00:31, 2.28it/s] 23%|██▎ | 21/93 [00:08<00:31, 2.31it/s] 24%|██▎ | 22/93 [00:09<00:34, 2.05it/s] 25%|██▍ | 23/93 [00:09<00:33, 2.10it/s] 26%|██▌ | 24/93 [00:10<00:30, 2.28it/s] 27%|██▋ | 25/93 [00:10<00:29, 2.32it/s] 28%|██▊ | 26/93 [00:10<00:27, 2.45it/s] 29%|██▉ | 27/93 [00:11<00:26, 2.49it/s] 30%|███ | 28/93 [00:13<00:58, 1.12it/s] 31%|███ | 29/93 [00:13<00:47, 1.34it/s] 32%|███▏ | 30/93 [00:14<00:40, 1.54it/s] 33%|███▎ | 31/93 [00:14<00:34, 1.81it/s] 34%|███▍ | 32/93 [00:14<00:31, 1.94it/s] 35%|███▌ | 33/93 [00:15<00:28, 2.08it/s] 37%|███▋ | 34/93 [00:15<00:25, 2.28it/s] 38%|███▊ | 35/93 [00:16<00:25, 2.26it/s] 39%|███▊ | 36/93 [00:16<00:24, 2.31it/s] 40%|███▉ | 37/93 [00:16<00:24, 2.27it/s] 41%|████ | 38/93 [00:17<00:25, 2.12it/s] 42%|████▏ | 39/93 [00:17<00:24, 2.17it/s] 43%|████▎ | 40/93 [00:18<00:23, 2.27it/s] 44%|████▍ | 41/93 [00:18<00:22, 2.34it/s] 45%|████▌ | 42/93 [00:19<00:21, 2.33it/s] 46%|████▌ | 43/93 [00:19<00:20, 2.40it/s] 47%|████▋ | 44/93 [00:20<00:20, 2.37it/s] 48%|████▊ | 45/93 [00:20<00:21, 2.27it/s] 49%|████▉ | 46/93 [00:21<00:22, 2.13it/s] 51%|█████ | 47/93 [00:21<00:19, 2.31it/s] 52%|█████▏ | 48/93 [00:21<00:19, 2.35it/s] 53%|█████▎ | 49/93 [00:22<00:18, 2.38it/s] 54%|█████▍ | 50/93 [00:22<00:17, 2.51it/s] 55%|█████▍ | 51/93 [00:22<00:17, 2.44it/s] 56%|█████▌ | 52/93 [00:23<00:18, 2.18it/s] 57%|█████▋ | 53/93 [00:24<00:18, 2.18it/s] 58%|█████▊ | 54/93 [00:24<00:17, 2.26it/s] 59%|█████▉ | 55/93 [00:24<00:15, 2.40it/s] 60%|██████ | 56/93 [00:25<00:14, 2.50it/s] 61%|██████▏ | 57/93 [00:25<00:14, 2.47it/s] 62%|██████▏ | 58/93 [00:25<00:13, 2.52it/s] 63%|██████▎ | 59/93 [00:26<00:12, 2.64it/s] 65%|██████▍ | 60/93 [00:26<00:13, 2.47it/s] 66%|██████▌ | 61/93 [00:27<00:14, 2.18it/s] 67%|██████▋ | 62/93 [00:27<00:13, 2.25it/s] 68%|██████▊ | 63/93 [00:28<00:12, 2.36it/s] 69%|██████▉ | 64/93 [00:28<00:12, 2.39it/s] 70%|██████▉ | 65/93 [00:28<00:11, 2.44it/s] 71%|███████ | 66/93 [00:29<00:11, 2.42it/s] 72%|███████▏ | 67/93 [00:29<00:10, 2.44it/s] 73%|███████▎ | 68/93 [00:30<00:10, 2.30it/s] 74%|███████▍ | 69/93 [00:30<00:11, 2.17it/s] 75%|███████▌ | 70/93 [00:31<00:10, 2.24it/s] 76%|███████▋ | 71/93 [00:31<00:10, 2.02it/s] 77%|███████▋ | 72/93 [00:32<00:09, 2.18it/s] 78%|███████▊ | 73/93 [00:32<00:08, 2.26it/s] 80%|███████▉ | 74/93 [00:32<00:08, 2.25it/s] 81%|████████ | 75/93 [00:33<00:08, 2.21it/s] 82%|████████▏ | 76/93 [00:33<00:07, 2.23it/s] 83%|████████▎ | 77/93 [00:34<00:06, 2.37it/s] 84%|████████▍ | 78/93 [00:34<00:06, 2.46it/s] 85%|████████▍ | 79/93 [00:35<00:05, 2.37it/s] 86%|████████▌ | 80/93 [00:35<00:05, 2.24it/s] 87%|████████▋ | 81/93 [00:35<00:05, 2.35it/s] 88%|████████▊ | 82/93 [00:36<00:04, 2.35it/s] 89%|████████▉ | 83/93 [00:36<00:04, 2.21it/s] 90%|█████████ | 84/93 [00:37<00:03, 2.26it/s] 91%|█████████▏| 85/93 [00:37<00:03, 2.31it/s] 92%|█████████▏| 86/93 [00:38<00:02, 2.34it/s] 94%|█████████▎| 87/93 [00:38<00:02, 2.44it/s] 95%|█████████▍| 88/93 [00:38<00:02, 2.48it/s] 96%|█████████▌| 89/93 [00:39<00:01, 2.35it/s] 97%|█████████▋| 90/93 [00:39<00:01, 2.36it/s] 98%|█████████▊| 91/93 [00:40<00:00, 2.23it/s] 99%|█████████▉| 92/93 [00:40<00:00, 2.23it/s] 100%|██████████| 93/93 [00:41<00:00, 2.43it/s] 100%|██████████| 93/93 [00:41<00:00, 2.25it/s]
***** predict_test_tr_TR metrics *****
predict_ex_match_acc = 0.4445
predict_ex_match_acc_stderr = 0.0091
predict_intent_acc = 0.7579
predict_intent_acc_stderr = 0.0079
predict_loss = 0.614
predict_runtime = 0:00:41.70
predict_samples = 2974
predict_samples_per_second = 71.31
predict_slot_micro_f1 = 0.6036
predict_slot_micro_f1_stderr = 0.0038
predict_steps_per_second = 2.23
02/05/2024 18:55:54 - INFO - __main__ - *** test_vi_VN ***
[INFO|trainer.py:718] 2024-02-05 18:55:54,298 >> The following columns in the test set don't have a corresponding argument in `MT5ForConditionalGeneration.forward` and have been ignored: intent_str, id, annot_utt. If intent_str, id, annot_utt are not expected by `MT5ForConditionalGeneration.forward`, you can safely ignore this message.
[INFO|trainer.py:3199] 2024-02-05 18:55:54,300 >> ***** Running Prediction *****
[INFO|trainer.py:3201] 2024-02-05 18:55:54,300 >> Num examples = 2974
[INFO|trainer.py:3204] 2024-02-05 18:55:54,301 >> Batch size = 32
0%| | 0/93 [00:00<?, ?it/s] 2%|▏ | 2/93 [00:00<00:21, 4.28it/s] 3%|▎ | 3/93 [00:00<00:27, 3.22it/s] 4%|▍ | 4/93 [00:01<00:36, 2.45it/s] 5%|▌ | 5/93 [00:02<00:40, 2.15it/s] 6%|▋ | 6/93 [00:02<00:40, 2.12it/s] 8%|▊ | 7/93 [00:03<00:42, 2.03it/s] 9%|▊ | 8/93 [00:03<00:43, 1.98it/s] 10%|▉ | 9/93 [00:04<00:40, 2.08it/s] 11%|█ | 10/93 [00:04<00:43, 1.90it/s] 12%|█▏ | 11/93 [00:05<00:43, 1.89it/s] 13%|█▎ | 12/93 [00:05<00:43, 1.84it/s] 14%|█▍ | 13/93 [00:06<00:46, 1.71it/s] 15%|█▌ | 14/93 [00:07<00:45, 1.72it/s] 16%|█▌ | 15/93 [00:07<00:48, 1.59it/s] 17%|█▋ | 16/93 [00:08<00:51, 1.49it/s] 18%|█▊ | 17/93 [00:09<00:49, 1.53it/s] 19%|█▉ | 18/93 [00:09<00:53, 1.41it/s] 20%|██ | 19/93 [00:10<00:48, 1.52it/s] 22%|██▏ | 20/93 [00:10<00:44, 1.64it/s] 23%|██▎ | 21/93 [00:11<00:42, 1.70it/s] 24%|██▎ | 22/93 [00:12<00:43, 1.62it/s] 25%|██▍ | 23/93 [00:12<00:42, 1.65it/s] 26%|██▌ | 24/93 [00:13<00:39, 1.74it/s] 27%|██▋ | 25/93 [00:13<00:40, 1.68it/s] 28%|██▊ | 26/93 [00:14<00:38, 1.76it/s] 29%|██▉ | 27/93 [00:15<00:44, 1.47it/s] 30%|███ | 28/93 [00:17<01:12, 1.12s/it] 31%|███ | 29/93 [00:17<00:58, 1.10it/s] 32%|███▏ | 30/93 [00:18<00:54, 1.16it/s] 33%|███▎ | 31/93 [00:19<00:46, 1.32it/s] 34%|███▍ | 32/93 [00:19<00:43, 1.41it/s] 35%|███▌ | 33/93 [00:20<00:40, 1.49it/s] 37%|███▋ | 34/93 [00:20<00:37, 1.57it/s] 38%|███▊ | 35/93 [00:21<00:35, 1.61it/s] 39%|███▊ | 36/93 [00:22<00:38, 1.46it/s] 40%|███▉ | 37/93 [00:22<00:36, 1.54it/s] 41%|████ | 38/93 [00:23<00:35, 1.55it/s] 42%|████▏ | 39/93 [00:25<00:57, 1.06s/it] 43%|████▎ | 40/93 [00:26<00:49, 1.07it/s] 44%|████▍ | 41/93 [00:26<00:42, 1.21it/s] 45%|████▌ | 42/93 [00:27<00:37, 1.34it/s] 46%|████▌ | 43/93 [00:27<00:34, 1.44it/s] 47%|████▋ | 44/93 [00:28<00:30, 1.63it/s] 48%|████▊ | 45/93 [00:29<00:30, 1.58it/s] 49%|████▉ | 46/93 [00:29<00:30, 1.54it/s] 51%|█████ | 47/93 [00:30<00:33, 1.39it/s] 52%|█████▏ | 48/93 [00:31<00:33, 1.32it/s] 53%|█████▎ | 49/93 [00:31<00:29, 1.49it/s] 54%|█████▍ | 50/93 [00:32<00:26, 1.62it/s] 55%|█████▍ | 51/93 [00:34<00:44, 1.05s/it] 56%|█████▌ | 52/93 [00:35<00:39, 1.05it/s] 57%|█████▋ | 53/93 [00:35<00:32, 1.22it/s] 58%|█████▊ | 54/93 [00:36<00:29, 1.33it/s] 59%|█████▉ | 55/93 [00:36<00:26, 1.45it/s] 60%|██████ | 56/93 [00:37<00:22, 1.63it/s] 61%|██████▏ | 57/93 [00:37<00:20, 1.73it/s] 62%|██████▏ | 58/93 [00:38<00:21, 1.66it/s] 63%|██████▎ | 59/93 [00:39<00:22, 1.54it/s] 65%|██████▍ | 60/93 [00:39<00:19, 1.70it/s] 66%|██████▌ | 61/93 [00:40<00:19, 1.62it/s] 67%|██████▋ | 62/93 [00:40<00:19, 1.62it/s] 68%|██████▊ | 63/93 [00:41<00:20, 1.48it/s] 69%|██████▉ | 64/93 [00:42<00:17, 1.62it/s] 70%|██████▉ | 65/93 [00:42<00:17, 1.64it/s] 71%|███████ | 66/93 [00:43<00:17, 1.57it/s] 72%|███████▏ | 67/93 [00:44<00:15, 1.63it/s] 73%|███████▎ | 68/93 [00:44<00:14, 1.74it/s] 74%|███████▍ | 69/93 [00:45<00:14, 1.68it/s] 75%|███████▌ | 70/93 [00:45<00:12, 1.78it/s] 76%|███████▋ | 71/93 [00:47<00:21, 1.00it/s] 77%|███████▋ | 72/93 [00:48<00:18, 1.16it/s] 78%|███████▊ | 73/93 [00:48<00:15, 1.31it/s] 80%|███████▉ | 74/93 [00:49<00:13, 1.43it/s] 81%|████████ | 75/93 [00:51<00:19, 1.09s/it] 82%|████████▏ | 76/93 [00:51<00:15, 1.09it/s] 83%|████████▎ | 77/93 [00:52<00:13, 1.21it/s] 84%|████████▍ | 78/93 [00:53<00:11, 1.34it/s] 85%|████████▍ | 79/93 [00:53<00:09, 1.45it/s] 86%|████████▌ | 80/93 [00:54<00:08, 1.51it/s] 87%|████████▋ | 81/93 [00:54<00:07, 1.55it/s] 88%|████████▊ | 82/93 [00:55<00:06, 1.66it/s] 89%|████████▉ | 83/93 [00:55<00:05, 1.71it/s] 90%|█████████ | 84/93 [00:56<00:04, 1.81it/s] 91%|█████████▏| 85/93 [00:57<00:04, 1.65it/s] 92%|█████████▏| 86/93 [00:57<00:04, 1.62it/s] 94%|█████████▎| 87/93 [00:58<00:03, 1.72it/s] 95%|█████████▍| 88/93 [00:58<00:02, 1.81it/s] 96%|█████████▌| 89/93 [00:59<00:02, 1.80it/s] 97%|█████████▋| 90/93 [00:59<00:01, 1.83it/s] 98%|█████████▊| 91/93 [01:00<00:01, 1.70it/s] 99%|█████████▉| 92/93 [01:00<00:00, 1.76it/s] 100%|██████████| 93/93 [01:01<00:00, 1.79it/s] 100%|██████████| 93/93 [01:01<00:00, 1.50it/s]
***** predict_test_vi_VN metrics *****
predict_ex_match_acc = 0.1416
predict_ex_match_acc_stderr = 0.0064
predict_intent_acc = 0.3584
predict_intent_acc_stderr = 0.0088
predict_loss = 0.6596
predict_runtime = 0:01:02.37
predict_samples = 2974
predict_samples_per_second = 47.683
predict_slot_micro_f1 = 0.3368
predict_slot_micro_f1_stderr = 0.0029
predict_steps_per_second = 1.491