INFO: 2024-10-14 17:12:15,109: llmtf.base.evaluator: Starting eval on ['darumeru/multiq', 'darumeru/parus', 'darumeru/rcb', 'darumeru/ruopenbookqa', 'darumeru/ruworldtree', 'darumeru/rwsd', 'darumeru/use', 'russiannlp/rucola_custom'] INFO: 2024-10-14 17:12:15,109: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:12:15,109: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:12:15,157: llmtf.base.evaluator: Starting eval on ['darumeru/rummlu'] INFO: 2024-10-14 17:12:15,157: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:12:15,157: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:12:16,297: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] INFO: 2024-10-14 17:12:16,297: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:12:16,297: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:12:18,005: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] INFO: 2024-10-14 17:12:18,006: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:12:18,006: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:12:19,753: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] INFO: 2024-10-14 17:12:19,753: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:12:19,753: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:12:21,769: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] INFO: 2024-10-14 17:12:21,769: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:12:21,769: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:12:23,453: llmtf.base.evaluator: Starting eval on ['darumeru/cp_sent_ru', 'darumeru/cp_sent_en', 'darumeru/cp_para_ru', 'darumeru/cp_para_en'] INFO: 2024-10-14 17:12:23,453: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:12:23,453: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:12:29,425: llmtf.base.darumeru/cp_sent_ru: Loading Dataset: 5.97s INFO: 2024-10-14 17:12:39,535: llmtf.base.daru/treewayextractive: Loading Dataset: 17.77s INFO: 2024-10-14 17:12:39,537: llmtf.base.daru/treewayextractive: Processing Dataset: 0.00s ERROR: 2024-10-14 17:12:39,537: llmtf.base.evaluator: return_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast. ERROR: 2024-10-14 17:12:39,542: llmtf.base.evaluator: Traceback (most recent call last): File "/scratch/tikhomirov/workdir/projects/llmtf_open/llmtf/evaluator.py", line 42, in evaluate self.evaluate_dataset(task, model, output_dir, prompt_max_len, few_shot_count, generation_config, batch_size, max_sample_per_dataset) File "/scratch/tikhomirov/workdir/projects/llmtf_open/llmtf/evaluator.py", line 65, in evaluate_dataset prompts, y_preds, infos = getattr(model, task.method + '_batch')(**messages_batch) File "/scratch/tikhomirov/workdir/projects/llmtf_open/llmtf/model.py", line 417, in calculate_logsoftmax_batch data = self.tokenizer( File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 2829, in __call__ encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 2915, in _call_one return self.batch_encode_plus( File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 3106, in batch_encode_plus return self._batch_encode_plus( File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py", line 788, in _batch_encode_plus raise NotImplementedError( NotImplementedError: return_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast. INFO: 2024-10-14 17:12:48,032: llmtf.base.daru/treewayabstractive: Loading Dataset: 28.28s INFO: 2024-10-14 17:12:51,035: llmtf.base.darumeru/MultiQ: Loading Dataset: 35.93s INFO: 2024-10-14 17:14:25,981: llmtf.base.darumeru/ruMMLU: Loading Dataset: 130.82s INFO: 2024-10-14 17:16:23,865: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 245.86s INFO: 2024-10-14 17:16:52,771: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 276.47s INFO: 2024-10-14 17:17:20,199: llmtf.base.darumeru/MultiQ: Processing Dataset: 269.16s INFO: 2024-10-14 17:17:20,200: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: INFO: 2024-10-14 17:17:20,204: llmtf.base.darumeru/MultiQ: {'f1': 0.48670297858620026, 'em': 0.372848948374761} INFO: 2024-10-14 17:17:20,215: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:17:20,216: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:17:23,562: llmtf.base.darumeru/PARus: Loading Dataset: 3.35s INFO: 2024-10-14 17:17:30,091: llmtf.base.darumeru/PARus: Processing Dataset: 6.53s INFO: 2024-10-14 17:17:30,093: llmtf.base.darumeru/PARus: Results for darumeru/PARus: INFO: 2024-10-14 17:17:30,106: llmtf.base.darumeru/PARus: {'acc': 0.72} INFO: 2024-10-14 17:17:30,108: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:17:30,108: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:17:34,240: llmtf.base.darumeru/RCB: Loading Dataset: 4.13s INFO: 2024-10-14 17:17:45,351: llmtf.base.darumeru/RCB: Processing Dataset: 11.11s INFO: 2024-10-14 17:17:45,353: llmtf.base.darumeru/RCB: Results for darumeru/RCB: INFO: 2024-10-14 17:17:45,362: llmtf.base.darumeru/RCB: {'acc': 0.55, 'f1_macro': 0.4643770525787481} INFO: 2024-10-14 17:17:45,365: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:17:45,366: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:18:02,200: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 16.83s INFO: 2024-10-14 17:19:12,133: llmtf.base.darumeru/cp_sent_ru: Processing Dataset: 402.71s INFO: 2024-10-14 17:19:12,136: llmtf.base.darumeru/cp_sent_ru: Results for darumeru/cp_sent_ru: INFO: 2024-10-14 17:19:12,140: llmtf.base.darumeru/cp_sent_ru: {'symbol_per_token': 3.958846363652826, 'len': 0.9810206403044489, 'lcs': 0.9815195071868583} INFO: 2024-10-14 17:19:12,144: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:19:12,144: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:19:16,530: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 74.33s INFO: 2024-10-14 17:19:16,531: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: INFO: 2024-10-14 17:19:16,545: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.7371134020618557, 'f1_macro': 0.7358424630320096} INFO: 2024-10-14 17:19:16,562: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:19:16,562: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:19:16,859: llmtf.base.darumeru/cp_sent_en: Loading Dataset: 4.71s INFO: 2024-10-14 17:19:19,074: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 2.51s INFO: 2024-10-14 17:19:23,376: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 4.30s INFO: 2024-10-14 17:19:23,378: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: INFO: 2024-10-14 17:19:23,383: llmtf.base.darumeru/ruWorldTree: {'acc': 0.8666666666666667, 'f1_macro': 0.8654128672745693} INFO: 2024-10-14 17:19:23,384: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:19:23,384: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:19:27,363: llmtf.base.darumeru/RWSD: Loading Dataset: 3.98s INFO: 2024-10-14 17:19:37,387: llmtf.base.darumeru/RWSD: Processing Dataset: 10.02s INFO: 2024-10-14 17:19:37,405: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: INFO: 2024-10-14 17:19:37,409: llmtf.base.darumeru/RWSD: {'acc': 0.5735294117647058} INFO: 2024-10-14 17:19:37,411: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:19:37,411: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:19:54,737: llmtf.base.darumeru/USE: Loading Dataset: 17.32s INFO: 2024-10-14 17:25:32,317: llmtf.base.darumeru/USE: Processing Dataset: 337.58s INFO: 2024-10-14 17:25:32,321: llmtf.base.darumeru/USE: Results for darumeru/USE: INFO: 2024-10-14 17:25:32,326: llmtf.base.darumeru/USE: {'grade_norm': 0.06176470588235293} INFO: 2024-10-14 17:25:32,333: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:25:32,334: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:25:48,469: llmtf.base.darumeru/ruMMLU: Processing Dataset: 682.49s INFO: 2024-10-14 17:25:48,470: llmtf.base.darumeru/ruMMLU: Results for darumeru/ruMMLU: INFO: 2024-10-14 17:25:48,496: llmtf.base.darumeru/ruMMLU: {'acc': 0.5159133991818817} INFO: 2024-10-14 17:25:48,576: llmtf.base.evaluator: Ended eval INFO: 2024-10-14 17:25:48,608: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruWorldTree 0.599 0.430 0.720 0.507 0.574 0.062 0.981 0.516 0.736 0.866 INFO: 2024-10-14 17:25:55,560: llmtf.base.russiannlp/rucola_custom: Loading Dataset: 23.23s INFO: 2024-10-14 17:26:02,923: llmtf.base.darumeru/cp_sent_en: Processing Dataset: 406.06s INFO: 2024-10-14 17:26:02,926: llmtf.base.darumeru/cp_sent_en: Results for darumeru/cp_sent_en: INFO: 2024-10-14 17:26:02,930: llmtf.base.darumeru/cp_sent_en: {'symbol_per_token': 4.416472661458526, 'len': 0.9634384484255067, 'lcs': 1.0} INFO: 2024-10-14 17:26:02,932: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:26:02,932: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:26:07,243: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 4.31s INFO: 2024-10-14 17:27:27,303: llmtf.base.russiannlp/rucola_custom: Processing Dataset: 91.74s INFO: 2024-10-14 17:27:27,307: llmtf.base.russiannlp/rucola_custom: Results for russiannlp/rucola_custom: INFO: 2024-10-14 17:27:27,320: llmtf.base.russiannlp/rucola_custom: {'acc': 0.5999282382490133, 'mcc': 0.26826918795125926} INFO: 2024-10-14 17:27:27,332: llmtf.base.evaluator: Ended eval INFO: 2024-10-14 17:27:27,344: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruWorldTree russiannlp/rucola_custom 0.617 0.430 0.720 0.507 0.574 0.062 0.963 0.981 0.516 0.736 0.866 0.434 INFO: 2024-10-14 17:28:31,573: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 727.71s INFO: 2024-10-14 17:28:31,575: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: INFO: 2024-10-14 17:28:31,622: llmtf.base.nlpcoreteam/enMMLU: metric subject abstract_algebra 0.380000 anatomy 0.592593 astronomy 0.736842 business_ethics 0.730000 clinical_knowledge 0.716981 college_biology 0.770833 college_chemistry 0.500000 college_computer_science 0.590000 college_mathematics 0.420000 college_medicine 0.676301 college_physics 0.509804 computer_security 0.810000 conceptual_physics 0.642553 econometrics 0.552632 electrical_engineering 0.675862 elementary_mathematics 0.603175 formal_logic 0.349206 global_facts 0.310000 high_school_biology 0.816129 high_school_chemistry 0.610837 high_school_computer_science 0.750000 high_school_european_history 0.787879 high_school_geography 0.808081 high_school_government_and_politics 0.870466 high_school_macroeconomics 0.712821 high_school_mathematics 0.518519 high_school_microeconomics 0.768908 high_school_physics 0.443709 high_school_psychology 0.860550 high_school_statistics 0.666667 high_school_us_history 0.852941 high_school_world_history 0.843882 human_aging 0.721973 human_sexuality 0.793893 international_law 0.785124 jurisprudence 0.787037 logical_fallacies 0.766871 machine_learning 0.473214 management 0.834951 marketing 0.888889 medical_genetics 0.790000 miscellaneous 0.793103 moral_disputes 0.687861 moral_scenarios 0.288268 nutrition 0.751634 philosophy 0.717042 prehistory 0.731481 professional_accounting 0.524823 professional_law 0.477184 professional_medicine 0.661765 professional_psychology 0.696078 public_relations 0.745455 security_studies 0.759184 sociology 0.830846 us_foreign_policy 0.850000 virology 0.493976 world_religions 0.807018 INFO: 2024-10-14 17:28:31,630: llmtf.base.nlpcoreteam/enMMLU: metric subject STEM 0.606564 humanities 0.683215 other (business, health, misc.) 0.677642 social sciences 0.770743 INFO: 2024-10-14 17:28:31,637: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.6845408165512243} INFO: 2024-10-14 17:28:31,706: llmtf.base.evaluator: Ended eval INFO: 2024-10-14 17:28:31,716: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU russiannlp/rucola_custom 0.623 0.430 0.720 0.507 0.574 0.062 0.963 0.981 0.516 0.736 0.866 0.685 0.434 INFO: 2024-10-14 17:30:17,754: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 804.98s INFO: 2024-10-14 17:30:17,758: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: INFO: 2024-10-14 17:30:17,804: llmtf.base.nlpcoreteam/ruMMLU: metric subject abstract_algebra 0.340000 anatomy 0.385185 astronomy 0.651316 business_ethics 0.630000 clinical_knowledge 0.558491 college_biology 0.527778 college_chemistry 0.420000 college_computer_science 0.560000 college_mathematics 0.360000 college_medicine 0.549133 college_physics 0.450980 computer_security 0.670000 conceptual_physics 0.582979 econometrics 0.447368 electrical_engineering 0.572414 elementary_mathematics 0.529101 formal_logic 0.357143 global_facts 0.320000 high_school_biology 0.696774 high_school_chemistry 0.522167 high_school_computer_science 0.700000 high_school_european_history 0.715152 high_school_geography 0.757576 high_school_government_and_politics 0.595855 high_school_macroeconomics 0.574359 high_school_mathematics 0.444444 high_school_microeconomics 0.563025 high_school_physics 0.384106 high_school_psychology 0.693578 high_school_statistics 0.518519 high_school_us_history 0.642157 high_school_world_history 0.687764 human_aging 0.582960 human_sexuality 0.625954 international_law 0.735537 jurisprudence 0.611111 logical_fallacies 0.588957 machine_learning 0.401786 management 0.669903 marketing 0.773504 medical_genetics 0.560000 miscellaneous 0.604087 moral_disputes 0.630058 moral_scenarios 0.226816 nutrition 0.663399 philosophy 0.639871 prehistory 0.577160 professional_accounting 0.361702 professional_law 0.371578 professional_medicine 0.470588 professional_psychology 0.539216 public_relations 0.545455 security_studies 0.657143 sociology 0.726368 us_foreign_policy 0.770000 virology 0.463855 world_religions 0.678363 INFO: 2024-10-14 17:30:17,812: llmtf.base.nlpcoreteam/ruMMLU: metric subject STEM 0.518465 humanities 0.573974 other (business, health, misc.) 0.542343 social sciences 0.624658 INFO: 2024-10-14 17:30:17,835: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.5648600813341976} INFO: 2024-10-14 17:30:17,918: llmtf.base.evaluator: Ended eval INFO: 2024-10-14 17:30:17,931: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU russiannlp/rucola_custom 0.618 0.430 0.720 0.507 0.574 0.062 0.963 0.981 0.516 0.736 0.866 0.685 0.565 0.434 INFO: 2024-10-14 17:35:49,735: llmtf.base.daru/treewayabstractive: Processing Dataset: 1381.70s INFO: 2024-10-14 17:35:49,744: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: INFO: 2024-10-14 17:35:49,749: llmtf.base.daru/treewayabstractive: {'rouge1': 0.3333904808228802, 'rouge2': 0.127931429649173} INFO: 2024-10-14 17:35:49,754: llmtf.base.evaluator: Ended eval INFO: 2024-10-14 17:35:49,768: llmtf.base.evaluator: mean daru/treewayabstractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU russiannlp/rucola_custom 0.591 0.231 0.430 0.720 0.507 0.574 0.062 0.963 0.981 0.516 0.736 0.866 0.685 0.565 0.434 INFO: 2024-10-14 17:36:46,624: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 639.38s INFO: 2024-10-14 17:36:46,627: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: INFO: 2024-10-14 17:36:46,630: llmtf.base.darumeru/cp_para_ru: {'symbol_per_token': 3.9840096010204995, 'len': 0.9812951313228986, 'lcs': 0.98} INFO: 2024-10-14 17:36:46,632: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147075, 198, 271] INFO: 2024-10-14 17:36:46,632: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] INFO: 2024-10-14 17:36:50,656: llmtf.base.darumeru/cp_para_en: Loading Dataset: 4.02s INFO: 2024-10-14 17:52:40,643: llmtf.base.darumeru/cp_para_en: Processing Dataset: 949.99s INFO: 2024-10-14 17:52:40,649: llmtf.base.darumeru/cp_para_en: Results for darumeru/cp_para_en: INFO: 2024-10-14 17:52:40,654: llmtf.base.darumeru/cp_para_en: {'symbol_per_token': 4.4342745144269164, 'len': 0.9926602787834059, 'lcs': 1.0} INFO: 2024-10-14 17:52:40,654: llmtf.base.evaluator: Ended eval INFO: 2024-10-14 17:52:40,665: llmtf.base.evaluator: mean daru/treewayabstractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU russiannlp/rucola_custom 0.641 0.231 0.430 0.720 0.507 0.574 0.062 1.000 0.980 0.963 0.981 0.516 0.736 0.866 0.685 0.565 0.434