Cannot reproduce results: evaluation tasks not found
#15
by
Detsutut
- opened
The lm-eval throws this error when trying to replicate the eval pipeline following the instructions in the "About" tab:
ValueError: Tasks m_mmlu_it were not found. Try `lm-eval --tasks list` for list of available tasks
same thing for hellaswag_it and arc_itdatasets.
Should we create new evaluation tasks to replicate the pipeline? If that's the case, could you provide the YAML configurations and utils for each dataset?
Hi no the tasks are already inside (for example here: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/okapi/mmlu_multilingual)
Try to paste the commands in one line; sometimes I don't, and it gives me the same error!
weird, but worked! Thanks
Detsutut
changed discussion status to
closed