Spaces:
Running
on
CPU Upgrade
latest commit breaks ability to submit mistral finetunes
this commit https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/commit/ead4c9640f6462903d8fb9579b6cb5f7ad0a71f7 seems to break the ability to submit mistral finetunes. The tokenizer that tried is the LlamaTokenizer which is pretty standard, but I get the error uses a tokenizer which is not in a transformers release
https://huggingface.co/openaccess-ai-collective/dpopenhermes-v1/blob/main/tokenizer_config.json#L56
Hi!
Thank you for your report.
Can you please provide the full error message that you got?
Here you go. I've verified the tokenizer loads properly with the autotokenizer
Super useful thank you! We were simply missing sentencepiece as a dependency in the front end, I added it - it should be good now, can you give it another go?
thanks for the fix!