Missing Tokenizer.json

#5
by tjake - opened

I manually used the one from non-chat model but should be here too

01-ai org

Hi πŸ‘‹@tjake, all the files for our base model are here: https://huggingface.co/01-ai/Yi-Coder-1.5B/tree/main

Yes that's what I used but my project depends on the tokenizer.json being checked in with each model.

Is there a reason you left it out? Would you mind adding it to the chat models?

01-ai org

Hi πŸ‘‹ Hello, the base model includes tokenizer.json for debugging convenience or special adjustments, while the chat model may only need tokenizer.model to improve loading and running efficiency. However, all 1.5B tokenizer.json files are the same. If your project needs it, you can download the 1.5B base tokenizer.json and add it to your project.

tjake changed discussion status to closed

Sign up or log in to comment