Updated tokenizer
#1
by
Dampfinchen
- opened
Hello,
the Mistral tokenizer has seen multiple changes over the last couple of days. An update would be nice!
https://huggingface.co/mistralai/Mistral-7B-v0.1/commits/main
Edit: looks like you've set the correct values already.
Dampfinchen
changed discussion status to
closed