Hi, I try to load with LlamaForCausalLM, LlamaTokenizer, but it show me the error that "not a string"
#64
by
hjewr
- opened
Thank u. Yeah, it works. But I wonder what was the reason for that? the sentencepice problem?
YEs, the reason is because as we mention on the card, the tokenizer is NOT sentencepiece based
The tokenizer.model
is from tiktoken
. We converted it to the tokenizers
format, so AutoTokenizer will use a PreTrainedTokenizerFast
Closing as solved!
ArthurZ
changed discussion status to
closed