Hi, I try to load with LlamaForCausalLM, LlamaTokenizer, but it show me the error that "not a string"

#64
by hjewr - opened

Hi, I try to load with LlamaForCausalLM, LlamaTokenizer, but it show me the error that "not a string".

It this because the naming problem? maybe tokenizer.json to tokenizer.model

image.png

image.png

Thank u. Yeah, it works. But I wonder what was the reason for that? the sentencepice problem?

Meta Llama org

YEs, the reason is because as we mention on the card, the tokenizer is NOT sentencepiece based

Meta Llama org

The tokenizer.model is from tiktoken. We converted it to the tokenizers format, so AutoTokenizer will use a PreTrainedTokenizerFast

Meta Llama org

Closing as solved!

ArthurZ changed discussion status to closed
Meta Llama org

Thanks @hjewr

Hey, thank u. @ArthurZ . It's kind and nice answer.

Sign up or log in to comment