Error with using AutoTokenizer to load
#89
by
thuzhizhi
- opened
I try to download the whole repo and try to use the files offline. Unfortunately I encountered the following question.
I use code
model_path = "/home/test/test06/jdz/PLMs/mistral-7b-it"
print(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=True)
print("load tokenizer success")
And I got the following report:
Traceback (most recent call last):
File "/home/test/test06/jdz/edgeLLM/fastercode/collect/base_run.py", line 125, in <module>
tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=True)
File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 907, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2208, in from_pretrained
return cls._from_pretrained(
File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2442, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 157, in __init__
super().__init__(
File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 132, in __init__
slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs)
File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 171, in __init__
self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False))
File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 201, in get_spm_processor
with open(self.vocab_file, "rb") as f:
TypeError: expected str, bytes or os.PathLike object, not NoneType
Can anyone help me pls?
thuzhizhi
changed discussion status to
closed