Error with using AutoTokenizer to load

#89
by thuzhizhi - opened

I try to download the whole repo and try to use the files offline. Unfortunately I encountered the following question.

I use code

    model_path = "/home/test/test06/jdz/PLMs/mistral-7b-it"
    print(model_path)
    tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=True)
    print("load tokenizer success")

And I got the following report:

Traceback (most recent call last):
  File "/home/test/test06/jdz/edgeLLM/fastercode/collect/base_run.py", line 125, in <module>
    tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=True)
  File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 907, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2208, in from_pretrained
    return cls._from_pretrained(
  File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2442, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 157, in __init__
    super().__init__(
  File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 132, in __init__
    slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs)
  File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 171, in __init__
    self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False))
  File "/home/test/test06/miniconda3/envs/llama/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 201, in get_spm_processor
    with open(self.vocab_file, "rb") as f:
TypeError: expected str, bytes or os.PathLike object, not NoneType

Can anyone help me pls?

thuzhizhi changed discussion status to closed

Sign up or log in to comment