FileNotFoundError: Could not find model in TheBloke/guanaco-33B-GPTQ
#12
by
ryzzlestrizzle
- opened
Hello! Thank you for your great work. I'm running into this dumb error when trying to load the model using auto_gptq - the code is copied from the "TheBloke/Nous-Hermes-13B-GPTQ"-repo. Any ideas on how to resolve it?
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
model_name_or_path = "TheBloke/guanaco-33B-GPTQ"
model_basename = "Guanaco-33B-GPTQ-4bit.act-order.safetensors"
use_triton = False
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
model_basename=model_basename,
use_safetensors=True,
trust_remote_code=True,
device="cuda:0",
use_triton=use_triton,
quantize_config=None)
It's because model_basename shouldn't include the .safetensors extension. It should be:
model_basename = "Guanaco-33B-GPTQ-4bit.act-order"
It's confusing I know - I've never liked how it's implemented. Should just be able to specify the whole filename.
Works now, thank you!
ryzzlestrizzle
changed discussion status to
closed