Will it run on a 4090?
#1
by
brifl
- opened
Unfortunately, I had errors trying to run this on a home PC with a GeForce 4090 RTX. I was using the latest oobabooga. I'm sharing my success or failure of running these models larger than 13b locally to save some people time, disk, and bandwidth.
I didn't troubleshoot other than trying the AWQ and GPTQ versions. Both failed. The error is regarding missing config.json.
What is your model loader? AutoAWQ and AutoGPTQ? You may want to try vllm for AWQ and exllamav2 for GPTQ