Add transformers + AWQ inference support
#1
by
ybelkada
- opened
This PR adds the transformers support for AWQ models. If you have AWQ kernels installed through autoawq package or llm-awq you can load this model directly through AutoModelForCausalLM.from_pretrained out of the box.
You can read more about the integration here: https://huggingface.co/docs/transformers/main_classes/quantization#awq-integration or this google colab demo: https://colab.research.google.com/drive/1HzZH89yAXJaZgwJDhQj9LqSBux932BvY (we'll announce it early next week)
ybelkada
changed pull request title from
Update config.json
to Add transformers + AWQ inference support
Pclanglais
changed pull request status to
merged