The quant doesn't load in vLLM / Aphrodite Engine

by av-codes - opened 5 days ago

5 days ago

Unable to run this in either vLLM or Aphrodite. vLLM silently fails with a missing response from RPC engine, Aphrodite get stuck at aqlm_dequant. I assume both are silent errors in the underlying quantization library.

The 3.1 8B 1x16 loads in the same setup with vLLM, after fixing tokenizer config (correct EOS token + add missing chat template).

gopi87

3 days ago

@av-codes can i use this type off quant in windows using vllm ?

av-codes

3 days ago

@gopi87 this report is about the fact that it doesn't load. I didn't do any tests on windows

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment