RuntimeError
#1
by
mhollomey
- opened
It seems to be incompatible with vLLM. I get this error even though max_model_len=4096 : "RuntimeError: start (0) + length (1280) exceeds dimension size (1024)." Is there any solution or work around?
Hey
@mhollomey
for now you have to install vllm from source instead of using pip to use this model.
But i think you can use it with the pip vllm package natively in the next days :)
Thank you very much I will install it from source!
Yes this is the common error because of the the new model architecture.
But it already works with vllm install from source.
Have fun!
DavidGF
changed discussion status to
closed