John6666

/

llama-joycaption-alpha-two-hf-llava-nf4

like 6

Basically, you can use the same procedure as the official JoyCaption below.
Only bitsandbytes may need to be installed separately.
https://github.com/fpgaminer/joycaption
https://github.com/fpgaminer/joycaption/tree/main/scripts

pip install bitsandbytes

@John6666 have you succesfully used NF4 ? I've tried hacking it in multiple ways but I always get a mat1 / mat2 exception or similar unmatching tensor stuff

Same here.😭
Loading to VRAM in NF4 quantization state gives an error.
It must be a bug in PEFT or transformers or a problem with the original model code.
So I have not officially released the script for now. I uploaded it as a sample...
Spaces is impossible because there are no Zero GPU slots left...
I have code that allows model switching, but the spaces library is buggy right now, so I can't change the model to any architecture other than the one I loaded once...