Out of gpu memory error
Hi. I tried to use this model with your sample code.
But I am facing the out of gpu memory error. My gpu is 1080ti.
Please tell me your environment which operation confirmation.
And I want to know how to adjust batch size, or Do you have a lighter model?
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 80.00 MiB (GPU 0; 10.91 GiB total capacity; 10.26 GiB already allocated; 29.44 MiB free; 10.46 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Thank you for your help.
Hi! We trained and tested this model using an A40 GPU (48 GB VRAM). It seems like just loading the weights would require around 30 GB of VRAM for the base blip model (Flan-T5-XXL) (see https://huggingface.co/Salesforce/blip2-flan-t5-xxl/discussions/2#63f78cdd0074cebe75d26c68 ), but this would not fit in you 1080ti GPU. We do not have a lighter model yet.
Thank you for help and reply.
I understood. this time I tried to run with Cpus.
this is the result. Is it correct and what does it mean?
It seems tensor does not appear.
deprecate(
Loading checkpoint shards: 100%|██████| 5/5 [00:32<00:00, 6.53s/it]
['opaque', 'translucent', 'transparent']
Thank you.
Sorry, it seems like we used a slightly modified version of LAVIS when testing the model. We updated the instructions in the README and recommend using the provided test.py and generate.py.
The tensor represents the "sequence scores" which are the log probabilities of generating the corresponding responses. See the updated end of the model card for how to interpret these values.
Thanks all for the lighting fast response! I understand. It was very helpful for me.