Make transformers inference code CPU compatible
#6
by
MoritzLaurer
HF staff
- opened
Very cool small models!
The custom code in modeling_internvl_chat.py currently has a GPU requirement hardcoded with model_inputs['input_ids'].cuda() etc.
To make the code also run on CPUs (especially for the nice small models), using e.g. model_inputs['input_ids'].to(model.device) would be better
Thanks for pointing that out! We'll replace .cuda() with .to(model.device) to support both GPUs and CPUs, making the code more flexible.
czczup
changed discussion status to
closed
Now this code is CPU compatible.