Endpoint encountered an error
#5
by
vanokio
- opened
Hi!
I'm trying to deploy this model by Inference Endpoints HF and took the server with Nvidia A10G 24GB and 28GB memory, but when my model was deploying, I got an error from HF server "Endpoint failed to start". I have no clue why it happened. Please, help me with your advice :)
I have attached screenshots of the error below
Hi! You need to pass trust_remote_code=True to the model initialization somehow.
I'm not familiar with the endpoints interface so I can't tell you exactly how to do it.
I would also recommend passing low_cpu_mem_usage=True to reduce initialization time and device_map="cuda" to reduce RAM requirements significantly.