socket hung up

#12
by MrCopperField - opened

I Used the docker image to use this model.
When I hit the api it return the error that socket hung up. I have 16 gb Nvidia 4060TI.

Call the server using curl:

curl -X POST "http://localhost:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "meta-llama/Llama-3.2-1B"
"messages": [
{"role": "user", "content": "Hello!"}
]
}'

Meta Llama org

Unfortunately it will be hard for us to help debug this given docker containers and the setup details.

You can try setting up a docker "distro" using llama-stack, details, which might be more straight forward.

Sign up or log in to comment