Any UI supports the Tool usage of this great model?

#3
by DrNicefellow - opened

As title suggests

Textgen webui supports it, I haven't tried the original safetensors but it works with a exl2 and gguf quant I made myself. LLama.cpp works too

deleted

how big is your gguf? With it being so huge, i didnt even bother trying.

You can use https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator to check the different sizes for gguf quants

Textgen webui supports it, I haven't tried the original safetensors but it works with a exl2 and gguf quant I made myself. LLama.cpp works too

Textgen webui supports it, I haven't tried the original safetensors but it works with a exl2 and gguf quant I made myself. LLama.cpp works too

How does text generation web ui supports it? Which plugin uses mistral defined tool usage format?

deleted

I used 8x7 in oobas text gen 'as is' nothing special was needed.

I used 8x7 in oobas text gen 'as is' nothing special was needed.

How did it use tools?

Sign up or log in to comment