LoneStriker

/

Nous-Hermes-2-Mixtral-8x7B-DPO-4.0bpw-h6-exl2

like 0

These are exllamav2 quantized versions of the models. You need to use exllamav2 itself to load these models via Python. There are simple examples in the Github project to load the model and run inference. If you want to use a GUI, use exui, ooba's text-generation-webui (with exllamav2 or exllamav2_hf as model loaders), or other packages like tabbyAPI.