EXL2 - 3.64 bpw

#2
by zaq-hack - opened

EXL2 ingests mixtral prompts considerably faster.
3.64 bpw plays nice with my 3090 @ 16-24k context.

https://huggingface.co/zaq-hack/Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss-bpw364-h6-exl2

Sign up or log in to comment