Mixtral-8x7B (gpt-fast edition)

This repo holds quantized Mixtral-8x7B weights to be used in gpt-fast.

Compatibility

Conversion to int4 was broken, so this repo only holds fp8 weights. Practically speaking this means your GPU(s) need to be Ada Lovelace or newer, and have enough VRAM to hold the model + KV cache + activations.

I'm hoping it can work on a pair of 4090s, which combined have 48 GiB (51.539607552 GB) of VRAM. Ignoring all overhead, this leaves ~4.74 GB for KV-cache and activations, which should be enough (?).

TODO: Test on 2x4090 with TP=2

Notes

Conversion was done with (commit 7510a9d)