Model Card for Mistral-7B-Instruct-v0.3 quantized to 4bit weights
- Weight-only quantization of Mistral-7B-Instruct-v0.3 via GPTQ to 4bits with group_size=128
- GPTQ optimized for 99.75% accuracy recovery relative to the unquantized model
Open LLM Leaderboard evaluation scores
Mistral-7B-Instruct-v0.3 | Mistral-7B-Instruct-v0.3-GPTQ-4bit (this model) |
|
---|---|---|
arc-c 25-shot |
63.48 | 63.40 |
mmlu 5-shot |
61.13 | 60.89 |
hellaswag 10-shot |
84.49 | 84.04 |
winogrande 5-shot |
79.16 | 79.08 |
gsm8k 5-shot |
43.37 | 45.41 |
truthfulqa 0-shot |
59.65 | 57.48 |
Average Accuracy |
65.21 | 65.05 |
Recovery | 100% | 99.75% |
vLLM Inference Performance
This model is ready for optimized inference using the Marlin mixed-precision kernels in vLLM: https://github.com/vllm-project/vllm
Simply start this model as an inference server with:
python -m vllm.entrypoints.openai.api_server --model neuralmagic/Mistral-7B-Instruct-v0.3-GPTQ-4bit
- Downloads last month
- 98,352
Model tree for neuralmagic/Mistral-7B-Instruct-v0.3-GPTQ-4bit
Base model
mistralai/Mistral-7B-v0.3
Finetuned
mistralai/Mistral-7B-Instruct-v0.3
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set self-reported63.400
- normalized accuracy on HellaSwag (10-Shot)validation set self-reported84.040
- mc2 on TruthfulQA (0-shot)validation set self-reported57.480
- accuracy on GSM8k (5-shot)test set self-reported45.410
- accuracy on MMLU (5-Shot)test set self-reported61.070
- accuracy on Winogrande (5-shot)validation set self-reported79.080