---
license: mit
datasets:
- NeelNanda/pile-10k
language:
- en
---


## Model Details

This model is an int4 model recipe with group_size 128 of [microsoft/Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct)  generated by [intel/auto-round](https://github.com/intel/auto-round). 
Inference of this model is compatible with AutoGPTQ's Kernel.







### Quantize the model

Here is the sample command to reproduce the model

```bash
pip install auto-round
auto-round
--model  microsoft/Phi-3-mini-128k-instruct \
--device 0 \
--group_size 128 \
--bits 4 \
--iters 200 \
--nsamples 512 \
--seqlen 4096 \
--minmax_lr 0.01 \
--format 'auto_gptq' \
--gradient_accumulate_steps 2 \
--batch_size 4 \
--output_dir "./tmp_autoround" \

```





## How to use

### INT4 Inference with IPEX on Intel CPU
Install the latest [Intel Extension for Pytorch](https://github.com/intel/intel-extension-for-pytorch) and [Intel Neural Compressor](https://github.com/intel/neural-compressor)

```bash
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install intel_extension_for_pytorch
pip install neural_compressor_pt 
```

```python
from transformers import AutoTokenizer
from neural_compressor.transformers import AutoModelForCausalLM

## note: use quantized model directory name below 
model_name_or_path="./tmp_autoround/<model directory name>"
q_model = AutoModelForCausalLM.from_pretrained(model_name_or_path)

prompt = "Once upon a time, a little girl"

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
print(tokenizer.decode(q_model.generate(**tokenizer(prompt, return_tensors="pt").to(q_model.device),max_new_tokens=50)[0]))
##Once upon a time, a little girl named Lily was playing in her backyard. She loved to explore and discover new things. One day, she stumbled upon a beautiful garden filled with colorful flowers andugh the garden, she noticed a
```

### INT4 Inference on Intel Gaudi Accelerator
docker image with Gaudi Software Stack is recommended. More details can be found in [Gaudi Guide](https://docs.habana.ai/en/latest/).

```python
import habana_frameworks.torch.core as htcore
from neural_compressor.torch.quantization import load
from transformers import  AutoTokenizer, AutoModelForCausalLM

## note: use quantized model directory name below
model_name_or_path="./tmp_autoround/<model directory name>"

model = load(
    model_name_or_path=model_name_or_path,
    format="huggingface",
    device="hpu"
)

prompt = "Once upon a time, a little girl"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
print(tokenizer.decode(model.generate(**tokenizer(prompt, return_tensors="pt").to("hpu"),max_new_tokens=50)[0]))

```

## Accuracy Result 


 | Metric   <img width=200>     | FP16  <img width=200>   | INT4  <img width=200>   |
| -------------- | ------ | ------ |
| Avg.           | 0.6364 | 0.6300 |
| mmlu           | 0.6215 | 0.6237 |
| lambada_openai | 0.6656 | 0.6433 |
| hellaswag      | 0.5979 | 0.5859 |
| winogrande     | 0.7324 | 0.7230 |
| piqa           | 0.7884 | 0.7846 |
| truthfulqa_mc1 | 0.3574 | 0.3562 |
| openbookqa     | 0.3900 | 0.3800 |
| boolq          | 0.8572 | 0.8489 |
| arc_easy       | 0.8119 | 0.8199 |
| arc_challenge  | 0.5418 | 0.5350 |

## Caveats and Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

Here are a couple of useful links to learn more about Intel's AI software:

* Intel Neural Compressor [link](https://github.com/intel/neural-compressor)



## Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.



## Cite

@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }

[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round)