File size: 3,139 Bytes
f7aa269
436416f
fb0c5c4
40b0f03
 
f7aa269
 
1056c77
f7aa269
 
287bcf3
3aa66e3
f7aa269
 
 
 
 
 
 
 
 
 
9b9a87f
f7aa269
 
b1f366e
 
 
f7aa269
 
 
 
9b9a87f
f7aa269
 
b1f366e
9b9a87f
b1f366e
9b9a87f
f7aa269
1056c77
6a6e7f7
1056c77
f7aa269
 
1056c77
f7aa269
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1056c77
 
f7aa269
 
 
 
 
 
 
 
 
 
 
 
 
fb0c5c4
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
---
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
license: other
license_name: llama3.1
license_link: https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE
---
# Meta-Llama-3.1-8B-Instruct-FP8-KV
- ## Introduction
  This model was created by applying [Quark](https://quark.docs.amd.com/latest/index.html) with calibration samples from Pile dataset.
- ## Quantization Stragegy
  - ***Quantized Layers***: All linear layers excluding "lm_head"
  - ***Weight***: FP8 symmetric per-tensor
  - ***Activation***: FP8 symmetric per-tensor
  - ***KV Cache***: FP8 symmetric  per-tensor
- ## Quick Start
1. [Download and install Quark](https://quark.docs.amd.com/latest/install.html)
2. Run the quantization script in the example folder using the following command line:
```sh
export MODEL_DIR = [local model checkpoint folder] or meta-llama/Meta-Llama-3.1-8B-Instruct 
# single GPU
python3 quantize_quark.py \ 
        --model_dir $MODEL_DIR \
        --output_dir Meta-Llama-3.1-8B-Instruct-FP8-KV \                           
        --quant_scheme w_fp8_a_fp8 \
        --kv_cache_dtype fp8 \
        --num_calib_data 128 \
        --model_export quark_safetensors \
        --no_weight_matrix_merge

# If model size is too large for single GPU, please use multi GPU instead.
python3 quantize_quark.py \ 
        --model_dir $MODEL_DIR \
        --output_dir Meta-Llama-3.1-8B-Instruct-FP8-KV \                           
        --quant_scheme w_fp8_a_fp8 \
        --kv_cache_dtype fp8 \
        --num_calib_data 128 \
        --model_export quark_safetensors \
       --no_weight_matrix_merge \
        --multi_gpu
```
## Deployment
Quark has its own export format and allows FP8 quantized models to be efficiently deployed using the vLLM backend(vLLM-compatible).

## Evaluation
Quark currently uses perplexity(PPL) as the evaluation metric for accuracy loss before and after quantization.The specific PPL algorithm can be referenced in the quantize_quark.py.
The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.

#### Evaluation scores
<table>
  <tr>
   <td><strong>Benchmark</strong>
   </td>
   <td><strong>Meta-Llama-3.1-8B-Instruct </strong>
   </td>
   <td><strong>Meta-Llama-3.1-8B-Instruct-FP8-KV(this model)</strong>
   </td>
  </tr>
  <tr>
   <td>Perplexity-wikitext2
   </td>
   <td>7.2169
   </td>
   <td>7.2752
   </td>
  </tr>
  
</table>



#### License
Copyright (c) 2018-2024 Advanced Micro Devices, Inc. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.