Files changed (1) hide show
  1. README.md +70 -3
README.md CHANGED
@@ -1,3 +1,70 @@
1
- ---
2
- license: llama3.1
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: llama3.1
3
+ ---
4
+ # Meta-Llama-3.1-70B-Instruct-FP8-KV
5
+ - ## Introduction
6
+ This model was created by applying [Quark](https://quark.docs.amd.com/latest/index.html) with calibration samples from Pile dataset.
7
+ - ## Quantization Stragegy
8
+ - ***Quantized Layers***:All linear layers excluding "lm_head"
9
+ - ***Weight***: FP8 symmetric per-tensor
10
+ - ***Activation***: FP8 symmetric per-tensor
11
+ - ***KV Cache***: FP8 symmetric per-tensor
12
+ - ## Quick Start
13
+ 1. [Download and install Quark](https://quark.docs.amd.com/latest/install.html)
14
+ 2. Run the quantization script in the example folder using the following command line:
15
+ ```sh
16
+ export MODEL_DIR = [local model checkpoint folder] or meta-llama/Meta-Llama-3.1-70B-Instruct
17
+ # single GPU
18
+ python3 quantize_quark.py \
19
+ --model_dir $MODEL_DIR \
20
+ --output_dir Meta-Llama-3.1-70B-Instruct-FP8-KV \
21
+ --quant_scheme w_fp8_a_fp8 \
22
+ --kv_cache_dtype fp8 \
23
+ --num_calib_data 128 \
24
+ --model_export quark_safetensors
25
+ # If model size is too large for single GPU, please use multi GPU instead.
26
+ python3 quantize_quark.py \
27
+ --model_dir $MODEL_DIR \
28
+ --output_dir Meta-Llama-3.1-70B-Instruct-FP8-KV \
29
+ --quant_scheme w_fp8_a_fp8 \
30
+ --kv_cache_dtype fp8 \
31
+ --num_calib_data 128 \
32
+ --model_export quark_safetensors \
33
+ --multi_gpu
34
+ ```
35
+ ## Deployment
36
+ Quark has its own export format and allows FP8 quantized models to be efficiently deployed using the vLLM backend(vLLM-compatible).
37
+ ## Evaluation
38
+ Quark currently uses perplexity(PPL) as the evaluation metric for accuracy loss before and after quantization.The specific PPL algorithm can be referenced in the quantize_quark.py.
39
+ The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.
40
+ #### Evaluation scores
41
+ <table>
42
+ <tr>
43
+ <td><strong>Benchmark</strong>
44
+ </td>
45
+ <td><strong>Meta-Llama-3.1-70B-Instruct </strong>
46
+ </td>
47
+ <td><strong>Meta-Llama-3.1-70B-Instruct-FP8-KV(this model)</strong>
48
+ </td>
49
+ </tr>
50
+ <tr>
51
+ <td>Perplexity-wikitext2
52
+ </td>
53
+ <td>5.3164
54
+ </td>
55
+ <td>5.4323
56
+ </td>
57
+ </tr>
58
+ </table>
59
+
60
+ #### License
61
+ Copyright (c) 2018-2024 Advanced Micro Devices, Inc. All Rights Reserved.
62
+ Licensed under the Apache License, Version 2.0 (the "License");
63
+ you may not use this file except in compliance with the License.
64
+ You may obtain a copy of the License at
65
+ http://www.apache.org/licenses/LICENSE-2.0
66
+ Unless required by applicable law or agreed to in writing, software
67
+ distributed under the License is distributed on an "AS IS" BASIS,
68
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
69
+ See the License for the specific language governing permissions and
70
+ limitations under the License.