| >FlexGen is a high-throughput generation engine for running large language models with limited GPU memory (e.g., a 16GB T4 GPU or a 24GB RTX3090 gaming card!). | |
| https://github.com/FMInference/FlexGen | |
| ## Installation | |
| No additional installation steps are necessary. FlexGen is in the `requirements.txt` file for this project. | |
| ## Converting a model | |
| FlexGen only works with the OPT model, and it needs to be converted to numpy format before starting the web UI: | |
| ``` | |
| python convert-to-flexgen.py models/opt-1.3b/ | |
| ``` | |
| The output will be saved to `models/opt-1.3b-np/`. | |
| ## Usage | |
| The basic command is the following: | |
| ``` | |
| python server.py --model opt-1.3b --loader flexgen | |
| ``` | |
| For large models, the RAM usage may be too high and your computer may freeze. If that happens, you can try this: | |
| ``` | |
| python server.py --model opt-1.3b --loader flexgen --compress-weight | |
| ``` | |
| With this second command, I was able to run both OPT-6.7b and OPT-13B with **2GB VRAM**, and the speed was good in both cases. | |
| You can also manually set the offload strategy with | |
| ``` | |
| python server.py --model opt-1.3b --loader flexgen --percent 0 100 100 0 100 0 | |
| ``` | |
| where the six numbers after `--percent` are: | |
| ``` | |
| the percentage of weight on GPU | |
| the percentage of weight on CPU | |
| the percentage of attention cache on GPU | |
| the percentage of attention cache on CPU | |
| the percentage of activations on GPU | |
| the percentage of activations on CPU | |
| ``` | |
| You should typically only change the first two numbers. If their sum is less than 100, the remaining layers will be offloaded to the disk, by default into the `text-generation-webui/cache` folder. | |
| ## Performance | |
| In my experiments with OPT-30B using a RTX 3090 on Linux, I have obtained these results: | |
| * `--loader flexgen --compress-weight --percent 0 100 100 0 100 0`: 0.99 seconds per token. | |
| * `--loader flexgen --compress-weight --percent 100 0 100 0 100 0`: 0.765 seconds per token. | |
| ## Limitations | |
| * Only works with the OPT models. | |
| * Only two generation parameters are available: `temperature` and `do_sample`. |