| These are the VRAM and RAM requirements (in MiB) to run some examples of models **in 16-bit (default) precision**: | |
| | model | VRAM (GPU) | RAM | | |
| |:-----------------------|-------------:|--------:| | |
| | arxiv_ai_gpt2 | 1512.37 | 5824.2 | | |
| | blenderbot-1B-distill | 2441.75 | 4425.91 | | |
| | opt-1.3b | 2509.61 | 4427.79 | | |
| | gpt-neo-1.3b | 2605.27 | 5851.58 | | |
| | opt-2.7b | 5058.05 | 4863.95 | | |
| | gpt4chan_model_float16 | 11653.7 | 4437.71 | | |
| | gpt-j-6B | 11653.7 | 5633.79 | | |
| | galactica-6.7b | 12697.9 | 4429.89 | | |
| | opt-6.7b | 12700 | 4368.66 | | |
| | bloomz-7b1-p3 | 13483.1 | 4470.34 | | |
| #### GPU mode with 8-bit precision | |
| Allows you to load models that would not normally fit into your GPU. Enabled by default for 13b and 20b models in this web UI. | |
| | model | VRAM (GPU) | RAM | | |
| |:---------------|-------------:|--------:| | |
| | opt-13b | 12528.1 | 1152.39 | | |
| | gpt-neox-20b | 20384 | 2291.7 | | |
| #### CPU mode (32-bit precision) | |
| A lot slower, but does not require a GPU. | |
| On my i5-12400F, 6B models take around 10-20 seconds to respond in chat mode, and around 5 minutes to generate a 200 tokens completion. | |
| | model | RAM | | |
| |:-----------------------|---------:| | |
| | arxiv_ai_gpt2 | 4430.82 | | |
| | gpt-neo-1.3b | 6089.31 | | |
| | opt-1.3b | 8411.12 | | |
| | blenderbot-1B-distill | 8508.16 | | |
| | opt-2.7b | 14969.3 | | |
| | bloomz-7b1-p3 | 21371.2 | | |
| | gpt-j-6B | 24200.3 | | |
| | gpt4chan_model | 24246.3 | | |
| | galactica-6.7b | 26561.4 | | |
| | opt-6.7b | 29596.6 | | |