Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
library_name: diffusers
|
3 |
+
license: other
|
4 |
+
license_name: flux-1-dev-non-commercial-license
|
5 |
+
license_link: LICENSE.md
|
6 |
+
---
|
7 |
+
|
8 |
+
## Running Flux.1-dev under 12GBs
|
9 |
+
|
10 |
+
This repository contains the mixed int8 params for the T5 and transformer of Flux.1-Dev.
|
11 |
+
|
12 |
+
This is how the checkpoints were obtained:
|
13 |
+
|
14 |
+
```python
|
15 |
+
import torch
|
16 |
+
from diffusers import BitsAndBytesConfig, FluxTransformer2DModel
|
17 |
+
from transformers import T5EncoderModel, BitsAndBytesConfig
|
18 |
+
from huggingface_hub import create_repo, upload_folder
|
19 |
+
import tempfile
|
20 |
+
|
21 |
+
model_id = "black-forest-labs/FLUX.1-dev"
|
22 |
+
|
23 |
+
config = BitsAndBytesConfig(load_in_8bit=True)
|
24 |
+
transformer = FluxTransformer2DModel.from_pretrained(
|
25 |
+
model_id, subfolder="transformer", quantization_config=config, torch_dtype=torch.float16
|
26 |
+
)
|
27 |
+
|
28 |
+
config = BitsAndBytesConfig(load_in_8bit=True)
|
29 |
+
t5 = T5EncoderModel.from_pretrained(
|
30 |
+
model_id, subfolder="text_encoder_2", quantization_config=config, torch_dtype=torch.float16
|
31 |
+
)
|
32 |
+
|
33 |
+
repo_id = create_repo("sayakpaul/flux.1-dev-int8-pkg", exist_ok=True).repo_id
|
34 |
+
|
35 |
+
with tempfile.TemporaryDirectory() as tmpdirname:
|
36 |
+
transformer.save_pretrained(tmpdirname)
|
37 |
+
upload_folder(repo_id=repo_id, folder_path=tmpdirname, path_in_repo="transformer")
|
38 |
+
|
39 |
+
with tempfile.TemporaryDirectory() as tmpdirname:
|
40 |
+
t5.save_pretrained(tmpdirname)
|
41 |
+
upload_folder(repo_id=repo_id, folder_path=tmpdirname, path_in_repo="text_encoder_2")
|
42 |
+
```
|
43 |
+
|
44 |
+
Respective `diffusers` PR: https://github.com/huggingface/diffusers/pull/9213/.
|
45 |
+
|
46 |
+
> [!NOTE]
|
47 |
+
> The checkpoints of this repository were optimized to run on a T4 notebook. More specifically, the compute datatype of the quantized checkpoints was kept to FP16. In practice, if you have a GPU card that supports BF16, you should change the compute datatype to BF16 (`bnb_4bit_compute_dtype`).
|