Hugging Face's logo

qualcomm
/
ControlNet

Unconditional Image Generation
PyTorch
generative_ai
quantized
android
Model card Files Community
1
qaihm-bot commited on
Commit
ce7e4cd
•
1 Parent(s): 62cd0c5

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +58 -42
README.md CHANGED
@@ -16,7 +16,7 @@ tags:
16
 
17
  On-device, high-resolution image synthesis from text and image prompts. ControlNet guides Stable-diffusion with provided input image to generate accurate images from given input prompt.
18
 
19
- This model is an implementation of ControlNet found [here](https://github.com/lllyasviel/ControlNet).
20
  This repository provides scripts to run ControlNet on Qualcomm® devices.
21
  More details on model performance across various devices, can be found
22
  [here](https://aihub.qualcomm.com/models/controlnet_quantized).
@@ -34,17 +34,23 @@ More details on model performance across various devices, can be found
34
  - ControlNet Number of parameters: 361M
35
  - Model size: 1.4GB
36
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
37
 
38
 
39
 
40
- | Device | Chipset | Target Runtime | Inference Time (ms) | Peak Memory Range (MB) | Precision | Primary Compute Unit | Target Model
41
- | ---|---|---|---|---|---|---|---|
42
- | Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Binary | 11.394 ms | 0 - 74 MB | UINT16 | NPU | [TextEncoder_Quantized.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/TextEncoder_Quantized.bin)
43
- | Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Binary | 262.52 ms | 11 - 17 MB | UINT16 | NPU | [UNet_Quantized.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/UNet_Quantized.bin)
44
- | Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Binary | 390.243 ms | 0 - 36 MB | UINT16 | NPU | [VAEDecoder_Quantized.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/VAEDecoder_Quantized.bin)
45
- | Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Binary | 100.33 ms | 2 - 68 MB | UINT16 | NPU | [ControlNet_Quantized.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/ControlNet_Quantized.bin)
46
-
47
-
48
 
49
  ## Installation
50
 
@@ -100,37 +106,43 @@ device. This script does the following:
100
  ```bash
101
  python -m qai_hub_models.models.controlnet_quantized.export
102
  ```
103
-
104
  ```
105
- Profile Job summary of TextEncoder_Quantized
106
- --------------------------------------------------
107
- Device: QCS8550 (Proxy) (12)
108
- Estimated Inference Time: 10.98 ms
109
- Estimated Peak Memory Range: 0.07-1.20 MB
110
- Compute Units: NPU (569) | Total (569)
111
-
112
- Profile Job summary of UNet_Quantized
113
- --------------------------------------------------
114
- Device: QCS8550 (Proxy) (12)
115
- Estimated Inference Time: 260.16 ms
116
- Estimated Peak Memory Range: 13.75-15.10 MB
117
- Compute Units: NPU (5433) | Total (5433)
118
-
119
- Profile Job summary of VAEDecoder_Quantized
120
- --------------------------------------------------
121
- Device: QCS8550 (Proxy) (12)
122
- Estimated Inference Time: 379.55 ms
123
- Estimated Peak Memory Range: 0.28-1.47 MB
124
- Compute Units: NPU (408) | Total (408)
125
-
126
- Profile Job summary of ControlNet_Quantized
127
- --------------------------------------------------
128
- Device: QCS8550 (Proxy) (12)
129
- Estimated Inference Time: 103.52 ms
130
- Estimated Peak Memory Range: 1.85-3.07 MB
131
- Compute Units: NPU (2405) | Total (2405)
132
-
133
-
 
 
 
 
 
 
 
134
  ```
135
 
136
 
@@ -254,15 +266,19 @@ provides instructions on how to use the `.so` shared library or `.bin` context b
254
  Get more details on ControlNet's performance across various devices [here](https://aihub.qualcomm.com/models/controlnet_quantized).
255
  Explore all available models on [Qualcomm® AI Hub](https://aihub.qualcomm.com/)
256
 
 
257
  ## License
258
- - The license for the original implementation of ControlNet can be found
259
- [here](https://github.com/lllyasviel/ControlNet/blob/main/LICENSE).
260
- - The license for the compiled assets for on-device deployment can be found [here](https://github.com/lllyasviel/ControlNet/blob/main/LICENSE)
 
261
 
262
  ## References
263
  * [Adding Conditional Control to Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.05543)
264
  * [Source Model Implementation](https://github.com/lllyasviel/ControlNet)
265
 
 
 
266
  ## Community
267
  * Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
268
  * For questions or feedback please [reach out to us](mailto:[email protected]).
 
16
 
17
  On-device, high-resolution image synthesis from text and image prompts. ControlNet guides Stable-diffusion with provided input image to generate accurate images from given input prompt.
18
 
19
+ This model is an implementation of ControlNet found [here]({source_repo}).
20
  This repository provides scripts to run ControlNet on Qualcomm® devices.
21
  More details on model performance across various devices, can be found
22
  [here](https://aihub.qualcomm.com/models/controlnet_quantized).
 
34
  - ControlNet Number of parameters: 361M
35
  - Model size: 1.4GB
36
 
37
+ | Model | Device | Chipset | Target Runtime | Inference Time (ms) | Peak Memory Range (MB) | Precision | Primary Compute Unit | Target Model
38
+ |---|---|---|---|---|---|---|---|---|
39
+ | TextEncoder_Quantized | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | QNN | 11.394 ms | 0 - 74 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/TextEncoder_Quantized.bin) |
40
+ | TextEncoder_Quantized | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | QNN | 8.08 ms | 0 - 137 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/TextEncoder_Quantized.bin) |
41
+ | TextEncoder_Quantized | QCS8550 (Proxy) | QCS8550 Proxy | QNN | 10.982 ms | 0 - 1 MB | UINT16 | NPU | Use Export Script |
42
+ | UNet_Quantized | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | QNN | 262.52 ms | 11 - 17 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/UNet_Quantized.bin) |
43
+ | UNet_Quantized | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | QNN | 192.789 ms | 3 - 1247 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/UNet_Quantized.bin) |
44
+ | UNet_Quantized | QCS8550 (Proxy) | QCS8550 Proxy | QNN | 260.158 ms | 14 - 15 MB | UINT16 | NPU | Use Export Script |
45
+ | VAEDecoder_Quantized | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | QNN | 390.243 ms | 0 - 36 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/VAEDecoder_Quantized.bin) |
46
+ | VAEDecoder_Quantized | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | QNN | 294.404 ms | 0 - 88 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/VAEDecoder_Quantized.bin) |
47
+ | VAEDecoder_Quantized | QCS8550 (Proxy) | QCS8550 Proxy | QNN | 379.548 ms | 0 - 1 MB | UINT16 | NPU | Use Export Script |
48
+ | ControlNet_Quantized | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | QNN | 100.33 ms | 2 - 68 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/ControlNet_Quantized.bin) |
49
+ | ControlNet_Quantized | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | QNN | 76.94 ms | 0 - 533 MB | UINT16 | NPU | [ControlNet.bin](https://huggingface.co/qualcomm/ControlNet/blob/main/ControlNet_Quantized.bin) |
50
+ | ControlNet_Quantized | QCS8550 (Proxy) | QCS8550 Proxy | QNN | 103.52 ms | 2 - 3 MB | UINT16 | NPU | Use Export Script |
51
 
52
 
53
 
 
 
 
 
 
 
 
 
54
 
55
  ## Installation
56
 
 
106
  ```bash
107
  python -m qai_hub_models.models.controlnet_quantized.export
108
  ```
 
109
  ```
110
+ Profiling Results
111
+ ------------------------------------------------------------
112
+ TextEncoder_Quantized
113
+ Device : Samsung Galaxy S23 (13)
114
+ Runtime : QNN
115
+ Estimated inference time (ms) : 11.4
116
+ Estimated peak memory usage (MB): [0, 74]
117
+ Total # Ops : 570
118
+ Compute Unit(s) : NPU (570 ops)
119
+
120
+ ------------------------------------------------------------
121
+ UNet_Quantized
122
+ Device : Samsung Galaxy S23 (13)
123
+ Runtime : QNN
124
+ Estimated inference time (ms) : 262.5
125
+ Estimated peak memory usage (MB): [11, 17]
126
+ Total # Ops : 5434
127
+ Compute Unit(s) : NPU (5434 ops)
128
+
129
+ ------------------------------------------------------------
130
+ VAEDecoder_Quantized
131
+ Device : Samsung Galaxy S23 (13)
132
+ Runtime : QNN
133
+ Estimated inference time (ms) : 390.2
134
+ Estimated peak memory usage (MB): [0, 36]
135
+ Total # Ops : 409
136
+ Compute Unit(s) : NPU (409 ops)
137
+
138
+ ------------------------------------------------------------
139
+ ControlNet_Quantized
140
+ Device : Samsung Galaxy S23 (13)
141
+ Runtime : QNN
142
+ Estimated inference time (ms) : 100.3
143
+ Estimated peak memory usage (MB): [2, 68]
144
+ Total # Ops : 2406
145
+ Compute Unit(s) : NPU (2406 ops)
146
  ```
147
 
148
 
 
266
  Get more details on ControlNet's performance across various devices [here](https://aihub.qualcomm.com/models/controlnet_quantized).
267
  Explore all available models on [Qualcomm® AI Hub](https://aihub.qualcomm.com/)
268
 
269
+
270
  ## License
271
+ * The license for the original implementation of ControlNet can be found [here](https://github.com/lllyasviel/ControlNet/blob/main/LICENSE).
272
+ * The license for the compiled assets for on-device deployment can be found [here](https://github.com/lllyasviel/ControlNet/blob/main/LICENSE)
273
+
274
+
275
 
276
  ## References
277
  * [Adding Conditional Control to Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.05543)
278
  * [Source Model Implementation](https://github.com/lllyasviel/ControlNet)
279
 
280
+
281
+
282
  ## Community
283
  * Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
284
  * For questions or feedback please [reach out to us](mailto:[email protected]).