readme updated
Browse files
README.md
CHANGED
|
@@ -1,3 +1,48 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
library_name: diffusers
|
| 6 |
+
tags:
|
| 7 |
+
- text-to-image
|
| 8 |
+
- prior
|
| 9 |
+
- eclipse
|
| 10 |
+
- unclip
|
| 11 |
+
- kandinskyv2.2
|
| 12 |
---
|
| 13 |
+
|
| 14 |
+
# Introduction
|
| 15 |
+
|
| 16 |
+
The λ-ECLIPSE model is a light weight support for multi-concept personalization. λ-ECLIPSE is tiny T2I prior model designed for Kandinsky v2.2 diffusion image generator.
|
| 17 |
+
|
| 18 |
+
λ-ECLIPSE model extends the [ECLIPSE-Prior](https://huggingface.co/ECLIPSE-Community/ECLIPSE_KandinskyV22_Prior) via incorporating the image-text interleaved data.
|
| 19 |
+
|
| 20 |
+
λ-ECLIPSE shows that we do not need to train the Personalized T2I (P-T2I) models on lot of resources. For instance, λ-ECLIPSE is trained on mere 74 GPU Hours (A100) compared to it's couterparts BLIP-Diffusion (2304 GPU hours) and Kosmos-G (12300 GPU hours).
|
| 21 |
+
|
| 22 |
+
- **Project Page:** [https://eclipse-t2i.github.io/Lambda-ECLIPSE/](https://eclipse-t2i.github.io/Lambda-ECLIPSE/)
|
| 23 |
+
- **GitHub:** [https://github.com/eclipse-t2i/eclipse-inference](https://github.com/eclipse-t2i/eclipse-inference)
|
| 24 |
+
- **Paper (arXiv):** [TBD](#)
|
| 25 |
+
|
| 26 |
+
Importantly, λ-ECLIPSE works in pure CLIP latent space without any additional information. Hence, it's performance can be easily imporved via test-time adaption to increase the concept alignment while having solid composition alignment.
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
More examples at: [Gallery](https://eclipse-t2i.github.io/Lambda-ECLIPSE/gallery.html)
|
| 32 |
+
|
| 33 |
+
## Installation
|
| 34 |
+
```bash
|
| 35 |
+
git clone [email protected]:eclipse-t2i/eclipse-inference.git
|
| 36 |
+
conda create -p ./venv python=3.9
|
| 37 |
+
pip install -r requirements.txt
|
| 38 |
+
```
|
| 39 |
+
|
| 40 |
+
## Run Inference
|
| 41 |
+
|
| 42 |
+
TBD
|
| 43 |
+
|
| 44 |
+
## Important Notes (and limitations):
|
| 45 |
+
|
| 46 |
+
- λ-ECLIPSE is trained to support upto four unique concepts, however, this version is trained on biased datasets heavily focusing on single and two subjects. Therefore, it maynot perform expectadly as number of subjects increases.
|
| 47 |
+
- As this model is trained for P-T2I specifically, it might not perform well on traditional T2I task.
|
| 48 |
+
- λ-ECLIPSE achieves SOTA compositional performance on composition alignment while maintaining the concept alignment. However, there is still a big gap compared to the finetuning based methodologies.
|