File size: 4,166 Bytes
52b0a56 4159e15 52b0a56 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
---
license: apache-2.0
---
# FaceScore
<p align="center">
๐ <a href="https://arxiv.org/abs/2406.17100" target="_blank">Paper</a> โข ๐ <a href="https://github.com/OPPO-Mente-Lab/FaceScore" target="_blank">Repo</a>
</p>
**FaceScore: Benchmarking and Enhancing Face Quality in Human Generation**
Traditional facial quality assessment focuses on whether a face is suitable for recognition, while image aesthetic scorers emphasize overall aesthetics rather than details. FaceScore is the first reward model that focuses on faces in text-to-image models, designed to score the faces generated in images. It is fine-tuned on positive and negative sample pairs generated using an inpainting pipeline based on real face images and surpasses previous models in predicting human preferences for generated faces.
- [LoRA base on SDXL](#lora-based-on-sdxl)
- [Citation](#citation)
## LoRA based on SDXL
We leverage FaceScore to filter data and perform direct preference optimization on SDXL.
The LoRA weight is [here](https://huggingface.co/AIGCer-OPPO/FaceScore-dpo-SDXL-LoRA/tree/main).
Here we provide a quick example:
```
from diffusers import StableDiffusionXLPipeline, UNet2DConditionModel
import torch
# load pipeline
inference_dtype = torch.float16
pipe = StableDiffusionXLPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=inference_dtype,
)
vae = AutoencoderKL.from_pretrained(
'madebyollin/sdxl-vae-fp16-fix',
torch_dtype=inference_dtype,
)
pipe.vae = vae
# You can load it locally
pipe.load_lora_weights("AIGCer-OPPO/FaceScore-dpo-SDXL-LoRA")
pipe.to('cuda')
generator=torch.Generator(device='cuda').manual_seed(42)
image = pipe(
prompt='A woman in a costume standing in the desert',
guidance_scale=5.0,
generator=generator,
output_type='pil',
).images[0]
image.save('A woman in a costume standing in the desert.png')
```
We provide some examples generated by ours (right) and compare with the original SDXL (left) below.
<div style="display: flex; justify-content: space-around;">
<div style="text-align: center;">
<img src="https://huggingface.co/AIGCer-OPPO/FaceScore-dpo-SDXL-LoRA/resolve/main/assets/desert.jpg" alt="ๅพ็1" style="width: 600px;" />
<p>A woman in a costume standing in the desert. </p>
</div>
<div style="text-align: center;">
<img src="https://huggingface.co/AIGCer-OPPO/FaceScore-dpo-SDXL-LoRA/resolve/main/assets/scarf.jpg" alt="ๅพ็2" style="width: 600px;" />
<p>A woman wearing a blue jacket and scarf.</p>
</div>
</div>
<div style="display: flex; justify-content: space-around;">
<div style="text-align: center;">
<img src="https://huggingface.co/AIGCer-OPPO/FaceScore-dpo-SDXL-LoRA/resolve/main/assets/stage.jpg" alt="ๅพ็1" style="width: 600px;" />
<p>A young woman in a blue dress performing on stage. </p>
</div>
<div style="text-align: center;">
<img src="https://huggingface.co/AIGCer-OPPO/FaceScore-dpo-SDXL-LoRA/resolve/main/assets/striped.jpg" alt="ๅพ็2" style="width: 600px;" />
<p>A woman with black hair and a striped shirt.</p>
</div>
</div>
<div style="display: flex; justify-content: space-around;">
<div style="text-align: center;">
<img src="https://huggingface.co/AIGCer-OPPO/FaceScore-dpo-SDXL-LoRA/resolve/main/assets/sword.jpg" alt="ๅพ็1" style="width: 600px;" />
<p>A woman with white hair and white armor is holding a sword. </p>
</div>
<div style="text-align: center;">
<img src="https://huggingface.co/AIGCer-OPPO/FaceScore-dpo-SDXL-LoRA/resolve/main/assets/white.jpg" alt="ๅพ็2" style="width: 600px;" />
<p>A woman with long black hair and a white shirt.</p>
</div>
</div>
## Citation
```
@misc{liao2024facescorebenchmarkingenhancingface,
title={FaceScore: Benchmarking and Enhancing Face Quality in Human Generation},
author={Zhenyi Liao and Qingsong Xie and Chen Chen and Hannan Lu and Zhijie Deng},
year={2024},
eprint={2406.17100},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2406.17100},
}
``` |