Demonstration of Cross-modal Retrieval (FLIP-based model)
FLIP (Facial Language Image Pretraining)
This repository is the official implementation of FaceCaption-15M.
Updates:
[24/07/20] The usage of FLIP has been released! OpenFace-CQUPT/FLIP-demo
[24/07/17] The model named FLIP has been released! OpenFace-CQUPT/FLIP
Overview of FLIP architecture.
Fig.1:(a). Same color represents shared parameters. “12x” stands for 12-layer transformer modules. (b), (c) and (d) FLIP-based model are applied to the tasks of text-image retrieval, facial attributes prediction and sketch less facial image retrieval, respectively.
Training
Coming soon......(Only for the datasets been published, the code of training is meaningful.)
python pretrain.py > log.log
Pre-trained Models
We provide pretrained model weights :
FLIP Base —— click here
FLIP Large —— coming soon......
Datasets
Download the FaceCaption-15M dataset from here.
Results
Task1: Text-Image Retrieval
Table 1: Comparison with other classical pretrained models. All pretrained model backbones are frozen, with only the linear layer being fine-tuned. † represents the model pretrained on the LAION-Face [86] dataset; * represents the model pretrained on the FaceCaption dataset constructed without using LLM text generation.
Task2: Facial Attributes Prediction
Table 2: Comparison with other classical models. † represents the model pre-trained on the original LAION-Face dataset.
Task3: Sketch Less Facial Image Retrieval
Table 3: Comparative results with different baseline methods. † represents the model pre-trained on the LAION-Face dataset.
Fig.2:Demonstration of our FLIP-based model on the SLFIR task. Both methods can retrieve the target face photo from the top-5 list using a partial sketch. Our proposed FLIP-based model can achieve this using fewer strokes than the baseline. The number at the bottom denotes the rank of the paired (true match) photos at every stage.
Contacts
mailto: [email protected] or [email protected]
Citation
@misc{dai202415mmultimodalfacialimagetext,
title={15M Multimodal Facial Image-Text Dataset},
author={Dawei Dai and YuTang Li and YingGe Liu and Mingming Jia and Zhang YuanHui and Guoyin Wang},
year={2024},
eprint={2407.08515},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2407.08515},
}