File size: 1,638 Bytes
57ffbe0 4e42a1b f53b713 4e42a1b f5bbdc7 4e42a1b d79654c 4e42a1b 57ffbe0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
---
license: apache-2.0
---
## Paper
arxiv.org/abs/2405.20222
## Introduction
This repo provides the inference Gradio demo for **Hybrid (Trajectory + Landmark)** Control of [MOFA-Video](https://myniuuu.github.io/MOFA_Video/).
## Environment Setup
```
cd MOFA-Hybrid
conda create -n mofa python==3.10
conda activate mofa
pip install -r requirements.txt
pip install opencv-python-headless
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
```
**IMPORTANT:** Gradio Version of **4.5.0** should be used since other versions may cause errors.
## Checkpoints Download
1. Download the checkpoint of CMP from [here](https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid/blob/main/models/cmp/experiments/semiauto_annot/resnet50_vip%2Bmpii_liteflow/checkpoints/ckpt_iter_42000.pth.tar) and put it into `./models/cmp/experiments/semiauto_annot/resnet50_vip+mpii_liteflow/checkpoints`.
2. Downloading the necessary pretrained checkpoints from [huggingface](https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid). It is recommended to directly using git lfs to clone the [huggingface repo](https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid). The checkpoints should be orgnized as `./ckpt_tree.md` (they will be automatically organized if you use git lfs to clone the [huggingface repo](https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid)).
## Run Gradio Demo
### Using audio to animate the facial part
`python run_gradio_audio_driven.py`
### Using refernce video to animate the facial part
`python run_gradio_video_driven.py`
**IMPORTANT:** Please refer to the instructions on the gradio interface during the inference process. |