|
--- |
|
license: mit |
|
tags: |
|
- VAE |
|
- Video-Generation |
|
--- |
|
|
|
# Reducio-VAE Model Card |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
This model is a 3D VAE that encodes video into a compact latent space conditioned on a content frame. It compresses a video by a factor of \\(\frac{T}{4}\times\frac{H}{32}\times\frac{W}{32}\\), enabling 4096x downsampling. |
|
It is part of the [Reducio-DiT](https://arxiv.org/abs/2411.13552), which is a video generation method. Codebase available [here](https://github.com/microsoft/Reducio-VAE). |
|
|
|
|
|
## Model Details |
|
|
|
### Model Sources |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** [GitHub Repository](https://github.com/microsoft/Reducio-VAE) |
|
- **Paper:** [arXiv](https://arxiv.org/abs/2411.13552) |
|
|
|
## Uses |
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
|
|
Common use scenario is described [here](https://github.com/microsoft/Reducio-VAE). |
|
|
|
### Direct Use |
|
|
|
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. --> |
|
|
|
The model is typically used for supporting training a video diffusion model. After using this model to convert the data to the latent space, you can train your own diffusion model on the extremely compressed latent space. |
|
|
|
|
|
## Results |
|
|
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
|
|
|
|
### Results |
|
|
|
Metrics on 1K Pexels validation set and UCF-101: |
|
|
|
|Method|Downsample Factor|\|z\||PSNR |SSIM |LPIPS |rFVD (Pexels)|rFVD (UCF-101)| |
|
|---------|---------------------|------------------|------------|--------------------|--------------|----------------|------------| |
|
|SD2.1-VAE|1\*8\*8|4|29.23|0.82|0.09|25.96|21.00| |
|
|SDXL-VAE|1\*8\*8|16|30.54|0.85|0.08|19.87|23.68| |
|
|OmniTokenizer|4\*8\*8|8|27.11|0.89|0.07|23.88|30.52| |
|
|OpenSora-1.2|4\*8\*8|16|30.72|0.85|0.11|60.88|67.52| |
|
|Cosmos Tokenizer|8\*8\*8|16|30.84|0.74|0.12|29.44|22.06| |
|
|Cosmos Tokenizer|8\*16\*16|16|28.14|0.65|0.18|77.87|119.37| |
|
|Reducio-VAE|4\*32\*32|16|35.88|0.94|0.05|17.88|65.17| |
|
|
|
|
|
## Citation |
|
|
|
**BibTeX:** |
|
|
|
``` |
|
@article{tian2024reducio, |
|
title={REDUCIO! Generating 1024*1024 Video within 16 Seconds using Extremely Compressed Motion Latents}, |
|
author={Tian, Rui and Dai, Qi and Bao, Jianmin and Qiu, Kai and Yang, Yifan and Luo, Chong and Wu, Zuxuan and Jiang, Yu-Gang}, |
|
journal={arXiv preprint arXiv:2411.13552}, |
|
year={2024} |
|
} |
|
``` |
