Papers
arxiv:2406.16863

FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models

Published on Jun 24
· Submitted by MoonQiu on Jun 26

Abstract

Diffusion model has demonstrated remarkable capability in video generation, which further sparks interest in introducing trajectory control into the generation process. While existing works mainly focus on training-based methods (e.g., conditional adapter), we argue that diffusion model itself allows decent control over the generated content without requiring any training. In this study, we introduce a tuning-free framework to achieve trajectory-controllable video generation, by imposing guidance on both noise construction and attention computation. Specifically, 1) we first show several instructive phenomenons and analyze how initial noises influence the motion trajectory of generated content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that enables trajectory control by modifying noise sampling and attention mechanisms. 3) Furthermore, we extend FreeTraj to facilitate longer and larger video generation with controllable trajectories. Equipped with these designs, users have the flexibility to provide trajectories manually or opt for trajectories automatically generated by the LLM trajectory planner. Extensive experiments validate the efficacy of our approach in enhancing the trajectory controllability of video diffusion models.

Community

Paper author Paper submitter
•
edited Jul 4

Want to enlarge/control the movement of pre-trained video generation models? Try FreeTraj!!!

FreeTraj is a tuning-free method for trajectory-controllable video generation based on pre-trained video diffusion models.

Project Page: http://haonanqiu.com/projects/FreeTraj.html
Code Repo: https://github.com/arthur-qiu/FreeTraj

Paper author Paper submitter
•
edited Jun 26

Hi @MoonQiu congrats on this work!

I see you've published the model at https://huggingface.co/VideoCrafter/VideoCrafter2, would be up for linking it to this paper page? See here on how to do that: https://huggingface.co/docs/hub/en/model-cards#linking-a-paper.

Also, download counts won't work for your model, here's a guide to make them work: https://huggingface.co/docs/hub/models-uploading#upload-a-pytorch-model-using-huggingfacehub

Update, looks like you're using the VideoCraft model which is authored by different people, will ping them.

·
Paper author

Hi, we use VideoCrafter2 as a pre-trained video generation model, but this model is not proposed by our method.

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2406.16863 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2406.16863 in a dataset README.md to link it from this page.

Spaces citing this paper 1

Collections including this paper 2