Image-2-Video with Control!
Added 2025-01-25 18:09:41 +0000 UTCVideo - https://youtu.be/2w_vlTFyntI
Overview
EasyAnimate have released their new v5.1 models giving us a variety of control methods such as pose, scribble, depth, trajectory and even camera control. Even better, it all looks to be under an Apache 2.0 license, making it the latest truly open source release.
To fit everything into VRAM you will need at least 16GB, though MS Windows users can always opt to "go slow" with Nvidia cards being able to use system RAM as shared memory.
TeaCache is built-in, so generation time can be at little as 2 minutes on a 3090.
Example download commands
If you don't have huggingface-cli installed as yet, you can do so with:
pip install -U "huggingface_hub[cli]"
To download the models:
huggingface-cli download alibaba-pai/EasyAnimateV5.1-12b-zh-InP --local-dir models/EasyAnimate/EasyAnimateV5.1-12b-zh-InP
huggingface-cli download alibaba-pai/EasyAnimateV5.1-12b-zh-Control --local-dir models/EasyAnimate/EasyAnimateV5.1-12b-zh-Control
huggingface-cli download alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera --local-dir models/EasyAnimate/EasyAnimateV5.1-12b-zh-Control-Camera
The optional smaller text_encoder:
huggingface-cli download Qwen/Qwen2-VL-7B-Instruct-GPTQ-Int8 --local-dir models/EasyAnimate/text_encoder_int8
Symbolic links
The files are huge and, like me, you may not want to keep three copies of the text_encoder directory which are all identical. The vae also looks to be the same in case too, though that is only 1GB.
To save a little space, you can use symbolic links. There are like "pointers" to data, rather than copying everything again.
Using Linux, the link command is "ln -s", though I'm using the Caja file manager on Ubuntu, meaning a symbolic link is just a right-click away for me ;)
For a great breakdown of various options for creating symbolic links using Microsoft Windows, take a look at this complete guide - https://www.howtogeek.com/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/
Links
ComfyUI README - https://github.com/aigc-apps/EasyAnimate/blob/main/comfyui/README.md
HuggingFace readme - https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-InP/blob/main/README_en.md
Qwen VL Int 8 - https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct-GPTQ-Int8
Workflows
A bit tricky to modify these workflows given their nature, but I've coloured the various sections, added groups, and changed the Chinese nodes to English for the examples used in the video.
The attached workflows are again numbered in order of appearance in the video.
