NerdyRodent

Image-2-Video with Cosmos in ComfyUI!

Added 2025-01-19 15:51:01 +0000 UTC

Video - https://youtu.be/fjUdzKEIypk

ComfyUI now supports some of the Nvidia Cosmos models, which means image to video fun at home, for free, on your own computer :)

Ok, so it's not as good as the pay-for websites, but it is still a lot of fun with plenty of potential in this first release.

It is fairly slow and needs 12+ GB VRAM to run, but it does seem to have a better license than other models.

Both Image-2-Video and Text-2-Video and possible, with each of the 7B models weighing in at around 14GB each.

Links

The blog has lots of useful and interesting info, with Comfy examples having the example workflows. Nvidia's video on Cosmos is well worth a look, and if you need ollama then that's there too!

Blog: https://blog.comfy.org/p/comfyui-now-supports-nvidia-cosmos

Example: https://comfyanonymous.github.io/ComfyUI_examples/cosmos/

Omniverse training data: https://www.youtube.com/watch?v=9Uch931cDx8

Ollama: https://ollama.com/

Workflow Information

The attached workflows are numbered in order of appearance in the video. There are 3 different versions -

Standard Image/Text to video - set the switch to pick which latent to use and select a model. Video2World model for Image-2-Video, and Text2World for Text-2-Video.
LLM option - As standard, but with an llm option. You'll need ollama running with your choice of vision LLM. You don't have to use the image though, and could disconnect that to use a smaller language model, for example.
Multi - As standard, but used to create a looping video. Takes 3 input images, and saves outputs individually (A, B and C) to allowing independent scene modifications. Stitch the pieces together in your favourite video editor, add some sound and enjoy!