Flux.1 + Lora + Controlnet!
Added 2024-08-13 20:20:28 +0000 UTCEven more Flux.1 updates - workflow attached as usual :)
If you've not updated ComfyUI in the last 16 hours, then you won't yet have the new ControlNet support, so be sure to update now :)
Loads of other updates top the workflows:
Optimised node placement so that bypassing IMG2IMG is quicker
Added More notes for all the various options
Added ControlNet - needs CFG 4! Switch back to ~2.1 when not using ControlNet
Added Lora
Added device choice nodes from ComfyUI Extra Models so you can pick devices for the VAE & CLIP. Running CLIP in CPU mode is the same as with the Pixart loaders now, so you can save a little VRAM for very little performance loss. Forcing the VAE to GPU makes the VAE operations a bit faster too
Added optional "magic" negative added with bypass switch
Added optional text box. Handy for forcing things like in the example workflow screenshot. There the denoise is set to 1, so it's only using the controlnet for the image shape - the rest of the description comes from Florence. Adding "Professional, cinematic photo. " as prefix text is a quick and easy way to change certain aspects of the Florence desdcription.
Changed default for the Download Florence Model to eager, which may be better supported on things like Microsoft Windows. I can't really see much difference between any of the choices, so just use whatever you prefer.
Update
Added a TXT2IMG version for those who never want to use images :)
Comments
Ok cool. Maybe something isn’t agreeing with my dependencies. I’ll give it a new start. Cheers!
Michael Childress
2024-08-21 18:48:28 +0000 UTCThat doesn't sound good! If you've got any errors as ComfyUI starts up, it's worth checking those. Firing up a new Comfy environment is also a great way to get a "known good" setup. I checked https://github.com/heshengtao/comfyui_LLM_party/issues?q= and no-one else has logged an issue
Nerdy Rodent
2024-08-21 18:46:25 +0000 UTCYou can just delete the LLM party group and set the text inputs back to widgets, or use the basic workflows from https://www.patreon.com/posts/flux-is-awesome-109279424 :)
Nerdy Rodent
2024-08-21 18:43:53 +0000 UTCIm getting a segmentation fault after installing the llm party node. Havent figured out a work around yet.
Michael Childress
2024-08-21 16:31:08 +0000 UTCAnyone getting a segmentation fault after installing Comfyui_LLM_Party?
Michael Childress
2024-08-21 16:29:43 +0000 UTCIs it possible to tweak this to use just a Clip text encode for positive and negative without the LLM addin?, been playing and dissecting this but to no luck.
Rhellis
2024-08-21 06:07:56 +0000 UTCFor MS Windows users with performance issues, this discussion is probably worth a look - https://github.com/comfyanonymous/ComfyUI/discussions/4457
Nerdy Rodent
2024-08-18 17:29:04 +0000 UTCAnyone else this goes really show?
KING LIFER
2024-08-18 16:48:41 +0000 UTCFlux LoRAs don't work with NF4. Will try GGUF next.
Nibmeister
2024-08-18 15:23:47 +0000 UTCYay! :)
Nerdy Rodent
2024-08-17 18:46:54 +0000 UTCMost likely, my dear little rodent. I got everything up and running again after some few hours of pure joy reinstalling, downloading, configuring... I'll be more careful now to not break it again. =F
Lúcio Nunes
2024-08-17 16:26:52 +0000 UTCHm. May need the conflicting node to not be installed at all then :/
Nerdy Rodent
2024-08-17 10:56:24 +0000 UTCWhat do you mean compact. Judy not sure what you mean.
KING LIFER
2024-08-16 23:10:18 +0000 UTCUsing the Flux Compact fp16 model (combines clip+vae) which loads and runs faster and can use Flux LoRAs.
Nibmeister
2024-08-16 22:39:47 +0000 UTCI tried disabling it, but still I got the error. I'll reinstall everything from scratch. ^^
Lúcio Nunes
2024-08-15 23:55:37 +0000 UTCUnfortunately if you have something installed which conflicts with pythongosssss/ComfyUI-Custom-Scripts then you'll need to disable it to use the workflow.
Nerdy Rodent
2024-08-15 23:50:59 +0000 UTCI found this warning message here on node management: 【pythongosssss/ComfyUI-Custom-Scripts】Conflicted Nodes (1) ShowText|pysssss [ComfyUI-Universal-Styler]
Lúcio Nunes
2024-08-15 19:05:02 +0000 UTCGuys, I'm trying to use the workflow Flux_Dev_IMG2IMG_Florence_Llama_Lora_Ctrl_v10.json, but I'm getting some red circles in the nodes "pythongosssss/ComfyUI-C...", more specifically a red circle in the letter "a". How can I get this resolved?
Lúcio Nunes
2024-08-15 19:00:10 +0000 UTCI can see wanting multiple gens with the same ollama description, maybe appending some extra text to it, so I'd make the checkpoint dump at the end optional. Thanks for pointing the setting and node out; I was looking for a way to forcibly dump VRAM since occasionally my 10GB 3080 gets in the same mode where it only runs 87s/it... otherwise, typically I can get a flux schnell image in about 50sec. I'm really hoping nVidia increases their VRAM size options soon. I had a 4060 in my cart for prime day, but after some quick checks realized it would be a processing step-down to gain the 6GB VRAM; so I'm waiting.
James
2024-08-15 16:00:28 +0000 UTCYup, slow mode caught a lot of people out when the did it!
Nerdy Rodent
2024-08-15 07:59:55 +0000 UTCI didn't explicitly enable it, but that is the default behavior for me. If I exceed the total VRAM (16GB), shared VRAM kicks in and performance becomes abysmal. I suppose this is a Good Thing(tm), because sometimes you just peak over max VRAM and then some gets freed up and things go faster.
Michael TODD Seals
2024-08-15 01:46:56 +0000 UTCThanks NerdyRodent, looks good! Lots of options. ComfyUI-LLM_Party keeps getting better and better.
Lord Banzai
2024-08-14 22:08:37 +0000 UTCAs you're on MS Windows, do you have the "go slow" option enabled? https://nvidia.custhelp.com/app/answers/detail/a_id/5490
Nerdy Rodent
2024-08-14 21:16:06 +0000 UTCRan into some serious performance issues whenever I was using the LLM Party section of this workflow on my Win11 box (RTX4060 16GB VRAM, i71400K with 192GB RAM). Generation speed without the LLM enabled was 2.4s/it, and it jumped to ~87.8 s/it with it enabled. It was basically an issue with the LLM and Comfy fighting for/starving each other of VRAM. Solved the issue by doing two things: First: Created a user environment variable named OLLAMA_KEEP_ALIVE with value of 0. This causes OLLAMA to unload its model immediately after returning the response to the workflow. 2nd: Added a node PurgeVRAM from the LayerUtility custom nodes to the end of the workflow. Attached it to the output of the VAE decode in parallel with the Save Image node in your workflow. This force-unloads the checkpoint after the image has been created. After these changes are completed, I'm at 2.43s/it when flux is working, and an overall total image prompt execution time of 65.62 seconds. So I'm paying a penalty of about 10s/image by force unloading the models, but that is way better than the 15+ minutes I had before!
Michael TODD Seals
2024-08-14 18:40:27 +0000 UTCUsually under 20s for me, but that's on a 3090
Nerdy Rodent
2024-08-14 17:44:20 +0000 UTCNot 100% sure if the loras and controlnets work with NF4, but maybe? There are some new controlnet & sampler nodes to test as well. The joys of new things!
Nerdy Rodent
2024-08-14 17:42:46 +0000 UTCYup! You're missing the switches ;) If you're bypassing IMG2IMG & ControlNet, you'll need to remember to set the latent switch to empty and the prompt switch to user input. I've added a txt2img workflow for those who don't want to use images
Nerdy Rodent
2024-08-14 17:39:16 +0000 UTCThere are no conflicting nodes listen in ComfyUI manager, so it shouldn't be that. Some node packages could still potentially conflict though. The import error should give an indication of the issues in your Comfy environment.
Nerdy Rodent
2024-08-14 17:36:48 +0000 UTCSo no one else had issues with LLM party installation?
SouthbayJay
2024-08-14 15:55:27 +0000 UTCI have looked into this, and they claim that 16 GB is the maximum system RAM available. It's a Medion Erazer Beast X30. However, there are hints that it might be possible I might just give it a go, if the RAM isn't just soldered in.
TigerFox57
2024-08-14 10:45:02 +0000 UTCYou have a laptop with a 16gb vram card? Are you sure you can't just upgrade your ram? Must be possible. Sometimes these things are a little fussier than they need to be but beats spending over one or another couple of thousand. Right?
Asyouare
2024-08-14 08:59:41 +0000 UTCOK thanks. I feel another big spend is looming on the horizon...
TigerFox57
2024-08-14 08:26:28 +0000 UTCNo luck here yet. I think the workflow will need to be modified for the bypass to work since it has dependencies from other nodes.
Richard Gates
2024-08-14 05:32:29 +0000 UTCThe full Flux dev model takes 85s per iteration on my 3070TI but the fp8 model gets that down to about 6s/it.
Nibmeister
2024-08-14 02:35:47 +0000 UTC32 GB RAM using ollama with the 4.7 GB model.
Nibmeister
2024-08-14 02:32:50 +0000 UTCHi. May I ask you if you are able to use ollama locally, and if so how much system RAM do you have. I have the workflow running on my laptop, but I have to use Openai's gpt-4o-mini via their API, as my local ollama install stalls the workflow. The laptop has 16 GB system RAM and 16 GB VRAM.
TigerFox57
2024-08-14 02:14:48 +0000 UTCsame, any luck yet?
Cameron
2024-08-14 01:53:07 +0000 UTCI got it to work by using the windows exe
Tom Sveen
2024-08-14 01:24:32 +0000 UTCwow this took really long to render
KING LIFER
2024-08-14 01:24:25 +0000 UTCgot this working btw... brilliant setup. Ollama was surprisingly easy to get going... been beat up by all the python/env/etc past two years.
kc
2024-08-14 01:24:25 +0000 UTCTo my surprise, I got the workflow to work on my poor, underpowered 3070TI with 8 GB of VRAM. Now to convert it to the NF4 model, if that's possible.
Nibmeister
2024-08-14 01:11:18 +0000 UTCI'm seeing that if I disable the IMG2IMG group, it stops because ControlNet doesn't load since it is dependent on that image, and if you disable both img2img and controlnet then the workflow fails because you lose the prompt since it comes out of controlnet. Unless I'm missing something.
Richard Gates
2024-08-13 23:43:00 +0000 UTCI'm having issues with the LLM PARTY nodes.. Import failed for them, that happened last week also and I gave up trying to fix it but now we have another workflow using it so I think I really need help getting it fixed. Anyone else have issues with it? My comfy is all up to date as of an hour ago so that's not an issue...
SouthbayJay
2024-08-13 22:11:22 +0000 UTCSuper helpful! Have not tried or installed Olama yet. Do you reco simply using the Windows EXE from their git? Or is it simpler whereas I just download a model to a folder?
kc
2024-08-13 21:31:19 +0000 UTCThanks!
Brian Monarch
2024-08-13 21:11:38 +0000 UTCI'm not seeing much of a speed change from the controlnet myself? With ControlNet: FP8: 16/16 [00:11<00:00, 1.38it/s] FP8/CLIP FP16: 16/16 [00:11<00:00, 1.34it/s] Without ControlNet: FP8: 16/16 [00:12<00:00, 1.32it/s] FP8/CLIP FP16: 16/16 [00:11<00:00, 1.35it/s]
Nerdy Rodent
2024-08-13 21:09:48 +0000 UTCTrying this out, very cool! is there a faster workflow for this? \
Scott Ownbey
2024-08-13 20:43:44 +0000 UTC
