Tell us what is missing in ControlNet Integrated

[lllyasviel]

The integrated ControlNet is not updated for a while, and we are going to make it a bit more up-to-date.

However that will happen after some other newer experiments. Old features are relatively lower priority - so that we can experiment with more new ideas first.

I personally know some models like uni-controlnet (or called promax by some non-research people) and some preprocessors like DSINE/DepthAnything/etc. Those are very easy to add – after I get other things done, those only need me 1 or 2 hours to add them all.

However I will not add the below things:

1. webui's IPA multi-inputs implementation. The reason is that I find them misleading and technically wrong. One can just use multiple ControlNet Units to have similar logic. The “multi-inputs” implies some feature mixture techniques when it is not. So it is wrong and I cannot add it. (but will consider some technically correct multiple IP mixture later if flexible
2. Automatic model filtering and ipadapter preprocessor recognizing. I find those things very annoying since model filtering is somewhat bind to model load and is very laggy. Also I cannot not know from UI what are the underlying rules and how those automatic recognizing works (and if they worked after all) when I click generate.
3. All things that I find hard to understand – if I cannot quickly understand what it is then most people will not understand it.

By the way, several experiments will be implemented in a “Forge Space” (that I will add later, similar to some local HuggingFace Spaces) rather than that ControlNet extension. So many functionalities will happen elsewhere.

This post is only for controlnets. Experiments with other newer diffusion models will happen before controlnets.

Please tell us what is missing in ControlNet Integrated. Please do not talk about things that are already mentioned in this post – I will take a look several days later when I finish some other things.

---

Update Sep 1:

The rewrite of ControlNet Intergrated will start at about Sep 29 (delayed) start at about Oct 15 (delayed) start at about Oct 30 (delayed) start at about Nov 20. (When this note is announced, the main targets include some diffusers formatted Flux ControlNets and some community implementation of Union ControlNets. However, this may be extended if stronger models come out after this note.)

---

About IPAdapter:

Note that the CLIP Vision names of sd-webui-controlnet are non-standard. Forge uses correct names:

SD v.	IPadapter	Img encoder
v1.5	ip-adapter_sd15	ViT-H
v1.5	ip-adapter_sd15_light	ViT-H
v1.5	ip-adapter-plus_sd15	ViT-H
v1.5	ip-adapter-plus-face_sd15	ViT-H
v1.5	ip-adapter-full-face_sd15	ViT-H
v1.5	ip-adapter_sd15_vit-G	ViT-bigG
SDXL	ip-adapter_sdxl	ViT-bigG
SDXL	ip-adapter_sdxl_vit-h	ViT-H
SDXL	ip-adapter-plus_sdxl_vit-h	ViT-H
SDXL	ip-adapter-plus-face_sdxl_vit-h	ViT-H
v1.5	FaceID	InsightFace+CLIP-H
v1.5	FaceID Plus	InsightFace+CLIP-H
v1.5	FaceID Plus v2	InsightFace+CLIP-H
v1.5	FaceID Portrait	InsightFace+CLIP-H
SDXL	FaceID	InsightFace+CLIP-H
SDXL	FaceID Plus v2	ViT-H

[sdbds]

There are 3 new controlnet-based architectures to try:

Update in Oct 15

ctrlora
Base ControlNet along with condition-specific LoRAs on base conditions with a large-scale dataset. Then, our Base ControlNet can be efficiently adapted to novel conditions by new LoRAs with as few as 1,000 images and less than 1 hour on a single GPU.

only SD1.5 for now.

AnyControl

One input with multiple preprocessor inputs

ControlNeXt

Modified variants of some blocks that seem to perform well on SVD and SDXL.

[sdbds]

update for ctrlora

[toyxyz]

Some Ctrlora training test!

Ctrlora actually works very well with small datasets.

[reaperhammer]

Did you train new models from Ctrlora base? This looks cool if I can sketch to image with stick people!

[BNP1111]

instant id+inpaint PLZ

[LeDXIII]

First of all I want to say - thanks for the Great tool. That the previous version was very good, that the new version looks very promising (judging by commits).

Regarding controlnet.

1. Already realized implementations. I'll just walk through them briefly to avoid forgetting anything.

• Anyline.
• A set of controlnets from Xinsir. In addition to Union (which has problems in realizing openpose in Automatic), the author has Tile, which also works in the Automatic controlnet not quite as expected.
• Ip-adapter compositions.
• I would also like to see SAM (2?), lama cleaner in controlnet in preprocessors.

2. From the new stuff.

• Photomaker 2. In the demo it shows very good results on face repeatability.
• CustomNet. Not sure how good it is.
• Controlnet of SD3 and other new models.
  Thank you.

[IPv6]

A feature request: segmentation tagging control net (not yet existing)

Inputs: character in default frontal pose (photo), a segment map where segment color defines clothing region in frontal pose, a segment map with same colors but in target pose.

Desired output: ControlNet should map clothing features from default pose into target pose using color-coded segments as strict guidance. That means - without adding features not present in segment in default pose, without symmetry breaks and so on. Probably specific color channel should hint about SYMMETRY for robustness. Target pose should not be restricted to "usual poses", theoretically upside-down poses, partially occluded poses, self occlusions can be resolved just fine by such 1-1 segmentation mapping

[ali0une]

The outpaint of xinsir union-controlnet-promax ... if i could outpaint with SDXL it would be wonderful.

Kepp up the good works ...

[dongxiat]

I hopefully update controlnet will work with 2 major extension :
segment-anything
replacer
its help my job alot...and want to use with in webui-forge

btw ur works is awsome

[andupotorac]

I would like to add to this. Don't include just SAM2, but also pair it with Florence as it makes for detecting areas much better.

1. https://huggingface.co/spaces/SkalskiP/florence-sam

[jordanjay29]

The biggest need here might simply be a document outlining how Forge's built-in CN differs in options and behavior. It's not much, but the will-not-adds and the workarounds to some of those would be useful for newcomers and converts from A1111.

[AxelFar94]

First of all, Thank you for your work lllyasviel.
I would like to point out that the tile_colorfix and tile_colorfix+sharp are making the images bleed their colors in a checkered pattern, and this happens since Forge launch, here some comparisons with Auto 1111 results using low ControlNet strength:

The checkered pattern is more noticeble in very low strength:

Would be great if you could fix it! It's a great way to make images strictly follow input image colors.
Thanks!

[GeminiSquishGames]

Well, I can't get it to work in the Photoshop Plugin/Extention... that's missing for me, but probably not something that can be helped since it's pretty much not getting updates on their end. I'd probably rather have it work faster in the UI than have a poor implementation in photoshop anyway at this point. Looking forward to seeing "The Chosen One" for character consistency soon as a CN model or extension of some kind.
https://omriavrahami.com/the-chosen-one/

[Crousere]

Fooocus sdxl inpaint. Inpaint_v26.fooocus.patch never worked as well on Forge for me as it did on Foocus.

[newxhy]

Inpaint_v26.fooocus.patch 在comfyui 上的效果也不如在 Foocus 上。
The effect of Inpaint-v26.foocus.patch on comfyui is not as good as on Foocus.

[andupotorac]

1. For Depth, I think it's best to look at https://depthfm.github.io/. Seems better than even Marigold.

2. Hand and feet fixing. For hands there are several approaches, and I believe the same algos could work on feet also. I'm not too technical to know, but I posted some of those data here: Can this be used for feet too, with datasets like FOOT3D and FIND? agnJason/XHand#2.

For hands you also have:
https://github.com/wenquanlu/HandRefiner

And this is looking interesting as well:
https://github.com/adventurer-w/ClickDiff

I've seen people add manual images to OpenPose, but ClickDiff does it automatically, and the other do too. Would be great if we can finally fix hands and feet with ControlNet.

3. There are some options you could add on the models to make them more robust:

• https://shariqfarooq123.github.io/loose-control/ (more flexibility for shapes)
• https://github.com/liuxiaoyu1104/SmartControl (takes context into account for better generations)
• https://github.com/open-mmlab/AnyControl (like multicontrol net, but better quality)

4. Would be nice to integrate LivePortrait (for humans and animals too) so we can have a ControlNet for emotions. :)

[softtaco1]

1.) SDXL / Pony inpainting would be amazing.

2.) A custom ControlNet which utilizes either photos or prompts to transfer outfits/clothing onto a generation. This could maybe be enhanced by utilizing layerdiffuse to overlay it correctly upon the subject through multiple passes. (Idea, not sure if could be implemented)

3.) I don't know if this is possible either, but instead of something like regional prompter or Forge Couple, creating a controlnet that instead recognizes separate individual characters interacting. I know there are things similar to this that exist, but they aren't quite what I'm looking for to generate multiple characters interacting easily while maintaining a level of versatility and not being constrained to one specific pose via OpenPose. If this idea is at all possible, a controlnet model could utilize a reference image to apply an interaction/action between multiple characters. Or instead of a reference image, it could utilize a syntax similar to Forge Couple's "NEWLINE" , but instead of generating in a new space of the 2d plane, applies character adherence to subject(s) in the scene. Then, an action could be specified, detailing how these characters are interacting. Basically, a prompt adherence helper for models that struggle with prompt adherence, especially as it relates to characteristics of specific individuals in a scene.

Example:
A dark cave
NEWCHAR A frightened middle-aged man, yellow poncho, soaking wet, holding a lantern
NEWCHAR An eldritch horror monster, devilish grin, glowing eyes.
ACTION Fighting

Again, not sure if this is even possible without a multimodal model, but I think it would be more useful to have something that helps ensure characters look a certain way than it is to make sure they are in a specific place/pose (most of the time.)

[pflky]

For me, what's missing is

1. Union ProMax model support
2. Dsine preprocessor support in Forge (most of the time it's better than all the other preprocessors, including depth and canny)

2a. Dsine on independent controlnet for A1111 has a bug that if you use a different resolution/ratio than the image, the preprocessor will generate zoomed into the image rather than crop or resize. Its been a bug since the release of Dsine as a preprocessor. This happens regardless of settings. You essentially have to format images to be the same exact resolution and ratio as the preprocessor settings, otherwise it zooms in, which can be a pain if you're working with images of mixed resolutions.

2b. Propose a feature based on the last problem: A new feature that allows people to use a draggable and resizable selector box over the controlnet image to determine the focus area of the preprocessor. I'd imagine this would also be useful for other controlnet preprocessors too, as you can then take any part of any image and make it the focus of the preprocessor on the fly. So for example, you drag a 1:1 ratio box overlay on the image, have it be resizable, and then it'll use the inputted resolution, even if the image itself is smaller; So you could zoom into a face on a 512x512 image and make just the face area preprocess at 1024x1024. Simple and versatile way of handling controlnet inputs of varying resolutions and aspect ratios and fills the gap that "just resize" and "crop and resize" doesn't cover.

3. Photomaker V2 support (faceid model)
4. DepthAnything V2 support (this outperforms marigold)
5. With the advent of Union models, the overhead for using multiple controlnet preprocessors should be smaller, because you should only need to load one model for multiple control units. I don't know if this is the current behavior, but if not, then if the same model is selected across different units, it should only be loaded into memory once.
6. There is a newer lineart preprocessor called "Anyline", which seems to create better preprocessing for lineart models. https://github.com/TheMistoAI/ComfyUI-Anyline
7. The ability to use extensions that rely on A1111 non-built-in controlnet (if possible). Some of these support things that Forge controlnet does not, like DepthAnything V2 for example. So, some kind of scheme that allows extensions that call on A1111 controlnet, to play nicely with Forge controlnet. It's really one of the only reasons I keep going back to main A1111. Forge has the memory management enhancements, but a lot of stuff gets made for A1111 that doesn't play well with Forge's built-in controlnet. So if it's possible to make things that call on A1111 controlnet to convert to use the built-in controlnet in Forge, that could solve a lot of problems and overhead, including having to go through the trouble of supporting certain new models.
8. "Contrast" feature for post-processing preprocessor results. For example, I often find things can get too blended with depth preprocessing images, where the white parts lose detail because of the depth estimation. The detail is technically still there, just in a very limited visual dynamic range. If you could essentially add contrast on the fly, you could change the preprocessor image to be more contrasty, allowing those details to be more visible, along with being more visible to the model itself. So for example, a simple contrast slider in controlnet that can apply an adjustment to the preprocessor image before it's plugged into the controlnet model. It can be just a little vertical bar beside the image that has like 5 stops on either end, so 0 as default and +5/-5. I think this could be useful for a variety of controlnet preprocessors, This should not apply to the base image, but the preprocessed image, so it should probably be in the preprocessor preview window, and maybe only visible if someone has the "allow preview" option activated. Users can then look at the preview to determine if they want to add or reduce contrast, and whatever setting is active when generation is started, is processed with that contrast on the fly and then sent off to the controlnet model.

[Satochiju]

Illyas, I'm very new to this and probably the most ignorant of all those involved here, but that doesn't stop me from telling you that I admire not only your great work in Forge, which by the way has been very useful for me as a graphic designer, but, above all, your selflessness. I'm sure that if there were more people like you in this world, things would be different, as we say in Chile. Thank you very much!! A big hug.

[Satochiju]

Ah! y lo nuevo de Forge funcionando muy bien en mi 3060ti, algo lento aún pero con muy buena calidad, Flux y SDXL

[liquid150]

You should volunteer to build it if you want to whine about it. This is FREE software. Oh, you can't code the python? Then get over it.

…

On Thu, Nov 21, 2024, 1:53 PM CCpt5 ***@***.***> wrote: Where is this sourced from? That's extremely disappointing if so. Is there not another developer who can assist? 3months is an eternity in terms of generative media. No disrespect of course to lllyasviel - he's a genius - clearly in high demand if that feb date is the next possible update date. — Reply to this email directly, view it on GitHub <#932 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BFC6YP4Q5XTWUBT2WF5HDUT2BY23JAVCNFSM6AAAAABMAFM6XCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZUGA3TCNI> . You are receiving this because you commented.Message ID: <lllyasviel/stable-diffusion-webui-forge/repo-discussions/932/comments/11340715 @github.com>

[CCpt5]

I wasn't rude at all, so stop misreading and acting all tough.

[liquid150]

Yeah, actually, you were rude. If you're not volunteering stop complaining.

…

On Thu, Nov 21, 2024, 2:07 PM CCpt5 ***@***.***> wrote: I wasn't rude at all, so stop misreading and acting all tough. — Reply to this email directly, view it on GitHub <#932 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BFC6YP5AWMX3ZBOFZVHGBBT2BY4QHAVCNFSM6AAAAABMAFM6XCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZUGA4DIOA> . You are receiving this because you commented.Message ID: <lllyasviel/stable-diffusion-webui-forge/repo-discussions/932/comments/11340848 @github.com>

[CCpt5]

You don't have any activity on your GitHub account, already have 3 downvotes on your initial quote of my post, and again I disagree.

I asked what the source of the screenshot was. There's no "whining". This is a discussion forum for adults and I don't think anything I said (which you quoted) is 1) offensive to the developer, 2) demanding. Request was for clarity on the image.

We disagree, and I don't like YOUR tone. So have a great day kidoo.

[ScionoicS]

The delayed rewrite coincides with the release of new tools I see. Perhaps there were insights provided by BFL in private when the delays were decided on.

Take your time to do it right guys. All the time you need.

Delays are a good thing. Do em more.

[blacklig]

Thing is, someone is waiting for this update (me included) and I am still waiting.. If I knew at beginning it would be so massively delayed, I would just switch my backend to comfy and vioala Forge.. But he delays it like +20 days, +15 days etc, so I am always like.. argh, I will wait it because I have all backend on Forge, so I am not that happy to rewrite it to comfy, but those continous delays already added pretty big pile of days where if I knew it on beginning it would take so long I would rewrite it and went comfy way.

But I know @lllyasviel is doing this in his free time and he is legend I appreciate his work very much and he is in no way obligated to deliver something. Just those continous delays are making situation bitter... moreover when he wrote somewhere that implementation of CN Union is like 1-2 hours for him :D so I am like, come on.. I am waiting for it 2 months and still not delivered..

whatever :)

I will give it a one more week and then really switching to comfy as it looks like they are shipping new things light fast

[blacklig]

@ScionoicS why some people have such a big problem understanding written text? but then somehow cannot resist urge saying shits like "you can write it yourself"... oh, dont you say.. thank you for your advice. My point was something else.. if you are continously saying wait 10 more days, wait 10 more days etc, that just sux, even if it is from volunteer. You know what, I will rewrite it, but wait til 5.12.... then I will see if I have time on that... and like that I can be postponing it till forever and everyone would be happy and giving me respect, yeah...

[ScionoicS]

Understanding the "written text" as if the writing is sacred?

I understand it from my perspective. Seeing people upset about free work being given to them.

No one here is the manager of the project except for those doing the work. No one here is a boss except for the people doing the work.

Be your own boss. Live the dream.

[ScionoicS]

The problem starts with the complete sole influence on a project, where everything is dependent on one man.

You're welcome to fork it. Anyone can have influence on the project. This "problem" you're so fixated on is really not a problem. It's just you reverse reasoning your own problems with waiting.

[JackDainzh]

The problem starts with the complete sole influence on a project, where everything is dependent on one man.

You're welcome to fork it. Anyone can have influence on the project. This "problem" you're so fixated on is really not a problem. It's just you reverse reasoning your own problems with waiting.

Not everyone is a programmer or at least can read code. You assume that everyone can do it, or anyone who is "upset" with the lack of proper management in favour of accessibility will go and do it themselves. It's not about impatience, it's about addressing structural issues in the project’s development process. Community feedback like this should be taken as constructive criticism, not dismissed as personal impatience nor disregarded by saying "you should just fix it yourself".

[ScionoicS]

You're welcome to pay anyone to develop it for you. The code base is open and free to modify.

You're just not willing to have influence outside of manufacturing problems rooted in your own selfish desires. Making up problems has never worked to influence the direction of FOSS. That's why Microsoft bought github instead.

[gabrielxcom]

Are we there yet? 🙃

[reaperhammer]

[jajajajajajajajajajajajajajajajajajajaj]

I have come to the conclusion, all 3 of you are stupid

[JackDainzh]

Fair game, just fork the repo with custom backend code and spend 5 years debunking it. Or wait 5 years without doing anything. OR move to the comfyui community with "toxic" community. It's your choice. I put knorr in my forge ui, it's my choice really

[ScionoicS]

"I dont understand so i'll call you stupid"
QQ

If you have to spend 5 years working out a project, you're not the man for the job. Hire someone to do it for you then. Oh , you want it for free? Oh you better wait for the volunteer who is doing it for free to do it then. Best to pay respect at least. You couldn't be so cheap that you can't even afford that, could you?

[JackDainzh]

You are abnormaly fucking stupid

[ScionoicS]

You're incapable and lashing out. Your copium might be contaminated with lead sir.

[Gushousekai195]

He's up to something....

[tazztone]

probably been cooking all this time and only now pushing the masterwork code to repo 🚀

[Gushousekai195]

Oh I do hope so

[Gushousekai195]

Oh my God! What has this thread become?! 😢

[SunGreen777]

I have not read everything, the author of the project is a good man, I believe that everything will be, probably the commenter is dissatisfied with the lack of information. I am patiently waiting, but sometimes it seems to me that the project has not developed for a couple of months. Against the background of the disappeared A1111, the silence is really scary. Some updates are coming, but there is nothing radically new.
I created a topic in a more positive way with a question to the author to tell about the plans.
And actually, for me, it’s important to either stay in ForgeUi and wait or master InvokeAi because Comfy is not about creativity but about experiments and comparisons.

[MixonUP]

What happened to the a1111

[rbrtwtrs]

Forge expanded on A1111 and there was no longer any real reason to stay with or update A1111. Biggest improvement being speed.

[MigCurto]

Maybe he has become fed up,? A lot of self imposed work for little or no recognition, know I would: #2151

It's not like there aren't alternatives, people just don't like to change, and to quote a game developer: " many when given a pile of gold all they can do is complain is too heavy"

EDIT: And its not like he is sitted on his hands, recognize the app background? should be be coming to Forge: https://github.com/lllyasviel/IC-Light

[ScionoicS]

I wouldn't put it past him. The other free options on the block, Comfyui and invoke, both are for profit companies that are losing money until they're popular and can monetize that fame.

They're likely both engaged in social engineering against this project. Comfy anon literally accused him of stealing his open source code, when comfyui is a lot of borrowed code itself.

All of it is shenanigans. I would not blame lllyasviel if he hung it all up. Bobby Fischer style.

[MigCurto]

Those arent the only options thought, if you dont like Comfy way of working you can give McMonkeys SwarmUI a go, its like a Comfy Frontend but very actively developed, If you're on a Mac Draw things is actually a viable option. Im sure there are others I havent tried, I do like Forge, hope it stays relevant, and would much rather use it for a lot of tasks.

[rbrtwtrs]

I have tried SwarmUI and while I can see how it tries to lessen the pain of Comfi it didn't do it for me. Just another layer on top. I am surely biased having used Forge for the last year but can and do give alternatives a fair try (except Apple products ;)

[Haoming02]

recognize the app background?

Just want to say that this is just how Gradio apps look

[mel870]

Hopefully Forge is updated at some point. I personally prefer it like others here. Seems I get better quality images especially with the Forge Realistic samplers. Not sure what these samplers are, but when comparing to other's in Forge or Comfy, I get the best Flux prompt adherence using it. However, I also use Comfy and Swarm since they are updated often. Plus, it's best to stay diversified just in case one app stops being updated.

[yamfun]

XYZ plot items for ControlNet values

[softtaco1]

Ooh I like that idea

[Gushousekai195]

@lllyasviel

[Hzr1324]

Let him or her cook.

also I would love support for Xinsir SDXL models :)

[wzwow]

xinsir model works in forge

[IPv6]

xinsir model works in forge

xinsir union (promax etc) are not working properly, Forge do not support setting controlenet type for union model, unfortunately

[wzwow]

the non union model seem to be working nicely

[Gushousekai195]

What does this mean?! What is "lll?" 😵‍💫

[Gushousekai195]

WELP NEVERMIND

I hate my life

[JinJomes]

They talking about 3yasviel. Different thing. AMD & mac only.

[ostap667inbox]

WELP NEVERMIND

And great. And good. Something's got to stay stable in these crazy times.

[Lexxxco]

Invoke 5.5 just introduced support for Flux Controlnet Canny and Depth. Maybe collaboration with them can help to integrate Flux Controlnet to Forge quicker?

[Gushousekai195]

@lllyasviel