Skip to content

Commit

Permalink
[Community] ControlNet Reference (#3508)
Browse files Browse the repository at this point in the history 
add controlnet reference and bugfix

Co-authored-by: Patrick von Platen <[email protected]>
  • Loading branch information
@okotaku @patrickvonplaten
okotaku and patrickvonplaten authored May 23, 2023
1 parent edc6505 commit b134f6a
Show file tree
Hide file tree
Showing 3 changed files with 900 additions and 24 deletions.
51 changes: 49 additions & 2 deletions examples/community/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1324,7 +1324,7 @@ image.save('tensorrt_img2img_new_zealand_hills.png')

### Stable Diffusion Reference

This pipeline uses the Reference only Control. Refer to the [sd-webui-controlnet discussion](https://github.com/Mikubill/sd-webui-controlnet/discussions/1236).
This pipeline uses the Reference Control. Refer to the [sd-webui-controlnet discussion: Reference-only Control](https://github.com/Mikubill/sd-webui-controlnet/discussions/1236)[sd-webui-controlnet discussion: Reference-adain Control](https://github.com/Mikubill/sd-webui-controlnet/discussions/1280).


```py
Expand Down Expand Up @@ -1365,6 +1365,54 @@ Output Image of `reference_attn=True` and `reference_adain=True`

![output_image](https://github.com/huggingface/diffusers/assets/24734142/3c5255d6-867d-4d35-b202-8dfd30cc6827)

### Stable Diffusion ControlNet Reference

This pipeline uses the Reference Control with ControlNet. Refer to the [sd-webui-controlnet discussion: Reference-only Control](https://github.com/Mikubill/sd-webui-controlnet/discussions/1236)[sd-webui-controlnet discussion: Reference-adain Control](https://github.com/Mikubill/sd-webui-controlnet/discussions/1280).


```py
import cv2
import torch
import numpy as np
from PIL import Image
from diffusers import UniPCMultistepScheduler
from diffusers.utils import load_image

input_image = load_image("https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png")

# get canny image
image = cv2.Canny(np.array(input_image), 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetReferencePipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
controlnet=controlnet,
safety_checker=None,
torch_dtype=torch.float16
).to('cuda:0')

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)

result_img = pipe(ref_image=input_image,
prompt="1girl",
image=canny_image,
num_inference_steps=20,
reference_attn=True,
reference_adain=True).images[0]
```

Reference Image

![reference_image](https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png)

Output Image

![output_image](https://github.com/huggingface/diffusers/assets/24734142/7b9a5830-f173-4b92-b0cf-73d0e9c01d60)


### Stable Diffusion on IPEX

This diffusion pipeline aims to accelarate the inference of Stable-Diffusion on Intel Xeon CPUs with BF16/FP32 precision using [IPEX](https://github.com/intel/intel-extension-for-pytorch).
Expand Down Expand Up @@ -1462,4 +1510,3 @@ latency = elapsed_time(pipe4)
print("Latency of StableDiffusionPipeline--fp32",latency)

```

Loading

6 comments on commit b134f6a

@reimager
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@okotaku Do you know which version of reference controlnet you based this off? 1.1.173? lllyasviel has made many fixes, specifically for the controlnet_reference pipeline which probably need to be ported.
The current diffusers pipeline suffers from instability and collapse, which has since been fixed in the sd-webui-controlnet repo

@okotaku
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reimager I implemented it based on v1.1.178.
Could you share input images and outputs of diffusers and controlnet?

@reimager
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@okotaku
Oh interesting.

Let me try and reduce it to a simple script and do a comparison between that an the sd-webui-controlnet version.
I will open an issue with the script and diffusers vs a1111 and tag you

Thank you for creating this pipeline!

@learningyan
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have you tried it ? I try some images and use StableDiffusionReferencePipeline to generate some images, but the results seems bad ....

@learningyan
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@okotaku Oh interesting.

Let me try and reduce it to a simple script and do a comparison between that an the sd-webui-controlnet version. I will open an issue with the script and diffusers vs a1111 and tag you

Thank you for creating this pipeline!
have you tried it ? I try some images and use StableDiffusionReferencePipeline to generate some images, but the results seems bad ..

@okotaku
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@learningyan #3566
Please discuss on this issue.

Please sign in to comment.