Add PAG support to SD1.5

[yiyixuxu]

We recently integrated PAG into diffusers! See this PR [here] (#7944) we added PAG to SDXL

we also want to add PAG support to SD1.5 pipelines! we will need:

• StableDiffusionPAGPipeline (assigned to @shauray8, PR add PAG support for SD architecture #8725)
• StableDiffusionPAGImg2ImgPipeline add PAG support for SD Img2Img #9463
• StableDiffusionPAGInpaintPipeline
• StableDiffusionControlNetPAGInpaintPipeline (Add PAG support to StableDiffusionControlNetPAGInpaintPipeline  #8875)
• StableDiffusionControlNetPAGPipeline (assigned to @tuanh123789 )
• StableDiffusionControlNetPAGImg2ImgPipeline (assigned to @Bhavay-2001 add PAG support for SD Controlnet Img2Img #8864)

1. You should put it under the pag folder
2. you can use the implementation of SDXL PAG pipelines as a reference (see this PRhttps://github.com/add PAG support  #7944 and you can find all the sdxl pag pipelines here https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/pag)
3. you need to add AutoPipeline so that you can use this API to create it

      AutoPipelineForImage2Image.from_pretrained(repo_id, controlnet=controlnet, enable_pag=True ...)

4. tests and docs

If you are interested in working on this, Let me know which pipeline(s) you want to work on:)

[blaisedmello]

Hello, I would like to have a go at it. Can I work on the StableDiffusionPAGInpaintPipeline?

[yiyixuxu]

@blaisedmello sure!

[shauray8]

Looks rather interesting, adding support for StableDiffusionPAGPipeline.

[sajadn]

Hey, I'll work on StableDiffusionPAGImg2ImgPipeline

[tuanh123789]

Hi, can i take StableDiffusionControlNetPAGPipeline

[yiyixuxu]

@tuanh123789 sure!

[Bhavay-2001]

Hi @yiyixuxu, I think I can work on StableDiffusionControlNetPAGImg2ImgPipeline. Pls assign this to me.

[yiyixuxu]

@Bhavay-2001 sure!

[AbhinavJangra29]

@yiyixuxu can i try ?

[darshil0805]

Hi, I'd like to work on StableDiffusionPAGInpaintPipeline if it's still open

[yiyixuxu]

@darshil0805 sure!

[darshil0805]

Hi @yiyixuxu , @a-r-r-o-w. I have opened a PR for SD Inpainting PAG Pipeline, can you review and help me to test this pipeline?

[crapthings]

how do i know the layer names?
only
up mid down?

[yiyixuxu]

cc @a-r-r-o-w, maybe we should update doc? (did we?)

[a-r-r-o-w]

how do i know the layer names?
only
up mid down?

The correct usage is to look at either the modeling code, or state dict keys, to see what the layer naming looks like. Then you can specify either the full layer names or partial layer names as valid regular expressions when initializing the pipeline (via pag_applied_layers). For more advanced usage, you can take a look at helper methods in the PAGMixin.

See the docs and examples usages here.

TLDR; Our naming convention follows the pattern of something_blocks.whatever or something_blocks.[layer_index} or blocks.{layer_index} or transformer_blocks.{layer_index}, so the easiest way to enable them without looking into the state dict or modeling code is to specify something like pag_applied_layers="blocks.(10|20|42)" or pag_applied_layers="mid"

cc @a-r-r-o-w, maybe we should update doc? (did we?)

I think we did, but I noticed that the introduction docs are out of sync with the current usage, and I'm not sure if the current example works. I'll test and open a PR soon

[airookie17]

HI @yiyixuxu I'd like to work on StableDiffusionPAGImg2ImgPipeline if it's still open. Thanks

[yiyixuxu]

@airookie17
I think I'm about to merge in this #9463