feat: new community mixture_tiling_sdxl pipeline for SDXL

[elismasilva]

What does this PR do?

This PR add a new community pipeline for SDXL support of Mixture-of-Diffusers, tiling version only. See original project: (https://github.com/albarji/mixture-of-diffusers). We already have pipeline for SD 1.5.

I already published a demo app for this:

For local reproduction

import torch
from diffusers import DPMSolverMultistepScheduler, AutoencoderKL
from mixture_tiling_sdxl import StableDiffusionXLTilingPipeline

device="cuda"

vae = AutoencoderKL.from_pretrained(
    "madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16
).to(device)

model_id="stablediffusionapi/yamermix-v8-vae"
scheduler = DPMSolverMultistepScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", num_train_timesteps=1000)
pipe = StableDiffusionXLTilingPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    vae=vae,
    scheduler=scheduler,
    use_safetensors=False    
).to(device)

pipe.enable_model_cpu_offload()
pipe.enable_vae_tiling()
pipe.enable_vae_slicing()

generator = torch.Generator(device).manual_seed(297984183)

# Mixture of Diffusers generation
image = pipe(
    prompt=[[
        "A charming house in the countryside, by jakub rozalski, sunset lighting, elegant, highly detailed, smooth, sharp focus, artstation, stunning masterpiece",
        "A dirt road in the countryside crossing pastures, by jakub rozalski, sunset lighting, elegant, highly detailed, smooth, sharp focus, artstation, stunning masterpiece",        
        "An old and rusty giant robot lying on a dirt road, by jakub rozalski, dark sunset lighting, elegant, highly detailed, smooth, sharp focus, artstation, stunning masterpiece"
    ]],
    tile_height=1024,
    tile_width=1280,
    tile_row_overlap=0,
    tile_col_overlap=256,
    guidance_scale_tiles=[[7, 7, 7]], # or guidance_scale=7 if is the same for all prompts
    height=1024,
    width=3840,
    target_size=(1024, 3840),
    generator=generator,
    num_inference_steps=30,
)["images"][0]

image.save("mixture_sdxl.png")

After published:

import torch
from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler, AutoencoderKL

device="cuda"

vae = AutoencoderKL.from_pretrained(
    "madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16
).to(device)

model_id="stablediffusionapi/yamermix-v8-vae"
scheduler = DPMSolverMultistepScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", num_train_timesteps=1000)
pipe = DiffusionPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    vae=vae,
    custom_pipeline="mixture_tiling_sdxl",
    scheduler=scheduler,
    use_safetensors=False    
).to(device)

pipe.enable_model_cpu_offload()
pipe.enable_vae_tiling()
pipe.enable_vae_slicing()

generator = torch.Generator(device).manual_seed(297984183)

# Mixture of Diffusers generation
image = pipe(
    prompt=[[
        "A charming house in the countryside, by jakub rozalski, sunset lighting, elegant, highly detailed, smooth, sharp focus, artstation, stunning masterpiece",
        "A dirt road in the countryside crossing pastures, by jakub rozalski, sunset lighting, elegant, highly detailed, smooth, sharp focus, artstation, stunning masterpiece",        
        "An old and rusty giant robot lying on a dirt road, by jakub rozalski, dark sunset lighting, elegant, highly detailed, smooth, sharp focus, artstation, stunning masterpiece"
    ]],
    tile_height=1024,
    tile_width=1280,
    tile_row_overlap=0,
    tile_col_overlap=256,
    guidance_scale_tiles=[[7, 7, 7]], # or guidance_scale=7 if is the same for all prompts
    height=1024,
    width=3840,
    target_size=(1024, 3840),
    generator=generator,
    num_inference_steps=30,
)["images"][0]

image.save("mixture_sdxl.png")

Final result

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@sayakpaul @yiyixuxu

[asomoza]

thanks, it looks fine and the results are nice, I noticed that the images are kind of stretched (like taller and thin subjects) but probably that's something from the original implementation.

I also tried it with a turbo model and it works well and it took lot less time.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[elismasilva]

I iam doing somes test using crops_coords_top_left in process, i think result is better:

with crop

without crop:

with crop

without crop

[asomoza]

yeah, those definitely look better, nice! Thanks for your contribution, failing test is not related.

[elismasilva]

yeah, those definitely look better, nice! Thanks for your contribution, failing test is not related.

tks, but i need commit one change on pipeline to work with crops

[asomoza]

oh sorry, I thought it was an option in the args so I merged it already, thinks its safer to open a new PR if you want to make changes to it, I also noticed afterwards that the title in the README is different.

[elismasilva]

oh sorry, I thought it was an option in the args so I merged it already, thinks its safer to open a new PR if you want to make changes to it, I also noticed afterwards that the title in the README is different.

on README we have three Stable Diffusion Mixture Tiling Pipeline SD 1.5 and Stable Diffusion Mixture Tiling Pipeline SD 1.5 Canvas, and now i added Stable Diffusion Mixture Tiling Pipeline SDXL.