Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cont'd] Add the SDE variant of DPM-Solver and DPM-Solver++ to DPM Single Step #8269

Merged
merged 27 commits into from
Jul 17, 2024

Conversation

tolgacangoz
Copy link
Contributor

@tolgacangoz tolgacangoz commented May 24, 2024

Thanks for the opportunity to continue #4251 @yiyixuxu.

There was uncertainty about whether sigma_s0 or sigma_s1 should be used in the second-order update in sde-dpmsolver++ when @cmdr2 was implementing. I tried both and chose sigma_s1:

sigma_s0 sigma_s1
sigma_s0 sigma_s1

Reproduction notebook.

@LuChengTHU I want to congratulate you on this invention and thank you for your previous DPM-Solver contributions in the repo. It would be very nice to review this version. This PR is the exact copy of your "Add the SDE variant of DPM-Solver and DPM-Solver++ [for multi-step]" #3344 PR except sigma_s1 is used in the second-order update rather than sigma_s0.

@tolgacangoz tolgacangoz changed the title [Cont] Add the SDE variant of DPM-Solver and DPM-Solver++ to DPM Single Step Add the SDE variant of DPM-Solver and DPM-Solver++ to DPM Single Step May 24, 2024
@tolgacangoz tolgacangoz changed the title Add the SDE variant of DPM-Solver and DPM-Solver++ to DPM Single Step [Cont'd] Add the SDE variant of DPM-Solver and DPM-Solver++ to DPM Single Step May 24, 2024
@sayakpaul sayakpaul requested a review from yiyixuxu June 1, 2024 07:12
@LuChengTHU
Copy link
Contributor

Thanks for this nice PR! For single-step method we indeed should use sigma_s1 because we always take a "large step" for single-step method and re-use some intermediate variables (i.e., at time s0). So I believe this PR is correct!

@tolgacangoz
Copy link
Contributor Author

Thank you very much for your review!

@tolgacangoz
Copy link
Contributor Author

tolgacangoz commented Jun 2, 2024

sde-dpmsolver++ seems reasonable; but sde-dpmsolver creates almost complete noise. I am into this.

!git clone -b sde-dpmsolver-single-step https://github.com/tolgacangoz/diffusers.git
%cd diffusers
!pip install -e . accelerate -q

from diffusers import AutoPipelineForText2Image, DPMSolverSinglestepScheduler
import torch

pipeline_text2image = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16",
    use_safetensors=True).to("cuda")
prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
generator = torch.Generator('cpu').manual_seed(0)
pipeline_text2image.scheduler = DPMSolverSinglestepScheduler.from_config(
    pipeline_text2image.scheduler.config, algorithm_type='sde-dpmsolver',
    final_sigmas_type='sigma_min')
image = pipeline_text2image(prompt=prompt, generator=generator).images[0]
image

image

sde-dpmsolver for DPMSolverMultistepScheduler also creates considerably noisy outputs; but not like complete noise.

@tolgacangoz
Copy link
Contributor Author

tolgacangoz commented Jun 18, 2024

For sde-dpmsolver:

Multi-step num_inference_steps=50 Single-step num_inference_steps=50
m_50 s_50
Multi-step num_inference_steps=100 Single-step num_inference_steps=100
m_100 s_100

Even multi-step with num_inference_steps=100 seems noisy 🤔. Tbh, my math understanding level is insufficient to understand this study well.
@LuChengTHU How should we conclude about sde-dpmsolver implementation?

@LuChengTHU
Copy link
Contributor

Hi @tolgacangoz , I would suggest only use sde-dpmsolver++ because this formulation is more suitable for SDEs and can include stochastic DDIM / DDPM as the first order case. For sde-dpmsolver it may even need 1000 steps to converge so it is hard to use in practice.

@tolgacangoz
Copy link
Contributor Author

Hi @tolgacangoz , I would suggest only use sde-dpmsolver++ because this formulation is more suitable for SDEs and can include stochastic DDIM / DDPM as the first order case. For sde-dpmsolver it may even need 1000 steps to converge so it is hard to use in practice.

OK, thank you for your feedback!

@tolgacangoz
Copy link
Contributor Author

@yiyixuxu, this is ready for your review now.
Will one continue with this?

@yiyixuxu
Copy link
Collaborator

yiyixuxu commented Jul 9, 2024

@LuChengTHU is this PR good to merge now?

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@tolgacangoz tolgacangoz changed the title [Cont'd] Add the SDE variant of DPM-Solver and DPM-Solver++ to DPM Single Step [Cont'd] Add the SDE variant of ~~DPM-Solver~~ and DPM-Solver++ to DPM Single Step Jul 11, 2024
@tolgacangoz tolgacangoz requested a review from yiyixuxu July 13, 2024 07:50
@yiyixuxu yiyixuxu merged commit e87bf62 into huggingface:main Jul 17, 2024
13 of 15 checks passed
@yiyixuxu
Copy link
Collaborator

@tolgacangoz
thank you very much! this is a very nice PR :)

@tolgacangoz
Copy link
Contributor Author

Thanks for merging and labeling as a good PR!

@tolgacangoz tolgacangoz deleted the sde-dpmsolver-single-step branch July 17, 2024 10:29
Disty0 pushed a commit to Disty0/diffusers that referenced this pull request Jul 18, 2024
…DPM Single Step (huggingface#8269)

* Add the SDE variant of DPM-Solver and DPM-Solver++ to DPM Single Step


---------

Co-authored-by: cmdr2 <[email protected]>
@tolgacangoz tolgacangoz changed the title [Cont'd] Add the SDE variant of ~~DPM-Solver~~ and DPM-Solver++ to DPM Single Step [Cont'd] Add the SDE variant of DPM-Solver and DPM-Solver++ to DPM Single Step Oct 12, 2024
sayakpaul pushed a commit that referenced this pull request Dec 23, 2024
…DPM Single Step (#8269)

* Add the SDE variant of DPM-Solver and DPM-Solver++ to DPM Single Step


---------

Co-authored-by: cmdr2 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants