Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Big refactor] move unets to unets module 🦋 #6630

Merged
merged 18 commits into from
Jan 23, 2024
Merged

[Big refactor] move unets to unets module 🦋 #6630

merged 18 commits into from
Jan 23, 2024

Conversation

sayakpaul
Copy link
Member

@sayakpaul sayakpaul commented Jan 18, 2024 •
edited
Loading

What does this PR do?

As discussed internally, we would like to move the UNet related stuff to their dedicated module unets just like we did with autoencoders.

Note to reviewers: It's a big refactor. So, please take your time in reviewing it. But sooner we're able to merge this, lesser the number of merge conflicts we'll run into.

@sayakpaul sayakpaul changed the title [Refactor] move unets to unets module 🦋 [Big refactor] move unets to unets module 🦋 Jan 18, 2024
@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@sayakpaul
Copy link
Member Author

sayakpaul commented Jan 18, 2024 •
edited
Loading

Out of the two failing tests, the first should pass as soon as https://huggingface.co/hf-internal-testing/tiny-sdxl-custom-all/discussions/2 is merged. - Handled it in a better way.

The second one fails even on main.

patrickvonplaten
patrickvonplaten reviewed Jan 19, 2024
View reviewed changes
src/diffusers/models/unets/unet_2d_blocks.py
@@ -0,0 +1,3591 @@
# Copyright 2023 The HuggingFace Team. All rights reserved.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this file only have 3591 lines but the original deleted unet_2d_blocks.py file has 3635 lines?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the exact same number here: https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/unet_2d_blocks.py

image

Am I missing something?

patrickvonplaten
patrickvonplaten reviewed Jan 19, 2024
View reviewed changes
Copy link
Contributor

@patrickvonplaten patrickvonplaten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok for me! Let's make sure that for import files such as unet_2d_condition.py, unet_2d_blocks.py, we keep the original file and import all classes of the moved files for backwards compatibility

@sayakpaul
Copy link
Member Author

Thanks, @patrickvonplaten. Added a couple classes in models for BC. Would you prefer more?

DN6
DN6 approved these changes Jan 22, 2024
View reviewed changes
Copy link
Collaborator

@DN6 DN6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine for me to merge. Failing pipeline test looks unrelated, think just rerunning the workflow should get it to pass.

@sayakpaul
Copy link
Member Author

I am going to merge this to avoid merge conflicts.

@sayakpaul sayakpaul merged commit 1f0705a into main Jan 23, 2024
16 checks passed
@sayakpaul sayakpaul deleted the unet-module branch January 23, 2024 03:28
a-r-r-o-w added a commit to a-r-r-o-w/diffusers that referenced this pull request Jan 23, 2024
@a-r-r-o-w
fix import error caused due to unet refactor in huggingface#6630
5674a71
DN6 added a commit that referenced this pull request Jan 24, 2024
@a-r-r-o-w @DN6 @patrickvonplaten
AnimateDiff Video to Video (#6328)
a517f66 
* begin animatediff img2video and video2video

* revert animatediff to original implementation

* add img2video as pipeline

* update

* add vid2vid pipeline

* update imports

* update

* remove copied from line for check_inputs

* update

* update examples

* add multi-batch support

* fix __init__.py files

* move img2vid to community

* update community readme and examples

* fix

* make fix-copies

* add vid2vid batch params

* apply suggestions from review

Co-Authored-By: Dhruv Nair <[email protected]>

* add test for animatediff vid2vid

* torch.stack -> torch.cat

Co-Authored-By: Dhruv Nair <[email protected]>

* make style

* docs for vid2vid

* update

* fix prepare_latents

* fix docs

* remove img2vid

* update README to :main

* remove slow test

* refactor pipeline output

* update docs

* update docs

* merge community readme from :main

* final fix i promise

* add support for url in animatediff example

* update example

* update callbacks to latest implementation

* Update src/diffusers/pipelines/animatediff/pipeline_animatediff_video2video.py

Co-authored-by: Patrick von Platen <[email protected]>

* Update src/diffusers/pipelines/animatediff/pipeline_animatediff_video2video.py

Co-authored-by: Patrick von Platen <[email protected]>

* fix merge

* Apply suggestions from code review

* remove callback and callback_steps as suggested in review

* Update tests/pipelines/animatediff/test_animatediff_video2video.py

Co-authored-by: Patrick von Platen <[email protected]>

* fix import error caused due to unet refactor in #6630

* fix numpy import error after tensor2vid refactor in #6626

* make fix-copies

* fix numpy error

* fix progress bar test

---------

Co-authored-by: Dhruv Nair <[email protected]>
Co-authored-by: Patrick von Platen <[email protected]>
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
@sayakpaul
[Big refactor] move unets to unets module 🦋 (huggingface#6630)
e503722 
* move unets to  module 🦋

* parameterize unet-level import.

* fix flax unet2dcondition model import

* models __init__

* mildly depcrecating models.unet_2d_blocks in favor of models.unets.unet_2d_blocks.

* noqa

* correct depcrecation behaviour

* inherit from the actual classes.

* Empty-Commit

* backwards compatibility for unet_2d.py

* backward compatibility for unet_2d_condition

* bc for unet_1d

* bc for unet_1d_blocks
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
@a-r-r-o-w @DN6 @patrickvonplaten
AnimateDiff Video to Video (huggingface#6328)
8ba9540 
* begin animatediff img2video and video2video

* revert animatediff to original implementation

* add img2video as pipeline

* update

* add vid2vid pipeline

* update imports

* update

* remove copied from line for check_inputs

* update

* update examples

* add multi-batch support

* fix __init__.py files

* move img2vid to community

* update community readme and examples

* fix

* make fix-copies

* add vid2vid batch params

* apply suggestions from review

Co-Authored-By: Dhruv Nair <[email protected]>

* add test for animatediff vid2vid

* torch.stack -> torch.cat

Co-Authored-By: Dhruv Nair <[email protected]>

* make style

* docs for vid2vid

* update

* fix prepare_latents

* fix docs

* remove img2vid

* update README to :main

* remove slow test

* refactor pipeline output

* update docs

* update docs

* merge community readme from :main

* final fix i promise

* add support for url in animatediff example

* update example

* update callbacks to latest implementation

* Update src/diffusers/pipelines/animatediff/pipeline_animatediff_video2video.py

Co-authored-by: Patrick von Platen <[email protected]>

* Update src/diffusers/pipelines/animatediff/pipeline_animatediff_video2video.py

Co-authored-by: Patrick von Platen <[email protected]>

* fix merge

* Apply suggestions from code review

* remove callback and callback_steps as suggested in review

* Update tests/pipelines/animatediff/test_animatediff_video2video.py

Co-authored-by: Patrick von Platen <[email protected]>

* fix import error caused due to unet refactor in huggingface#6630

* fix numpy import error after tensor2vid refactor in huggingface#6626

* make fix-copies

* fix numpy error

* fix progress bar test

---------

Co-authored-by: Dhruv Nair <[email protected]>
Co-authored-by: Patrick von Platen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@patrickvonplaten patrickvonplaten patrickvonplaten left review comments

@DN6 DN6 DN6 approved these changes

Assignees
No one assigned
Labels
refactor
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sayakpaul @HuggingFaceDocBuilderDev @DN6 @patrickvonplaten