Tencent Hunyuan Team: add HunyuanDiT related updates #8240

gnobitab · 2024-05-23T09:03:20Z

This PR did the following things:

Created HunyuanDiTPipeline in src/diffusers/pipelines/hunyuandit/ and HunyuanDiT2DModel in ./src/diffusers/models/transformers/.
To support HunyuanDiT2DModel, added HunyuanDiTBlock and helper functions in src/diffusers/models/attention.py .
Uploaded the safetensors model to my huggingface: XCLiu/HunyuanDiT-0523
Tested the output of the migrated model+code is the same as our repo (https://github.com/Tencent/HunyuanDiT). Have tested different resolutions and batch sizes > 1 and made sure they work correctly.

In this branch, you can run HunyuanDiT in FP32 with:

python3 test_hunyuan_dit.py

which includes the following codes:

import torch
from diffusers import HunyuanDiTPipeline

pipe = HunyuanDiTPipeline.from_pretrained("XCLiu/HunyuanDiT-0523", torch_dtype=torch.float32)
pipe.to('cuda')

### NOTE: HunyuanDiT supports both Chinese and English inputs
prompt = "一个宇航员在骑马"
#prompt = "An astronaut riding a horse"
image = pipe(height=1024, width=1024, prompt=prompt).images[0]

image.save("./img.png")

Dependency:
maybe the timm package

TODO lists:

FP16 support: I didn't change the parameter use_fp16 in HunyuanDiTPipeline.__call__(). The reason is BertModel does not support FP16 quantization. In our repo we only quantize the diffusion transformer to FP16. I guess there must be some smart way to support FP16.
Simplify and refactor the HunyuanDiTBlock related codes in src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py.
Refactor the pipeline and HunyuanDiT2DModel to diffusers style.
doc

Thank you so much! I'll be there and help with everything.

cc: @sayakpaul @yiyixuxu

sayakpaul

Thanks for working on this.

I left a number of design related comments. I think we need to get them sorted first and then work on other nits. But let's also wait for @yiyixuxu to comment further before making any changes :)

src/diffusers/models/attention.py

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

yiyixuxu

Thanks for the PR!
I think the first a few main tasks are:

rewrite attentions using diffusers (I will help with this)
remove timm dependency (I think you only use it for a simple function, it should be easy to remove)
refactor the pipelines (I agree with @sayakpaul's comments there, we can start with these things he has pointed out)

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

yiyixuxu · 2024-05-23T16:52:55Z

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

+            embedder_t5=embedder_t5,
+        )
+
+        self.text_encoder.pooler.to_empty(device='cpu') ### workaround for the meta device in pooler...


ohh I see https://github.com/huggingface/transformers/blob/965e98dc54c4853b34b3a49810f24856a9fcdba8/src/transformers/models/bert/modeling_bert.py#L976

@sayakpaul can you look into how we can make it work with from_pretrained()?
do we need to make a wrapper of bert model inside diffusers?

yiyixuxu · 2024-05-24T02:21:09Z

Hi:

I removed HunyuanDiTAttention and HunyuanDiTCrossAttention to use our Attention class with a HunyuanAttnProcessor2_0 attention processor instead,

feel free to test out the PR branch and cherry-pick this commit 3f85b1d if results are ok to you

I included a testing script here #8265 (comment)

… norm3; update test file

gnobitab · 2024-05-24T14:37:06Z

Hi I did the following things:

cleaned the pipeline/transformer codes according to Sayak's suggestions.
switched to yiyi's refactored attention.
new test code in test_hunyuan_dit.py. based on yiyi's test code, switched norm2 and norm3, fixed generator to seed 0. the image should be:

For now, I will not change the remote state_dict XCLiu/HunyuanDiT-0523. will update after we finished everything.

Please review and comment, thx!

@sayakpaul @yiyixuxu

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

sayakpaul

Thanks for the changes and for incorporating our feedback so nicely so far! I have a left a couple of comments.

Broadly, I think, for the next phase of the integration, we could focus on:

Solving the BertModel pooler problem as mentioned in the comments.
Remove all the unnecessary blocks related to attention, etc.
Move the embedding related functionalities to embeddings.py.
Try to have Rotary Embeddings as a class.

Would like to also cross-check with @yiyixuxu here at this point if these points make sense.

yiyixuxu

thanks!
looks much better now! I left a few more comments.
remaining to-dos:

can you address the comments on the pipeline?
I will help refactor models
working on tests (cc @sayakpaul here, can you give more guidance)
make the BertModel work with DiffusionPipeline.from_pretrained() cc @sayakpaul and @DN6 here see https://github.com/huggingface/diffusers/pull/8240/files#r1611508812

src/diffusers/models/attention.py

src/diffusers/models/attention_processor.py

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

gnobitab · 2024-05-27T10:01:09Z

I made some improvements according to Sayak and Yiyi's suggestions.

Several additional problems :

Our BertModel cannot work in FP16. So to make our pipeline to FP16, we have to make only the transformer to FP16 instead of just pipe = HunyuanDiTPipeline.from_pretrained("XCLiu/HunyuanDiT-0523", transformer=model, torch_dtype=torch.float16)
I left some comments in the above conversations regarding the magic numbers. Please help me on the deisgn choice.

sayakpaul · 2024-05-27T12:20:19Z

Our BertModel cannot work in FP16. So to make our pipeline to FP16, we have to make only the transformer to FP16 instead of just pipe = HunyuanDiTPipeline.from_pretrained("XCLiu/HunyuanDiT-0523", transformer=model, torch_dtype=torch.float16)

Why wouldn't it work, though? Could you provide more details here? In order for the pipeline to operate in torch.float16 all the components need to be in torch.float16, or we have deal with it differently like so:

diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py

Line 1264 in b3d10d6

    
           needs_upcasting = self.vae.dtype == torch.float16 and self.vae.config.force_upcast

Additionally, could you see if this worked so that we can avoid self.text_encoder.pooler.to_empty(device='cpu')?

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

src/diffusers/models/attention_processor.py

yiyixuxu · 2024-05-29T07:05:49Z

I refactored the model here #8310
you can see the changed I made in this commit b0e0da2

I did below things:

refactor HunyuanDiT2DModel: removed HunyuanDiTPatchEmbed, HunyuanDiTTimestepEmbedder
refactor apply_rotary_emb: moved all the get_2d_rotary_pos_embed_* functions and apply_rotary_emb to embeddings.py
moved HunyuanDiTBlock and other Hunyuan-specific block to the same file as HunyuanDiT2DModel

I changed arg names to be more aligned with our transformer models and blocks also removed bunch of functionalities that are not used in this implementation - let me know if I did anything wrong or any of the changes does not make sense!

feel free to just pick the commit and make any modifications on your PR.

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

gnobitab · 2024-05-30T06:16:24Z

Our BertModel cannot work in FP16. So to make our pipeline to FP16, we have to make only the transformer to FP16 instead of just pipe = HunyuanDiTPipeline.from_pretrained("XCLiu/HunyuanDiT-0523", transformer=model, torch_dtype=torch.float16)

Why wouldn't it work, though? Could you provide more details here? In order for the pipeline to operate in torch.float16 all the components need to be in torch.float16, or we have deal with it differently like so:

diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py

Line 1264 in b3d10d6

needs_upcasting = self.vae.dtype == torch.float16 and self.vae.config.force_upcast

Additionally, could you see if this worked so that we can avoid self.text_encoder.pooler.to_empty(device='cpu')?

Hi Sayak,
my server was down. I restarted the sever, reinstalled the environment, and the model works with FP16 now... I have no idea why, but the current environment is,

torch==2.0.1
transformers==4.41.1

I guess the reason is I'm using the latest transformers library now. @sayakpaul

sayakpaul · 2024-05-30T06:19:26Z

Okay great. It seems like the FP16 problem and also the to_empty() problem (solution here) are solved now?

…omments

gnobitab · 2024-05-30T13:55:02Z

I pushed a new version. In this version:

Checked and Merged Yiyi's refactor in b0e0da2 (PR HunyuanDIT model refactor #8310)
Fixed FP16 and to_empty problem.
polished the whole codebase following the new comments

The new test file is updated in test_hunyuan_dit.py

Thank you for the help! Please review the new version @yiyixuxu @sayakpaul

gnobitab · 2024-05-30T14:04:30Z

Note:
(1) Fix FP16: update the transformers to 4.41.1 or later.
(2) to_empty: https://github.com/huggingface/diffusers/pull/8240/files#r1617108976

from transformers import BertModel
bert_model = BertModel.from_pretrained("XCLiu/HunyuanDiT-0523", add_pooling_layer=True, subfolder="text_encoder")

pipe = HunyuanDiTPipeline.from_pretrained("XCLiu/HunyuanDiT-0523", text_encoder=bert_model, transformer=model, torch_dtype=torch.float32)
pipe.to('cuda')
pipe.save_pretrained("HunyuanDiT-ckpt")

del pipe

pipe = HunyuanDiTPipeline.from_pretrained("HunyuanDiT-ckpt")
pipe.cuda()

yiyixuxu

thanks!!
I think we can merge this soon

We have tests and doc left - doc can be added in a separate PR if you need more time, but let's quickly add a test, you can use pixart_sigma test as reference https://github.com/huggingface/diffusers/tree/main/tests/pipelines/pixart_sigma

yiyixuxu · 2024-05-30T20:22:43Z

src/diffusers/models/embeddings.py

@@ -806,6 +920,27 @@ def forward(self, caption):
        return hidden_states


+# YiYi notes: combine PixArtAlphaTextProjection and HunYuanTextProjection?


cc @DN6 here should we use PixArtAlphaTextProjectioninstead? it seems like this projection layer is going to be used a lot

Yeah I think we can make it a generic TextProjection class and reuse it.

yiyixuxu · 2024-05-30T20:25:30Z

src/diffusers/models/transformers/hunyuan_transformer_2d.py

+        ).to(origin_dtype)
+
+
+class AdaLayerNormShift(nn.Module):


@gnobitab, was this a method your team came up with? Is there any reason you chose this over the regular Ada layer norm with both scale and shift?
I want to understand this because we will need to decide to keep this module here or put it in the embeddings.py so it is easier for other researchers to use it

Yeah I think AdaLayerNorm can be reused here as well

diffusers/src/diffusers/models/normalization.py

Line 28 in f9fa8a8

class AdaLayerNorm(nn.Module):

I'll go double check with the team and get back to you later

to-do [2]: refactor?

src/diffusers/models/transformers/hunyuan_transformer_2d.py

src/diffusers/pipelines/pipeline_utils.py

src/diffusers/models/transformers/hunyuan_transformer_2d.py

…to HEAD

src/diffusers/models/normalization.py

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

src/diffusers/models/transformers/hunyuan_transformer_2d.py

yiyixuxu · 2024-06-01T22:18:34Z

src/diffusers/models/embeddings.py

@@ -507,6 +626,88 @@ def forward(self, timestep, class_labels, hidden_dtype=None):
        return conditioning


+class HunyuanDiTAttentionPool(nn.Module):


to-do[1] (low prior): see if we can consolidate this class with

diffusers/src/diffusers/models/embeddings.py

Line 588 in bc108e1

class AttentionPooling(nn.Module):

and

diffusers/src/diffusers/models/unets/unet_kandinsky3.py

Line 483 in bc108e1

class Kandinsky3AttentionPooling(nn.Module):

can ask community for help

yiyixuxu · 2024-06-01T22:19:24Z

src/diffusers/models/transformers/hunyuan_transformer_2d.py

+        ).to(origin_dtype)
+
+
+class AdaLayerNormShift(nn.Module):


to-do [2]: refactor?

yiyixuxu · 2024-06-01T22:20:51Z

src/diffusers/models/transformers/hunyuan_transformer_2d.py

+        return hidden_states
+
+
+class HunyuanDiT2DModel(ModelMixin, ConfigMixin):


to-do[3] optimization! cc @sayakpaul

yiyixuxu · 2024-06-01T22:23:24Z

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

+        text_encoder: BertModel,
+        tokenizer: BertTokenizer,
+        transformer: HunyuanDiT2DModel,
+        scheduler: DDPMScheduler,


[to-do 3] Does hunyuan DiT work with other schedulers? @gnobitab
if so, we need to change the type here, and also let's update the doc example with a more efficient scheduler

Suggested change

scheduler: DDPMScheduler,

scheduler: KarrasDiffusionSchedulers,

yiyixuxu · 2024-06-01T22:27:34Z

tests/pipelines/hunyuan_dit/test_hunyuan_dit.py

+enable_full_determinism()
+
+
+class HunyuanDiTPipelineFastTests(PipelineTesterMixin, unittest.TestCase):


to-do[4]

I added s simple fast test and slow test. @gnobitab can we add more tests? see examples https://github.com/huggingface/diffusers/blob/bc108e15333cb0e8a092647320cbb4d70d6d0f03/tests/pipelines/pixart_sigma/test_pixart.py

we help with this too but we prefer authors to add tests and make sure the current implementation is correct!

yiyixuxu · 2024-06-01T22:28:30Z

tests/pipelines/hunyuan_dit/test_hunyuan_dit.py

+        max_diff = np.abs(image_slice.flatten() - expected_slice).max()
+        self.assertLessEqual(max_diff, 1e-3)
+
+    def test_sequential_cpu_offload_forward_pass(self):


to-do[5] (slow prior) the sequential cpu offload does not work, can look into this. but not a priority since the other offloading method works and is way more popular cc @sayakpaul

yiyixuxu · 2024-06-01T22:40:56Z

merging now!

I left a bunch to-dos in the comments too

for @sayakpaul
can you help with optimization? make sure torch.compile works and all the other meaningful optimization method works here

for @gnobitab

add doc
add more tests if possible
I left a few questions for you about the ada layer norm and the scheduler
move the checkpoint to the official account.
do we need a conversion script? cc @sayakpaul too. is the diffuser checkpoint the main checkpoint that is going to be used by the community?

here is the script that works now for the checkpoint in my PR to your repo https://huggingface.co/XCLiu/HunyuanDiT-0523/discussions/2

# integration test (hunyuan dit)
import torch

from diffusers import HunyuanDiTPipeline


device = "cuda"
dtype = torch.float16

repo = "XCLiu/HunyuanDiT-0523"

pipe = HunyuanDiTPipeline.from_pretrained(repo, revision="refs/pr/2", torch_dtype=dtype)
pipe.enable_model_cpu_offload()

### NOTE: HunyuanDiT supports both Chinese and English inputs
prompt = "一个宇航员在骑马"
#prompt = "An astronaut riding a horse"
generator=torch.Generator(device="cuda").manual_seed(0)
image = pipe(height=1024, width=1024, prompt=prompt, generator=generator).images[0]

image.save("yiyi_test_out.png")

and this is the script I used to convert the current checkpoint in "XCLiu/HunyuanDiT-0523"
basically I did these two changes:

changed the module/folder name for t5 text_encoder and tokenizer to text_encoder_2 and tokenizer_2
updated the state dict for transformers because I refactored the embedding layers for the extra conditions

import torch
from huggingface_hub import hf_hub_download

from diffusers import HunyuanDiTPipeline, HunyuanDiT2DModel
from transformers import T5EncoderModel, T5Tokenizer

import safetensors.torch

device = "cuda"
dtype = torch.float32

repo = "XCLiu/HunyuanDiT-0523"

tokenizer_2 = T5Tokenizer.from_pretrained(repo, subfolder = "tokenizer_t5")
text_encoder_2 = T5EncoderModel.from_pretrained(repo, subfolder = "embedder_t5", torch_dtype=dtype)

model_config = HunyuanDiT2DModel.load_config("XCLiu/HunyuanDiT-0523", subfolder="transformer")
model = HunyuanDiT2DModel.from_config(model_config).to(device)

ckpt_path = hf_hub_download(
    "XCLiu/HunyuanDiT-0523",
    filename ="diffusion_pytorch_model.safetensors",
    subfolder="transformer",
)

state_dict = safetensors.torch.load_file(ckpt_path)

prefix = "time_extra_emb."

# time_embedding.linear_1 -> timestep_embedder.linear_1 
state_dict[f"{prefix}timestep_embedder.linear_1.weight"] = state_dict["time_embedding.linear_1.weight"]
state_dict[f"{prefix}timestep_embedder.linear_1.bias"] = state_dict["time_embedding.linear_1.bias"]
state_dict.pop("time_embedding.linear_1.weight")
state_dict.pop("time_embedding.linear_1.bias")

# time_embedding.linear_2 -> timestep_embedder.linear_2
state_dict[f"{prefix}timestep_embedder.linear_2.weight"] = state_dict["time_embedding.linear_2.weight"]
state_dict[f"{prefix}timestep_embedder.linear_2.bias"] = state_dict["time_embedding.linear_2.bias"]
state_dict.pop("time_embedding.linear_2.weight")
state_dict.pop("time_embedding.linear_2.bias")

# pooler.positional_embedding
state_dict[f"{prefix}pooler.positional_embedding"] = state_dict["pooler.positional_embedding"]
state_dict.pop("pooler.positional_embedding")

# pooler.k_proj
state_dict[f"{prefix}pooler.k_proj.weight"] = state_dict["pooler.k_proj.weight"]
state_dict[f"{prefix}pooler.k_proj.bias"] = state_dict["pooler.k_proj.bias"]
state_dict.pop("pooler.k_proj.weight")
state_dict.pop("pooler.k_proj.bias")

#pooler.q_proj
state_dict[f"{prefix}pooler.q_proj.weight"] = state_dict["pooler.q_proj.weight"]
state_dict[f"{prefix}pooler.q_proj.bias"] = state_dict["pooler.q_proj.bias"]
state_dict.pop("pooler.q_proj.weight")
state_dict.pop("pooler.q_proj.bias")

#  pooler.v_proj
state_dict[f"{prefix}pooler.v_proj.weight"] = state_dict["pooler.v_proj.weight"]
state_dict[f"{prefix}pooler.v_proj.bias"] = state_dict["pooler.v_proj.bias"]
state_dict.pop("pooler.v_proj.weight")
state_dict.pop("pooler.v_proj.bias")

# pooler.c_proj
state_dict[f"{prefix}pooler.c_proj.weight"] = state_dict["pooler.c_proj.weight"]
state_dict[f"{prefix}pooler.c_proj.bias"] = state_dict["pooler.c_proj.bias"]
state_dict.pop("pooler.c_proj.weight")
state_dict.pop("pooler.c_proj.bias")

# style_embedder.weight
state_dict[f"{prefix}style_embedder.weight"] = state_dict["style_embedder.weight"]
state_dict.pop("style_embedder.weight")

# extra_embedder.linear_1
state_dict[f"{prefix}extra_embedder.linear_1.weight"] = state_dict["extra_embedder.linear_1.weight"]
state_dict[f"{prefix}extra_embedder.linear_1.bias"] = state_dict["extra_embedder.linear_1.bias"]
state_dict.pop("extra_embedder.linear_1.weight")
state_dict.pop("extra_embedder.linear_1.bias")

# extra_embedder.linear_2
state_dict[f"{prefix}extra_embedder.linear_2.weight"] = state_dict["extra_embedder.linear_2.weight"]
state_dict[f"{prefix}extra_embedder.linear_2.bias"] = state_dict["extra_embedder.linear_2.bias"]
state_dict.pop("extra_embedder.linear_2.weight")
state_dict.pop("extra_embedder.linear_2.bias")

model.load_state_dict(state_dict)
model.to(dtype)

pipe = HunyuanDiTPipeline.from_pretrained(
    repo, 
    tokenizer_2 = tokenizer_2,
    text_encoder_2 = text_encoder_2,
    transformer = model,
    torch_dtype=dtype)

gnobitab · 2024-06-03T14:38:14Z

@yiyixuxu

Thanks for merging!

Reply to your TODOs:

Doc added here: https://moon-ci-docs.huggingface.co/docs/diffusers/pr_8383/en/api/pipelines/hunyuandit
I manually tested and verified the results. The current version has the same output as the original model. I will try to add more rigorous tests later.
(1) AdaLayerNormShift: I asked the team member and they were following the practice in SDXL: See

diffusers/src/diffusers/models/resnet.py

Line 343 in 4136044

if self.time_emb_proj is not None:

(L343 - L351). I notice the place of self.norm in HunyuanDiT is different from SDXL, but I think it is just a small mistake. As far as I learned, it is not a scientific innovation and I suggest to keep it inside hunyuan_transformer_2d.py

(2) Scheduler: I tested several fast samplers. From my test, I think it is safe to switch from ``DDPMSchedulertoDDIMScheduler`. Other schedulers seem to be fine but I would like to leave that for the community.

4 and 5: I merged your PR in XCLiu/HunyuanDiT-0523 and saved a new checkpoint for the diffusers pipeline. We are moving the new checkpoint to the official account after going through some internal checks. The name will be Tencent-Hunyuan/HunyuanDiT-Diffusers. Let's change the example doc in the pipeline file when it's officially online.

* Hunyuan Team: add HunyuanDiT related updates --------- Co-authored-by: XCLiu <[email protected]> Co-authored-by: yiyixuxu <[email protected]>

diffusers commit 4136044 Tencent Hunyuan Team: add HunyuanDiT related updates huggingface/diffusers#8240

* Hunyuan Team: add HunyuanDiT related updates --------- Co-authored-by: XCLiu <[email protected]> Co-authored-by: yiyixuxu <[email protected]>

Hunyuan Team: add HunyuanDiT related updates

154e150

sayakpaul reviewed May 23, 2024

View reviewed changes

yiyixuxu reviewed May 23, 2024

View reviewed changes

gnobitab force-pushed the hunyuandit branch from 978281d to 154e150 Compare May 24, 2024 14:18

resolved sayak comments; switch to yiyi attention; switched norm2 and…

5915fb4

… norm3; update test file

sayakpaul reviewed May 24, 2024

View reviewed changes

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py Show resolved Hide resolved

sayakpaul reviewed May 24, 2024

View reviewed changes

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py Outdated Show resolved Hide resolved

sayakpaul reviewed May 24, 2024

View reviewed changes

yiyixuxu reviewed May 25, 2024

View reviewed changes

XCLiu added 2 commits May 27, 2024 17:57

modified pipeline with yiyi and sayak comments

eb91b37

modified pipeline with yiyi and sayak comments

12565f3

yiyixuxu mentioned this pull request May 27, 2024

[do not merge] HunyuanDIT refactor #8290

Closed

3 tasks

yiyixuxu reviewed May 28, 2024

View reviewed changes

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py Outdated Show resolved Hide resolved

yiyixuxu reviewed May 28, 2024

View reviewed changes

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py Outdated Show resolved Hide resolved

yiyixuxu reviewed May 28, 2024

View reviewed changes

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py Outdated Show resolved Hide resolved

yiyixuxu reviewed May 28, 2024

View reviewed changes

src/diffusers/models/attention_processor.py Outdated Show resolved Hide resolved

yiyixuxu reviewed May 29, 2024

View reviewed changes

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py Outdated Show resolved Hide resolved

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py Outdated Show resolved Hide resolved

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py Outdated Show resolved Hide resolved

XCLiu added 3 commits May 30, 2024 15:14

modifies from yiyi and sayak new comments

88d8d7e

merged yiyi refactor; fixed to_empty problem; polished according to c…

37c7ab6

…omments

change as yiyi new comments on 0530

b394813

yiyixuxu reviewed May 30, 2024

View reviewed changes

yiyixuxu and others added 9 commits May 31, 2024 18:25

Update src/diffusers/pipelines/pipeline_utils.py

7dba464

style

410b006

refacor embeddings for hunyuan

072514f

Merge branch 'hunyuandit' of github.com:gnobitab/diffusers-hunyuan in…

d4d27d6

…to HEAD

style

0cec08a

update test

f9ce46c

Merge branch 'hunyuandit' of github.com:gnobitab/diffusers-hunyuan in…

c6e7370

…to HEAD

add slow test and clean up doc strings

71e8b10

remove the test file

e829806

yiyixuxu reviewed Jun 1, 2024

View reviewed changes

src/diffusers/models/normalization.py Outdated Show resolved Hide resolved

Update src/diffusers/models/normalization.py

449d134

yiyixuxu reviewed Jun 1, 2024

View reviewed changes

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py Outdated Show resolved Hide resolved

Update src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

fa7cf19

yiyixuxu reviewed Jun 1, 2024

View reviewed changes

src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py Outdated Show resolved Hide resolved

Update src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

fb1f78e

yiyixuxu reviewed Jun 1, 2024

View reviewed changes

src/diffusers/models/transformers/hunyuan_transformer_2d.py Outdated Show resolved Hide resolved

yiyixuxu reviewed Jun 1, 2024

View reviewed changes

src/diffusers/models/transformers/hunyuan_transformer_2d.py Outdated Show resolved Hide resolved

yiyixuxu and others added 3 commits June 1, 2024 11:37

Update src/diffusers/models/transformers/hunyuan_transformer_2d.py

4f8bf55

Update src/diffusers/models/transformers/hunyuan_transformer_2d.py

6321762

fix

042cbe2

yiyixuxu approved these changes Jun 1, 2024

View reviewed changes

yiyixuxu merged commit 4136044 into huggingface:main Jun 1, 2024
15 checks passed

sayakpaul pushed a commit that referenced this pull request Jun 4, 2024

Tencent Hunyuan Team: add HunyuanDiT related updates (#8240)

a83f85d

* Hunyuan Team: add HunyuanDiT related updates --------- Co-authored-by: XCLiu <[email protected]> Co-authored-by: yiyixuxu <[email protected]>

XSE42 added a commit to XSE42/diffusers3d that referenced this pull request Jun 23, 2024

[Sync] diffusers commit 4136044

17c651b

diffusers commit 4136044 Tencent Hunyuan Team: add HunyuanDiT related updates huggingface/diffusers#8240

a-r-r-o-w mentioned this pull request Jul 22, 2024

Adding Differential Diffusion to Kolors, Auraflow, HunyuanDiT #8924

Closed

3 tasks

This was referenced Jul 26, 2024

Adding Differential Diffusion to HunyuanDiT #8992

Closed

Add Differential Diffusion to HunyuanDiT. #9040

Merged

sayakpaul pushed a commit that referenced this pull request Dec 23, 2024

Tencent Hunyuan Team: add HunyuanDiT related updates (#8240)

4351058

* Hunyuan Team: add HunyuanDiT related updates --------- Co-authored-by: XCLiu <[email protected]> Co-authored-by: yiyixuxu <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tencent Hunyuan Team: add HunyuanDiT related updates #8240

Tencent Hunyuan Team: add HunyuanDiT related updates #8240

gnobitab commented May 23, 2024

sayakpaul left a comment

yiyixuxu left a comment

yiyixuxu May 23, 2024

yiyixuxu commented May 24, 2024 •

edited

Loading

gnobitab commented May 24, 2024 •

edited

Loading

sayakpaul left a comment

yiyixuxu left a comment •

edited

Loading

gnobitab commented May 27, 2024

sayakpaul commented May 27, 2024

yiyixuxu commented May 29, 2024 •

edited

Loading

gnobitab commented May 30, 2024

sayakpaul commented May 30, 2024

gnobitab commented May 30, 2024

gnobitab commented May 30, 2024

yiyixuxu left a comment

yiyixuxu May 30, 2024

DN6 May 31, 2024

yiyixuxu May 30, 2024

DN6 May 31, 2024 •

edited

Loading

gnobitab May 31, 2024

yiyixuxu Jun 1, 2024

yiyixuxu Jun 1, 2024

yiyixuxu Jun 1, 2024

yiyixuxu Jun 1, 2024

yiyixuxu Jun 1, 2024

yiyixuxu Jun 1, 2024

yiyixuxu Jun 1, 2024

yiyixuxu commented Jun 1, 2024

gnobitab commented Jun 3, 2024

		@@ -806,6 +920,27 @@ def forward(self, caption):
		return hidden_states


		# YiYi notes: combine PixArtAlphaTextProjection and HunYuanTextProjection?

		@@ -507,6 +626,88 @@ def forward(self, timestep, class_labels, hidden_dtype=None):
		return conditioning


		class HunyuanDiTAttentionPool(nn.Module):

		return hidden_states


		class HunyuanDiT2DModel(ModelMixin, ConfigMixin):

	scheduler: DDPMScheduler,
	scheduler: KarrasDiffusionSchedulers,

		enable_full_determinism()


		class HunyuanDiTPipelineFastTests(PipelineTesterMixin, unittest.TestCase):

Tencent Hunyuan Team: add HunyuanDiT related updates #8240

Tencent Hunyuan Team: add HunyuanDiT related updates #8240

Conversation

gnobitab commented May 23, 2024

sayakpaul left a comment

Choose a reason for hiding this comment

yiyixuxu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yiyixuxu commented May 24, 2024 • edited Loading

gnobitab commented May 24, 2024 • edited Loading

sayakpaul left a comment

Choose a reason for hiding this comment

yiyixuxu left a comment • edited Loading

Choose a reason for hiding this comment

gnobitab commented May 27, 2024

sayakpaul commented May 27, 2024

yiyixuxu commented May 29, 2024 • edited Loading

gnobitab commented May 30, 2024

sayakpaul commented May 30, 2024

gnobitab commented May 30, 2024

gnobitab commented May 30, 2024

yiyixuxu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DN6 May 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yiyixuxu commented Jun 1, 2024

gnobitab commented Jun 3, 2024

yiyixuxu commented May 24, 2024 •

edited

Loading

gnobitab commented May 24, 2024 •

edited

Loading

yiyixuxu left a comment •

edited

Loading

yiyixuxu commented May 29, 2024 •

edited

Loading

DN6 May 31, 2024 •

edited

Loading