Fails merging with SDXL models

[JujoHotaru]

いつもお世話になっております。
最新ビルドのSuperMergerをForgeで運用中ですが、最新のリビジョンで、SDXLモデルのマージができなくなっているようです。（SD1モデルでは動作しました）
マージ時に以下のエラーが出ております。

Traceback (most recent call last):
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\queueing.py", line 536, in process_events
    response = await route_utils.call_process_api(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\route_utils.py", line 285, in call_process_api
    output = await app.get_blocks().process_api(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\blocks.py", line 1923, in process_api
    result = await self.call_function(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\blocks.py", line 1508, in call_function
    prediction = await anyio.to_thread.run_sync(  # type: ignore
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\utils.py", line 818, in wrapper
    response = f(*args, **kwargs)
  File "C:\stable-diffusion-webui-forge\extensions\sd-webui-supermerger\scripts\mergers\mergers.py", line 119, in smergegen
    result,currentmodel,modelid,theta_0,metadata = smerge(
  File "C:\stable-diffusion-webui-forge\extensions\sd-webui-supermerger\scripts\mergers\mergers.py", line 452, in smerge
    weight_index = BLOCKIDXLL.index(blocks26)
ValueError: 'Not Merge' is not in list

なお、ForgeではなくA1111上で動作させてみても同様でした。
ご確認いただければ幸いです。

[hako-mikan]

ありがとうございます。修正しました。

[JujoHotaru]

@hako-mikan

ご対応ありがとうございました。
早速アップデートしてマージを試してみたところ、残念ながらまだ一部ケースでマージが成功しておりません。
「Weight sum」と「Triple sum」を使用していたのですが、この両者で発生するエラーが異なるようでした。
「オプション」は「use cuda」と「safetensors」のみオンにして、「Merge!」ボタンを押しています。
動作環境はForgeの最新リビジョンになります。

• Weight sumの場合：「VAEを焼き込む」を使っていると以下のエラーになります。VAE無しなら成功しました。

Loading weights [LoRABreed_v20] from file
Loading weights [VioletBreed_v10] from file
Stage 1/2: 100%|##################################################################| 2515/2515 [00:07<00:00, 336.84it/s]
Stage 2/2: 100%|###############################################################################| 12/12 [00:00<?, ?it/s]
Baking in VAE from C:\stable-diffusion-webui-forge\models\VAE\xlVAEC_f2.safetensors
Traceback (most recent call last):
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\queueing.py", line 536, in process_events
    response = await route_utils.call_process_api(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\route_utils.py", line 285, in call_process_api
    output = await app.get_blocks().process_api(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\blocks.py", line 1923, in process_api
    result = await self.call_function(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\blocks.py", line 1508, in call_function
    prediction = await anyio.to_thread.run_sync(  # type: ignore
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\utils.py", line 818, in wrapper
    response = f(*args, **kwargs)
  File "C:\stable-diffusion-webui-forge\extensions\sd-webui-supermerger\scripts\mergers\mergers.py", line 119, in smergegen
    result,currentmodel,modelid,theta_0,metadata = smerge(
  File "C:\stable-diffusion-webui-forge\extensions\sd-webui-supermerger\scripts\mergers\mergers.py", line 611, in smerge
    for key in vae_dict.keys():
AttributeError: 'NoneType' object has no attribute 'keys'

• Triple sumの場合：VAEの焼き込み有無に関係なく以下のエラーになるようです。「use cuda」をオフにしても変わりませんでした。

Loading weights [LoRABreed_v20] from file
Loading weights [VioletBreed_v10] from file
Loading weights [LoRABreed_Petite_v20] from file
Stage 1/2:   0%|                                                                              | 0/2515 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\queueing.py", line 536, in process_events
    response = await route_utils.call_process_api(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\route_utils.py", line 285, in call_process_api
    output = await app.get_blocks().process_api(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\blocks.py", line 1923, in process_api
    result = await self.call_function(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\blocks.py", line 1508, in call_function
    prediction = await anyio.to_thread.run_sync(  # type: ignore
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\gradio\utils.py", line 818, in wrapper
    response = f(*args, **kwargs)
  File "C:\stable-diffusion-webui-forge\extensions\sd-webui-supermerger\scripts\mergers\mergers.py", line 119, in smergegen
    result,currentmodel,modelid,theta_0,metadata = smerge(
  File "C:\stable-diffusion-webui-forge\extensions\sd-webui-supermerger\scripts\mergers\mergers.py", line 482, in smerge
    theta_0_a =lerp(theta_0_a.to(torch.float32),lerp(theta_1[key].to(torch.float32),theta_2[key].to(torch.float32),current_beta/(current_alpha + current_beta)),current_alpha + current_beta).to(theta_0_a.dtype)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

引き続きご確認いただければ幸いです。

[JujoHotaru]

もう1件問題がありましたので記載いたします。

上記で、VAEを焼き込まないWeight sumは成功したと記述しましたが、それによって生成されたXLモデル（「save model」をチェックしてファイルに保存）を生成に用いると、WebUI側でエラーが発生します。
どうも正常動作しないモデルになってしまうようです。
以下Forge側のログです。

Loading Model: {'checkpoint_info': {'filename': 'C:\\stable-diffusion-webui-forge\\models\\Stable-diffusion\\xl_merge_test.safetensors', 'hash': 'f3591a7a'}, 'additional_modules': ['C:\\stable-diffusion-webui-forge\\models\\VAE\\xlVAEC_f2.safetensors'], 'unet_storage_dtype': None}
StateDict Keys: {'unet': 1680, 'vae': 248, 'text_encoder': 0, 'text_encoder_2': 0, 'ignore': 0}
Traceback (most recent call last):
  File "C:\stable-diffusion-webui-forge\modules_forge\main_thread.py", line 30, in work
    self.result = self.func(*self.args, **self.kwargs)
  File "C:\stable-diffusion-webui-forge\modules\txt2img.py", line 123, in txt2img_function
    processed = processing.process_images(p)
  File "C:\stable-diffusion-webui-forge\modules\processing.py", line 841, in process_images
    manage_model_and_prompt_cache(p)
  File "C:\stable-diffusion-webui-forge\modules\processing.py", line 800, in manage_model_and_prompt_cache
    p.sd_model, just_reloaded = forge_model_reload()
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "C:\stable-diffusion-webui-forge\modules\sd_models.py", line 504, in forge_model_reload
    sd_model = forge_loader(state_dict, additional_state_dicts=additional_state_dicts)
  File "C:\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "C:\stable-diffusion-webui-forge\backend\loader.py", line 283, in forge_loader
    component = load_huggingface_component(estimated_config, component_name, lib_name, cls_name, local_path, component_sd)
  File "C:\stable-diffusion-webui-forge\backend\loader.py", line 59, in load_huggingface_component
    assert isinstance(state_dict, dict) and len(state_dict) > 16, 'You do not have CLIP state dict!'
AssertionError: You do not have CLIP state dict!
You do not have CLIP state dict!

[JujoHotaru]

上記ですが、モデルをファイルに保存しないで「Merge&Gen」をすると、SuperMerger上では正常に画像が生成されました。

[kwikmn]

I have a similar issue.

SuperMerger gives me error messages whenever I try to merge SDXL LoRAs, but it works just fine for SD1.5 LoRAs. (I'm not merging the loras into a checkpoint, I am trying to merge two or three LoRAs).

Here is the error:
merging R:\Forge\webui\models\Lora\SDXL_Pony\sdxl_People\SDXL_UHNX_woman.safetensors: [0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8]
dim: [32], alpha: [16.0]
merging...
Traceback (most recent call last):
File "R:\Forge\webui\extensions\sd-webui-supermerger\scripts\mergers\pluslora.py", line 426, in lmerge
sd = merge_lora_models(ln, lr, settings, False, calc_precision)
File "R:\Forge\webui\extensions\sd-webui-supermerger\scripts\mergers\pluslora.py", line 483, in merge_lora_models
ratio = ratios[blockfromkey(key, keylist, isv2)]
IndexError: list index out of range

[duskfallcrew]

Weird #415 - Similar different issue but the most recent update seems to bork the pluslora part - I've fixed most of it temporary by using the v20 tree from 3+ weeks ago - something in the recent branch feels like OpenAI/ChatGPT (which is valid, that is a very great tool) but someone forgot to clean up the squabble logic of "random python modules that don't exist" - I'm no coder, so i could be wrong on that one, but at least the patch i put together by just using an older version of pluslora worked

[duskfallcrew]

Ok funny note, installing V20 branch causes the issue in this one, but taking the pluslora.py last time out of v20 but keeping the main seemed to not cause too many issues ;_; immadie now.