1080TI (Family) Issue No Kernel Image is available

[d8ahazard]

This is a centralized place for folks with a 1080TI Family GPU to discuss issues related to that specific card and the above error below.

As this is hardware-specific, I'll try to redirect any related issues here.

My best guess is that this is related to 8bit adam or xformers, as both are using precompiled libraries which probably require specific code for this family of GPU - but that's just a guess.

#52

[ZeroCool22]

I will only say i can use this DB repo's with a 1080 TI without problems:

https://github.com/ShivamShrirao/diffusers/tree/main/examples/dreambooth (WSL + Ubuntu)
https://github.com/smy20011/dreambooth-gui (Docker image ready)

And my config in my Train file looks like this:

export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH
export MODEL_NAME="runwayml/stable-diffusion-v1-5"
export INSTANCE_DIR="training"
export CLASS_DIR="classes"
export OUTPUT_DIR="my_model"

accelerate launch train_dreambooth.py \
  --pretrained_vae_name_or_path="stabilityai/sd-vae-ft-mse" \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --class_data_dir=$CLASS_DIR \
  --output_dir=$OUTPUT_DIR \
  --with_prior_preservation --prior_loss_weight=1.0 \
  --instance_prompt="ddaannii33l" \
  --class_prompt="man" \
  --resolution=512 \
  --train_batch_size=1 \
  --learning_rate=5e-6 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --gradient_accumulation_steps=4 --gradient_checkpointing \
  --use_8bit_adam \
  --max_train_steps=5500 \
  --save_interval=500 \
  --seed=3434554

[d8ahazard]

Also related to:

#47

[hypervoxel]

I am having this same problem. GTX 1080Ti Windows 10

Error no kernel image is available for execution on the device at line 167 in file D:\ai\tool\bitsandbytes\csrc\ops.cu

[Simonvinder]

I have the same issue.

[hypervoxel]

Is there any possibility that might be fixable ? I would love to avoid buying a RTX 3090 / 4090 to run this. Thank you

[Vininski]

I'm in the same boat, depressing how expensive those cards are. Miss the days of top tier cards being under $1K. I'm going to have to try this extension in WSL since I had one of those other repos working before with 8bitadam.

[ZeroCool22]

Is there any possibility that might be fixable ? I would love to avoid buying a RTX 3090 / 4090 to run this. Thank you

https://github.com/smy20011/dreambooth-gui

[sgsdxzy]

I think you can use Linux or WSL to get 8bit AdamW. For Windows, there is just no precompiled dlls.

[RandomChisk]

I'm using Titan Xp, and i have the same issue here

[andykaufseo]

that path D:\ai\tool\bitsandbytes\csrc\ops.cu is hardcoded somewhere, we just need to figure out where (which file) and point it ot the local bitsandbytes

[Kumimono]

I did find a reference to D:\ai\tool\bitsandbytes\csrc\ops.cu in K:\Webui\stable-diffusion-webui\extensions\sd_dreambooth_extension\bitsandbytes_windows\libbitsandbytes_cuda116.dll.

Could even change the drive letter, but, there's no ops.cu located anywhere in the webui installation. Besides, I think it's just a error message, not actually calling that file. But, I know nothing about these python things.

[shairez]

[Kumimono]

Pardon? That's, I think, my only comment the whole of github, and relevant to the issue. How is that spam?

[shairez]

LOL, they removed it already so now it seems like I'm pointing at your comment :)

I'll delete mine now

[Kumimono]

Hah, mods are too quick. :)

[ZeroCool22]

You finally fixed it, 1080 TI working now, how you did it?

[Kumimono]

I'd imagine it was compiled on a different setup, and dependencies and such just, were in the right places? Something like that seemed to be the issue. Not a coder. But can confirm, managed to train with Dreambooth for some time before my 1080ti died. Unrelated causes. 🙂

[yuhaiwan]

solved this by seting use_8bit_adam = 0

[seonnath]

Also worked for me after disabling 8-bit-adam. I was trying to train Lora on the Kohya-ss gui, and for the past 3 days was trying to find some solution and I found it today, finally. Running on 1060 gtx 6gb

[zhouyi311]

I'm using a 1080 graphic card, had the same issue. Made it running just now.

Don't use the "AdamW8bit" optimizer (I randomly picked Lion, might work with other options haven't been tested yet).
don't check "use 8bit adam" under advanced configuration.

At least it's running:
epoch 2/10
steps: 16%|██████████▌ | 296/1800 [08:24<42:44, 1.70s/it, loss=0.103]

[DS-27]

Thanks everyone in this discussion. I have few problems include OP problem. Just in case put here my way to fix it.

"Okay, I solved the problem. Pytorch downloaded by itself after a restart. Then there were problems in the "sample" folder (it's easy to delete). And at the end there was a problem due to the fact that I have a 1080ti.
As I understand it, this problem is on all GPU 1000 series.
The AdamW8bit optimizer doesn't exactly work. Lion works exactly. The rest have not been tested yet.

I'm writing just in case it helps someone."

[CoolDUp]

use bitsandbytes0.38.1 and torch-2.0.1+cu118 then 8-bit AdamW optimizer is work.

bitsandbytes-windows