remove hook for bnb 4-bit

[SunMarc]

What does this PR do ?

This PR relax the condition that 4-bit bnb models needs to have a hook to work. It is not necessary as we can move the 4-bit model since bnb 0.43.2.

cc @sayakpaul

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[matthewdouglas]

LGTM apart from the quality check. Thanks!

[BenjaminBossan]

Some tests in PEFT are failing since this PR has been merged. Here is a simple reproducer:

import torch
from transformers import AutoModelForSequenceClassification, BitsAndBytesConfig

model_id = "facebook/opt-125m"
model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
)
input = torch.arange(10).view(-1, 1)  # <= on CPU

# RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
model(input)
# _hf_hook no longer exists
assert hasattr(model.score, "_hf_hook")

I think it can be argued that the code was incorrect to begin with but there is the risk that with the next accelerate release, a bunch of similar code out there could start breaking.

To reproduce:

• bad commit: 8ade23cc6aec7c3bd3d80fef6378cafaade75bbe
• good commit: c0552c9012a9bae7f125e1df89cf9ee0b0d250fd

[SunMarc]

I think it can be argued that the code was incorrect to begin with but there is the risk that with the next accelerate release, a bunch of similar code out there could start breaking.

I think we should add a note in the next release that the user should specify which device the bnb model will be loaded on + move the inputs to the right device.

import torch
from transformers import AutoModelForSequenceClassification, BitsAndBytesConfig

model_id = "facebook/opt-125m"
model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
    device_map="cuda"
)
input = torch.arange(10).view(-1, 1)  # <= on CPU
input.to("cuda")

model(input)