[fix] FreeInit step index out of bounds

[a-r-r-o-w]

What does this PR do?

When using schedulers that maintain an internal _step_index, FreeInit with fast mode disabled fails with an out of bounds error. Discovered here

Code to reproduce

import cProfile

import torch
from diffusers import MotionAdapter, AnimateDiffPipeline, DDIMScheduler, DPMSolverMultistepScheduler
from diffusers.utils import export_to_gif


def main():
    adapter = MotionAdapter.from_pretrained("guoyww/animatediff-motion-adapter-v1-5-2")
    model_id = "SG161222/Realistic_Vision_V5.1_noVAE"
    pipe = AnimateDiffPipeline.from_pretrained(model_id, motion_adapter=adapter, torch_dtype=torch.float16).to("cuda")
    # pipe.scheduler = DDIMScheduler.from_pretrained(
    pipe.scheduler = DPMSolverMultistepScheduler.from_pretrained(
        model_id,
        subfolder="scheduler",
        algorithm_type="dpmsolver++",
        beta_schedule="linear",
        clip_sample=False,
        timestep_spacing="linspace",
        steps_offset=1
    )

    pipe.enable_free_init(use_fast_sampling=False)

    output = pipe(
        prompt="a panda playing a guitar, on a boat, in the ocean, high quality",
        negative_prompt="bad quality, worse quality",
        num_frames=16,
        guidance_scale=7.5,
        num_inference_steps=20,
        generator=torch.Generator("cpu").manual_seed(42),
    )

    frames = output.frames[0]
    export_to_gif(frames, "animation.gif")

# main()
cProfile.run("main()", filename="log.txt")

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@DN6

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

nice catch!
should we add a test for when fast mode is disabled?

[a-r-r-o-w]

nice catch! should we add a test for when fast mode is disabled?

thanks. yes, i think we should have another tiny freeinit test for atleast one scheduler that manages an internal state _step_index. will add it shortly

[a-r-r-o-w]

@yiyixuxu WDYT? Without the change in free_init_utils.py, the test here fails. I think DPMSolver and DDIM (used in every test except the one just added) cover all internal scheduler implementation cases. Added LCM just for safety since it is widely used with AnimateLCM. If it's alright, I'll add it to the other pipeline tests.

[yiyixuxu]

@a-r-r-o-w
That sounds good to me
I thought the issue was not with specific schedulers, but simply that when you use use_fast_sampling=False (the timesteps are not set in that case), I would imagine making another test_free_init with use_fast_sampling=False is already sufficient; but testing with more schedulers is better if it does not slow down too much :)

[a-r-r-o-w]

You're partially right - when use_fast_sampling is False, the timesteps and _step_index are never reset, but that's not a problem with something like DDIM. For DPMSolver, however, the _step_index continues to increase unless reset, which we do now in the PR.

I don't think it's really necessary to add any more schedulers because I've rarely come across newer workflows with recommended settings being anything but Euler, Euler A, DDIM, DPMSolver variants, LCM and Align Your Steps for AnimateDiff.

i also think FreeInit is one of the lesser used features so we shouldn't need to burden our fast tests with too many tests for all settings of it. It can stabilize videos from models that don't work too well for video, but people wouldn't be using bad models in serious use cases anyway. AnimateLCM atleast pretty much doesn't seem to benefit much from the overhead of extra steps added IMO. We can revisit if it helps in specific workflows in the future :)