Fix for Mask Patch Failure and Quantization Issues in Latest transformers Versions

[Aintor]

In the latest versions of the transformers library (specifically above version 4.34.1), the _make_causal_mask function in the modeling_clip module has been removed. Previously, code that utilized this function looked like this:

causal_attention_mask = _make_causal_mask(input_shape, hidden_states.dtype, device=hidden_states.device)

However, with recent updates, this call has been replaced by:

causal_attention_mask = _create_4d_causal_attention_mask(input_shape, hidden_states.dtype, device=hidden_states.device)

You can see more in this thread huggingface/transformers#28305.

This change disrupts the functionality in python_coreml_stable_diffusion/torch2coreml.py, where the following line:

modeling_clip._make_causal_mask = patched_make_causal_mask

can no longer patch the _make_causal_mask function as expected, resulting in the following error during quantization:

ValueError: Input X contains infinity or a value too large for dtype('float64').

See related issues: #331, #303, #325, #246

This PR addresses the issue by adding a monkey patch to modeling_clip for the _create_4d_causal_attention_mask function, thereby fixing the mask patch failure and restoring compatibility with the --quantize-nbits feature in the latest transformers versions. It also retains the original function override to maintain support for older transformers versions.

[Aintor]

By the way, I have tested this PR in a new conda environment initialized with pip install -e ., and the --quantize-nbits flag works as expected.

[Aintor]

@aseemw I noticed PR #316 mentions transformers version 4.29.2, where it uses the following line:
causal_attention_mask = self._build_causal_attention_mask(bsz, seq_len, hidden_states.dtype, device=hidden_states.device)
Do you think I should add support for versions below 4.29.2 as well?

[Aintor]

@aseemw I noticed PR #316 mentions transformers version 4.29.2, where it uses the following line: causal_attention_mask = self._build_causal_attention_mask(bsz, seq_len, hidden_states.dtype, device=hidden_states.device) Do you think I should add support for versions below 4.29.2 as well?

However, I don’t recommend adding support for versions below 4.30.0, as the mask functionality has changed frequently in the modeling_clip module since its creation and remained unstable until version 4.30.0. I believe it would be best to discontinue support for versions earlier than 4.30.0.