3.2.0 release breaks CLIPModel

[BoPeng]

After upgrading from version 3.1.1 to 3.2, loading clip-ViT-L-14 gives the following error:

>>> SentenceTransformer('clip-ViT-L-14')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 306, in __init__
    modules, self.module_kwargs = self._load_sbert_model(
                                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 1722, in _load_sbert_model
    module = module_class(model_name_or_path, cache_dir=cache_folder, backend=self.backend, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: CLIPModel.__init__() got an unexpected keyword argument 'cache_dir'

This should be caused by the recent removal of the try/except block around the model loading code:

sentence-transformers/sentence_transformers/SentenceTransformer.py

Lines 1721 to 1724 in 29535eb

	# try:
	module = module_class(model_name_or_path, cache_dir=cache_folder, backend=self.backend, **kwargs)
	# except TypeError:
	# module = module_class.load(model_name_or_path)

[PraNavKumAr01]

Same issue, did you find a fix?

[BoPeng]

Downgrading to 3.1.1 fixed the issue temporarily.

[PraNavKumAr01]

Got it, but im trying to quantize the clip model to qint8 onnx, but for that the functions are in 3.2 only

[BoPeng]

I just created a PR (#3007). Before this PR is merged, you can use my fork (e.g. use git+https://github.com/BoPeng/sentence-transformers.git@issue3005#egg=sentence-transformers instead of sentence_transformers==3.2.0 in requirements.txt).

[ir2718]

Hi @BoPeng,

I tried installing from the source and it works as expected, so this might already be fixed elsewhere.

>>> import sentence_transformers as stf
>>> from sentence_transformers import SentenceTransformer
>>> st = SentenceTransformer('clip-ViT-L-14')
>>> stf.__version__
'3.3.0.dev0'

[BoPeng]

You are right. I am closing the PR and will use the master branch for now.

[PraNavKumAr01]

from sentence_transformers import SentenceTransformer, export_dynamic_quantized_onnx_model

st = SentenceTransformer('clip-ViT-B-16')
st.save("clip_vit_b_16_main", "onnx")

model = SentenceTransformer("/content/clip_vit_b_16_main", backend="onnx")
export_dynamic_quantized_onnx_model(model, "avx512_vnni", "clip-ViT-B-32", push_to_hub=False, create_pr=False)

Doing this still gives me the same cache_dir error, even after building from source

[ir2718]

@PraNavKumAr01

Yeah, it seems that ONNX is not supported currently for the CLIP model. In the meantime, if you are okay with using other libraries, check out this issue mlfoundations/open_clip#186, and this repo https://github.com/jina-ai/clip-as-service.

[tomaarsen]

Thanks for reporting this! I can reproduce it now, but only if the CLIP model is saved in the root of the model repository/directory. Some more details in #3007 (comment).

This PR should fix this issue, and I'll include it in a patch release very soon.

I do want to mention that ONNX exporting indeed isn't supported for CLIP models I'm afraid, as it's only supported in the Transformer module, not in the CLIPModel module.

• Tom Aarsen