Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Sentences Transformers and torch optional #250

Closed
szaher opened this issue Jan 10, 2024 · 2 comments
Closed

Make Sentences Transformers and torch optional #250

szaher opened this issue Jan 10, 2024 · 2 comments
Assignees
Milestone

Comments

@szaher
Copy link

szaher commented Jan 10, 2024

NeMo Guardrails depends on torch/cuda to generate embeddings when using SentenceTransformers models but in other cases that's not required. Can we move the torch import to the sentence transformer class and make it optional if the user is going to use OpenAI embeddings models ?

move it from https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/nemoguardrails/embeddings/basic.py#L19 to https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/nemoguardrails/embeddings/basic.py#L117

The main reason is when installing nemo-guardrails the container image gets really big because of torch and cuda however we might not need them.

@drazvan drazvan self-assigned this Jan 10, 2024
@drazvan
Copy link
Collaborator

drazvan commented Jan 10, 2024

Yes, we can definitely do that. Can you test if this is the only change that is needed?
It is on the roadmap to make SentenceTransformers an optional dependency, but might take another couple of months. The changes you suggested, if enough, can be included in the 0.7.0 release at the end of this month.

@drazvan drazvan added this to the v0.7.0 milestone Jan 10, 2024
@drazvan
Copy link
Collaborator

drazvan commented Jan 31, 2024

This is now covered in https://github.com/NVIDIA/NeMo-Guardrails/releases/tag/v0.7.0. If you don't install SentenceTransformers, it will still work. In 0.7.1, scheduled in a day or so, we'll also add fastembed as the default option, which is much more lightweight.

@drazvan drazvan closed this as completed Jan 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants