-
Notifications
You must be signed in to change notification settings - Fork 27.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generate: TF contrastive search with XLA support #20050
Conversation
The documentation is not available anymore as the PR was closed or merged. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very clean, thanks for doing all the ground work in the PyTorch method in previous PRs!
(@Rocketknight1 is off, so I'm merging this to not slow down the corresponding blog post, which will contain TF examples thanks to this PR :D In any case, have a quick look when you're back, to ensure we kill any bad pattern before v4.25 gets released!) |
* Add contrastive search
What does this PR do?
Adds contrastive search to TF, with XLA support.
In essence, TF's contrastive search is very similar to PT's, adapted to the structure that is present in other TF XLA generation functions (i.e. has a dedicated function to loop over, a separate function to update
model_kwargs
when in XLA mode, ...). The most notable difference is how the best candidate token (and associated model variables) are gathered -- PT relies on slicing, which TF doesn't support, so atf.gather
alternative is used.The exact same integration tests (with the same input, model, and outputs) were added whenever possible. Three integration tests were not added, which will be addressed in a follow-up PR: