Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collapse reference+learner hydra heads when using LoRa #320

Open
jon-tow opened this issue Feb 21, 2023 · 6 comments
Open

Collapse reference+learner hydra heads when using LoRa #320

jon-tow opened this issue Feb 21, 2023 · 6 comments
Labels
contributions welcome PR from open source contributors welcome to solve this issue. feature request New feature or request good first issue Good for newcomers

Comments

@jon-tow
Copy link
Collaborator

jon-tow commented Feb 21, 2023

🚀 The feature, motivation, and pitch

With additive (delta-style) parameter-efficient tuning methods such as LoRa, we should be able to make a slightly more mem-efficient hydra architecture by using a single block that does ~frozen_head + tunable_weights for the learner/policy head's fwd-pass and simply frozen_head for the reference, instead of maintaining 2x heads.

CC @LouisCastricato and @cat-state for pointing this out

Alternatives

No response

Additional context

No response
@jon-tow jon-tow added feature request New feature or request contributions welcome PR from open source contributors welcome to solve this issue. good first issue Good for newcomers labels Feb 21, 2023
@LouisCastricato
Copy link
Contributor

Haha, I was not aware that Aman proposed the same thing.

@jon-tow jon-tow changed the title Collapse reference-learner hydra heads when using LoRa Collapse reference+learner hydra heads when using LoRa Feb 21, 2023
@Opdoop
Copy link

Opdoop commented Apr 18, 2023

When will this feature be available?

@LouisCastricato
Copy link
Contributor

I am not sure anyone started. cc @jon-tow

@jon-tow
Copy link
Collaborator Author

jon-tow commented Apr 18, 2023 •
edited
Loading

Not yet. @glerzing is looking into peft migration, which should make this very simple as the package provides a context manager to disable adapters; then, we can use it over reference model calls:

https://github.com/huggingface/peft/blob/34027fe813756897767b9a6f19ae7f1c4c7b418c/src/peft/peft_model.py#L290-L299

@Opdoop
Copy link

Opdoop commented Apr 19, 2023

Cool!!! Looking forward to this update. 👍

@glerzing
Copy link
Contributor

Sorry to make you wait, but it should take a few weeks to get this done. As a very rough estimate, I would say that I may push a tested solution around the 10th may. But I'm new here so I don't know how much time would then happen before a new release version.

glerzing added a commit to glerzing/trlx that referenced this issue May 13, 2023
@glerzing
Migrate to peft from opendelta for parameter efficient tuning methods (…
32bbebe 
…CarperAI#434) +  Collapse reference+learner hydra heads when using LoRa (CarperAI#320)
glerzing added a commit to glerzing/trlx that referenced this issue May 13, 2023
@glerzing
Migrate to peft from opendelta for parameter efficient tuning methods (…
267c971 
…CarperAI#434) +  Collapse reference+learner hydra heads when using LoRa (CarperAI#320)
glerzing added a commit to glerzing/trlx that referenced this issue May 23, 2023
@glerzing
Migrate to peft from opendelta for parameter efficient tuning methods (…
3430029 
…CarperAI#434) +  Collapse reference+learner hydra heads when using LoRa (CarperAI#320)
glerzing added a commit to glerzing/trlx that referenced this issue May 23, 2023
@glerzing
Migrate to peft from opendelta for parameter efficient tuning methods (…
66a56f5 
…CarperAI#434) +  Collapse reference+learner hydra heads when using LoRa (CarperAI#320)
glerzing added a commit to glerzing/trlx that referenced this issue May 24, 2023
@glerzing
Migrate to peft from opendelta for parameter efficient tuning methods (…
9a86e63 
…CarperAI#434) +  Collapse reference+learner hydra heads when using LoRa (CarperAI#320)
glerzing added a commit to glerzing/trlx that referenced this issue May 24, 2023
@glerzing
Migrate to peft from opendelta for parameter efficient tuning methods (…
5abd209 
…CarperAI#434) +  Collapse reference+learner hydra heads when using LoRa (CarperAI#320)
glerzing added a commit to glerzing/trlx that referenced this issue May 24, 2023
@glerzing
Migrate to peft from opendelta for parameter efficient tuning methods (…
1e73234 
…CarperAI#434) +  Collapse reference+learner hydra heads when using LoRa (CarperAI#320)
glerzing added a commit to glerzing/trlx that referenced this issue May 24, 2023
@glerzing
Migrate to peft from opendelta for parameter efficient tuning methods (…
7e48164 
…CarperAI#434) +  Collapse reference+learner hydra heads when using LoRa (CarperAI#320)
@glerzing glerzing mentioned this issue May 24, 2023
Merged
jon-tow added a commit that referenced this issue Jun 23, 2023
@glerzing @jon-tow @maxreciprocate
peft to opendelta migration (#434) + memory optimization (#320) (#486)
d47996d 
* Migrate to peft from opendelta  for parameter efficient tuning methods (#434) +  Collapse reference+learner hydra heads when using LoRa (#320)

* fix from_config

* Review corrections

* ILQL generate when temperature is 0.

* revert: guard against experimental 8-bit loading support

* format: run `black`

---------

Co-authored-by: jon-tow <[email protected]>
Co-authored-by: maxreciprocate <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
contributions welcome PR from open source contributors welcome to solve this issue. feature request New feature or request good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@LouisCastricato @Opdoop @jon-tow @glerzing