Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LoRA typo fix + bias=True #1881

Merged
merged 3 commits into from
Oct 22, 2024
Merged

LoRA typo fix + bias=True #1881

merged 3 commits into from
Oct 22, 2024

Conversation

felipemello1
Copy link
Contributor

@felipemello1 felipemello1 commented Oct 22, 2024 •
edited
Loading

Context

What is the purpose of this PR? Is it to

  • add a new feature
  • fix a bug
  • update tests and/or documentation
  • other (please add here)

Test plan

tune run --nproc_per_node 2 lora_finetune_distributed --config llama3_2_vision/11B_lora model.apply_lora_to_mlp=True model.apply_lora_to_output=True metric_logger=torchtune.training.metric_logging.WandBLogger compile=True batch_size=4 log_peak_memory_stats=True

image

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Oct 22, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1881

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 5 Cancelled Jobs

As of commit aba63fe with merge base ca37c59 (image):

NEW FAILURES - The following jobs have failed:

  • Build Docs / build_docs (3.11) (gh)
    sphinx.ext.autosummary.ImportExceptionGroup: no module named torchtune.training
  • GPU tests / gpu_test (3.11, stable) (gh)
    E ImportError: cannot import name 'TensorCoreTiledLayout' from 'torchao.dtypes' (/home/ec2-user/actions-runner/_work/torchtune/torchtune/3/envs/test/lib/python3.11/site-packages/torchao/dtypes/__init__.py)
  • Recipe Tests / recipe_test (3.10) (gh)
    E ImportError: cannot import name 'TensorCoreTiledLayout' from 'torchao.dtypes' (/usr/share/miniconda3/envs/test/lib/python3.10/site-packages/torchao/dtypes/__init__.py)
  • Recipe Tests / recipe_test (3.9) (gh)
    E ImportError: cannot import name 'TensorCoreTiledLayout' from 'torchao.dtypes' (/usr/share/miniconda3/envs/test/lib/python3.9/site-packages/torchao/dtypes/__init__.py)
  • Unit Test / unit_tests (3.9) (gh)
    E ImportError: cannot import name 'TensorCoreTiledLayout' from 'torchao.dtypes' (/usr/share/miniconda3/envs/test/lib/python3.9/site-packages/torchao/dtypes/__init__.py)

CANCELLED JOBS - The following jobs were cancelled. Please retry:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 22, 2024
felipemello1
felipemello1 commented Oct 22, 2024
View reviewed changes
torchtune/models/clip/_component_builders.py
Comment on lines +12 to +30
from torchtune.models.clip._position_embeddings import (
TiledTokenPositionalEmbedding,
TilePositionalEmbedding,
TokenPositionalEmbedding,
)

from torchtune.modules import (
TransformerSelfAttentionLayer,
FeedForward,
Fp32LayerNorm,
MultiHeadAttention,
TanhGate,
FeedForward,
Fp32LayerNorm
TransformerSelfAttentionLayer,
)

from torchtune.modules.common_utils import reparametrize_as_dtype_state_dict_post_hook

from torchtune.modules.peft import DoRALinear, LORA_ATTN_MODULES, LoRALinear

from torchtune.modules.vision_transformer import CLSProjection, VisionTransformer
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

precommit hook reordering

torchtune/models/clip/_component_builders.py
Comment on lines +506 to +521
in_dim=in_dim,
out_dim=hidden_dim,
rank=lora_rank,
alpha=lora_alpha,
dropout=lora_dropout,
quantize_base=quantize_base,
use_bias=True,
)
down_proj = adapter_cls(
in_dim=hidden_dim,
out_dim=dim,
out_dim=out_dim,
rank=lora_rank,
alpha=lora_alpha,
dropout=lora_dropout,
quantize_base=quantize_base,
use_bias=True,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this changed

pbontrager
pbontrager approved these changes Oct 22, 2024
View reviewed changes
Copy link
Contributor

@pbontrager pbontrager left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing the lora builder

@felipemello1 felipemello1 merged commit 1e5f0d5 into pytorch:main Oct 22, 2024
7 of 17 checks passed
@felipemello1 felipemello1 deleted the lora_typo branch October 22, 2024 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pbontrager pbontrager pbontrager approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@felipemello1 @pbontrager @facebook-github-bot