Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Support new accelerate api #33

Draft
wants to merge 22 commits into
base: master
Choose a base branch
from
Draft

[WIP] Support new accelerate api #33

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
b9b27dc
Fix padding side
thomasw21 Jul 12, 2022
8599efe
Wrong assumption
thomasw21 Jul 12, 2022
1e0dd7e
Remove wrong comment
thomasw21 Jul 12, 2022
d1ebec7
Update accelerate API
thomasw21 Jul 13, 2022
d3e7e52
Better pathing
thomasw21 Jul 13, 2022
de18a76
Woops
thomasw21 Jul 13, 2022
49dd828
Woops
thomasw21 Jul 13, 2022
9776cd9
Woops
thomasw21 Jul 13, 2022
23ac6c8
Logs arguments + prevent from overwriting existing results
thomasw21 Jul 13, 2022
391759a
Improve API to run all templates
thomasw21 Jul 13, 2022
dd93bb2
Hack my way so that 'None' string is considered like None
thomasw21 Jul 13, 2022
624b621
Woops
thomasw21 Jul 13, 2022
f5a7fbd
Allow distributed computation of metrics
thomasw21 Jul 13, 2022
83a2751
This is hard
thomasw21 Jul 13, 2022
b187b26
Allow one to erase
thomasw21 Jul 13, 2022
40e8cb0
Hopefully this is much better
thomasw21 Jul 13, 2022
7107053
Help with distributed training
thomasw21 Jul 13, 2022
50c27b5
added template_config_name arg so that the dataset and prompt templat…
Jul 15, 2022
e02813d
Add a hack for me
thomasw21 Jul 15, 2022
3ad79d2
escape /
thomasw21 Jul 15, 2022
ec346e9
Fix t-zero
thomasw21 Jul 17, 2022
0300d08
Remove debug results
thomasw21 Jul 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions evaluation/run_eval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,12 @@ def parse_args():
default=None,
help="The configuration name of the dataset to use (via the datasets library).",
)
parser.add_argument(
"--template_config_name",
type=str,
default=None,
help="The name of the dataset_config_name of the template we want to use, example: use XNLI En prompts for XNLI Fr",
)
parser.add_argument(
"--template_name",
type=str,
Expand Down Expand Up @@ -366,10 +372,16 @@ def main():

# Get the prompt to apply and the possible targets.
# TODO(Victor): If pulling from pre-processed data, remove this logic.

if args.dataset_config_name is None or args.template_config_name is None or args.dataset_name == "anli":
Copy link
Member Author

@thomasw21 thomasw21 Jul 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woops merged too fast, this should've been

if (args.dataset_config_name is None and args.template_config_name is None) or args.dataset_name == "anli"

cc @lintangsutawika

prompt_dataset_name = f"{args.dataset_name}"
elif args.template_config_name is not None:
prompt_dataset_name = f"{args.dataset_name}/{args.template_config_name}"
else:
prompt_dataset_name = f"{args.dataset_name}/{args.dataset_config_name}"

prompts = DatasetTemplates(
f"{args.dataset_name}"
if args.dataset_config_name is None or args.dataset_name == "anli"
else f"{args.dataset_name}/{args.dataset_config_name}"
prompt_dataset_name
)

if args.template_name is not None:
Expand Down