Skip to content

Commit

Permalink
remove group from bigbench task configs (#2663)
Browse files Browse the repository at this point in the history
* remove group from task configs

* add tags

* update readme
baberabb authored Jan 29, 2025
1 parent 4b4b036 commit fe9c570
Showing 4 changed files with 9 additions and 3 deletions.
6 changes: 6 additions & 0 deletions lm_eval/tasks/bigbench/README.md
Original file line number Diff line number Diff line change
@@ -30,6 +30,12 @@ Homepage: https://github.com/google/BIG-bench

* `group_name`: `Short description`

#### Tags

* `bigbench_generate_until`
* `bigbench_multiple_choice_a`
* `bigbench_multiple_choice_b`

#### Tasks

* `task_name`: `1-sentence description of what this particular task does`
2 changes: 1 addition & 1 deletion lm_eval/tasks/bigbench/generate_until_template_yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
group: bigbench_generate_until
tag: bigbench_generate_until
dataset_path: hails/bigbench
output_type: generate_until
dataset_kwargs:
2 changes: 1 addition & 1 deletion lm_eval/tasks/bigbench/multiple_choice_template_a_yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
group: bigbench_multiple_choice
tag: bigbench_multiple_choice_a
dataset_path: hails/bigbench
dataset_kwargs:
# num_shots: 0 # TODO: num of shots for `bigbench` HF dataset should be controlled through this, not through the typical methods
2 changes: 1 addition & 1 deletion lm_eval/tasks/bigbench/multiple_choice_template_b_yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
group: bigbench_multiple_choice
tag: bigbench_multiple_choice_b
dataset_path: hails/bigbench
dataset_kwargs:
# num_shots: 0 # TODO: num of shots for `bigbench` HF dataset should be controlled through this, not through the typical methods

0 comments on commit fe9c570

Please sign in to comment.