Adding new task: Boxes #1557
Adding new task: Boxes #1557
Conversation
irafayabdul
requested review from
haileyschoelkopf and
lintangsutawika
as code owners
|
|
| dataset_path: json | ||
| dataset_name: null | ||
| dataset_kwargs: | ||
| data_files: {'test': 'test-subsample-states-t5.jsonl'} |
There was a problem hiding this comment.
This points to a local file?
There was a problem hiding this comment.
Yes, this points to a file inside dataset dir stored locally. I downloaded dataset from git, it is not on hugging face. How should I go about it excluding uploading data on hugging face?
I can try to replicate only flan-t5-xl results (since GPT3, 3.5, 4 are not an option). Please let me know if I understood it correctly :) Thank you in anticipation
There was a problem hiding this comment.
Flan-T5 results would be great!
Could you open an issue on the authors' github repo and ask them if they would be alright with uploading the dataset to Huggingface as a gated repo?
There was a problem hiding this comment.
Hi Hailey, apologies for delay
I asked Dr. Sebastian Schuster and create this gated repo Boxes task
Further, I am attaching results for flan-t5 base and xl which are almost same as the results presented in paper.
here I tried to produce results based on number of operations affecting box state just as reported in paper but the task does not necessary need this in general hence implementation is kept same.
A task probing to what extent a language model can infer the final state of an entity given an English description of the initial state and a series of state-changing operations is presented.