-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add global mmlu lite sensitivity cards #1568
Conversation
Add two task cards for Global-MMLU-Lite dataset: - CS card for culturally sensitive questions - CA card for culturally agnostic questions Both cards include: - Support for 14 languages - Multiple choice QA format - Topic mapping and preprocessing steps
Add two task cards for Global-MMLU-Lite dataset: - CS card for culturally sensitive questions - CA card for culturally agnostic questions Both cards include: - Support for 14 languages - Multiple choice QA format - Topic mapping and preprocessing steps
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is all the difference between the 3 files of global mmlu is the filtering lambda?
can you just have them in one python file with loop over the lambdas:
for func in [None, "lambda x: x['cultural_sensitivity_label'] == 'CA'", ...
Also can you run make pre-commit
before committing to fix the style of the code (once you run it once it will persist to affect your code before new commits)
Not exactly. I combined the two files of cultural_sensitivity_label, but there are important differences between these files and the
|
feat: add Global-MMLU-Lite CS/CA task cards
Add two task cards for Global-MMLU-Lite dataset:
Both cards include: