Korbench

[epsilondylan]

Motivation

This PR introduces the implementation of the KOR-Bench dataset and evaluator into the project. The goal is to support the evaluation of models on KOR-Bench tasks, enabling researchers and developers to assess model performance on various reasoning tasks in the Korean language.

Modification

• Dataset Implementation:

  • Added korbenchDataset class for loading and processing the KOR-Bench dataset.
  • Included support for multiple tasks (cipher, logic, operation, puzzle, counterfactual, and mixed) and modes (zero-shot, three-shot, subquestions).
  • Implemented data loading functions such as read_yaml, read_json_or_jsonl, and read_json_or_jsonl_with_idx to handle dataset files and configurations.

• Evaluator Implementation:

  • Developed KorBenchEvaluator class for evaluating model responses against the KOR-Bench benchmarks.
  • Integrated evaluation functions like evaluate_responses and evaluate_response_vs_answer to compare model outputs with ground truth answers.
  • Added support for calculating accuracy, pass rates, and handling special cases like mixed modes and counterfactual reasoning.

• Utilities and Helper Functions:

  • Added helper functions for text extraction, cleaning, and comparison (e.g., extract_text_from_brackets, compare_math_expressions).
  • Included logging for better debugging and information tracking.
  • Ensured compatibility with the OpenCompass framework by registering the dataset and evaluator modules.

• Documentation and Testing:

  • Updated docstrings and comments for better code understanding.
  • Added unit tests to cover the new functionalities and ensure correctness.

BC-breaking (Optional)

This PR does not introduce any backward compatibility issues. Existing functionalities and modules remain unaffected.

Use cases (Optional)

• Model Evaluation: Researchers can now evaluate their models on KOR-Bench tasks using the provided dataset and evaluator classes.
• Benchmarking: The implementation allows for consistent benchmarking across different models and configurations on Korean reasoning tasks.
• Flexible Configurations: Supports various modes (zero-shot, three-shot, subquestions) and tasks, providing flexibility in evaluation settings.

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug is added in the unit tests.
• The modification is covered by complete unit tests. Added unit tests ensure the correctness of the new code.
• The documentation has been modified accordingly, including docstrings and example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
• CLA has been signed and all committers have signed the CLA in this PR.

[MaiziXiao]

LGTM