Add BibleNLP dataset

[davidstap]

Checklist for adding MMTEB dataset

This dataset has an extremely large coverage of 829 languages, and includes numerous low-resource languages. For this reason I only considered English-centric directions, resulting in a reasonable 1656 pairs. I will add performance results later, but initial testing suggests that while quality is not good for most (low-resource) languages, it is far better than random guesses, so IMO a good addition to the benchmark.

• I have tested that the dataset runs with the mteb package.
• I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command.
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • intfloat/multilingual-e5-small
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
• I have filled out the metadata object in the dataset file (find documentation on it here).
• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.
• I have added points for my submission to the points folder using the PR number as the filename (e.g. 438.jsonl).

[davidstap]

Could you review this PR @KennethEnevoldsen @imenelydiaker? I'm personally quite excited by the large language coverage.

[isaac-chung]

This is awesome! Just a few comments.

[dokato]

Really interesting dataset :) find some of my suggestions below

[davidstap]

I have addressed your comments @isaac-chung @dokato, thanks for your suggestions!

Results are included now as well. intfloat/multilingual-e5-small performs best, but like I pointed out the performance is bad for most low-resource languages. However, almost all directions are way better than random chance (1/256), so this seems like a valuable addition to evaluate (future) multilingual models.

I think this is ready to merge now, do you agree? Let me know, then I'll calculate the points. Thanks!

[isaac-chung]

Thanks for iterating! I think it's good enough to merge. In terms of points, the dataset is 2 + 4 * number of new languages added to the BitextMining task. Please check if any of the languages have already been covered in that folder and the multilingual subfolder as well. Then 2 pts each to @dokato and I for the reviews.

[davidstap]

Thanks for your quick response! I calculated the number of new languages using this script: 756 new languages (out of 829 languages used in this task).

I'll update the points and merge.

[davidstap]

Ready to merge now @isaac-chung (I don't have write access)