add xsimplusplus in retrieval category

[jaygala24]

Checklist for adding MMTEB dataset

Reason for dataset addition:

• I have tested that the dataset runs with the mteb package.
• I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command.
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • intfloat/multilingual-e5-small
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
• I have filled out the metadata object in the dataset file (find documentation on it here).
• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.
• I have added points for my submission to the points folder using the PR number as the filename (e.g. 438.jsonl).

[jaygala24]

Adding xsim++ dataset spanning over 200 languages in retrieval category task.

[KennethEnevoldsen]

A question related to the task formulation

[KennethEnevoldsen]

Bitext mining? What is the reason we formulate it as a retrieval task then?

[jaygala24]

The default formulation for bitext mining expects one-to-one pair mapping between source and target translations where all the examples except the corresponding index act as negative samples. However, in the case of xsim++, there are a variable number of negative samples for each of the English sentences based on the perturbation choices and note that there is no corresponding positive pair (perturbation applied on target language sentences).

The most appropriate task formulation that we think fits with the xsim++ dataset in MTEB is retrieval where each query (target language sentence) can have a variable number of positive (ground truth English sentence) and negative pairs (perturbed English sentences). This is similar to the following task in the retrieval category as of now.

[KennethEnevoldsen]

Thanks @jaygala24, my suspicion was that it was due to the negative examples as well. Do note that it is possible to introduce new task types to MTEB, however, let us keep it as a retrieval task for now.

[Andrian0s]

@jaygala24 @KennethEnevoldsen

Ah now I followed through this conversation. Xsim++ was developed to evaluate Bitext mining systems cross-lingually.

Implementation wise, a Retrieval Evaluation code re-use makes sense. However, the retrieval metrics are not really relevant to the task (there are no evaluations of them within the paper), as they only mention their pairwise models accuracy. Perhaps I would suggest finding a case where this dataset was used beyond pairwise comparison and providing the same metrics.

Conceptually, it is an adversarial evaluation, similar to what Cross Lingual Semantic Discrimination Task (pr coming) will be. When I am through with that addition, it might make sense to have xsim++ as a tab within that section

Moving this also here for visibility:
Isn't this by definition a CrossLingual Task (EN-Other languages) and therefore, shouldn't this extend the CrosslingualTask instead of MultilingualTask? Could this fix the issue without being able to use fast-load?

[Andrian0s]

And one more thing, using the retrieval code, you are encoding the "corpus" - which is again similar size text to the original using model.encode_corpus. In models where this matter (e5-multilingual for example), this would give the wrong prompt to the model, altering the results.

Consider adding a binary variable in RetrievalEvaluator within the search function, which will allow for the case that the corpus also needs to be encoded using the encode_queries. Should be an easy fix and will bring difference in results.

[KennethEnevoldsen]

Is there really only 1012 samples in total?

Any reason to use both the dev and devtest?

[jaygala24]

The dataset has 1012 samples per language and there are 200 languages so in total it would amount to 200k samples.

The dataset has both dev and devtest splits but we can choose to only include the devtest as the eval split as people usually report the result for the same.

[KennethEnevoldsen]

We should remove the dev set (200k is already on the higher end). Please add the fast loading introduced in #572 to avoid extensive long loading times.

[KennethEnevoldsen]

Then feel free to add the points as well as updating the scores using the fast implementation (just to show that the implementation work as intended)