Create new IntersectCorrespondingFields operator

[pklpriv]

In this PR Intersect() operator is updated. It allows us to filter out some values from data when running benchmarks. Additionally, test were updated and rewritten for more readability.

[elronbandel]

Two issues:
(1) This is not backward compatible, what if anyone using it now?
(2) The same behavior can be achieved with current implementation:
Intersect(field_to_field={"label": "label", "name": "name"}, allowed_values=["b", "f"])

Am I missing something?

[yoavkatz]

Two issues: (1) This is not backward compatible, what if anyone using it now? (2) The same behavior can be achieved with current implementation: Intersect(field_to_field={"label": "label", "name": "name"}, allowed_values=["b", "f"])

Am I missing something?

This relates to a need (e.g in ner datasets) to filter multiple fields together ( e.g keep only two entity types and remove the start ,end, and label fields).

The current code changes are not backwards compatible (e.g hard code the field to " labels").

I'll write more details later on how to address.

[yoavkatz]

To keep things backward compatible, but still consistent - I've add a new Operator called IntersectCorrespondingFields.

"""Intersects the value of a field, which must be a list, with a given list , and removes corresponding elements from other list fields.

For example:

Assume the instances contain a field of 'labels' and a field with the labels' corresponding 'positions' in the text.

IntersectCorrespondingFields(field="label",
                             allowed_values=["b", "f"],
                             corresponding_fields_to_intersect=["position"])

would keep only "b" and "f" values in 'labels' field and
their respective values in the 'position' field.
(All other fields are not effected)

Given this input:

[
    {"label": ["a", "b"],"position": [0,1],"other" : "not"},
    {"label": ["a", "c", "d"], "position": [0,1,2], "other" : "relevant"},
    {"label": ["a", "b", "f"], "position": [0,1,2], "other" : "field"}
]

So the output would be:
[
        {"label": ["b"], "position":[1],"other" : "not"},
        {"label": [], "position": [], "other" : "relevant"},
        {"label": ["b", "f"],"position": [1,2], "other" : "field"},
]

Args:
    field - the field to intersected (must contain list values)
    allowed_values (list) - list of values to keep
    corresponding_fields_to_intersect (list) - additional list fields from which values
    are removed based the corresponding index of values removed from the 'field'
"""

Also updated the test cases.

Fixed