Can anyone reproduce the baseline results reported in the paper with the current version of codes?

[bozheng-hit]

I think the following codes should be inserted here: https://github.com/alexa/massive/blob/main/src/massive/utils/training_utils.py#L472

for i, x in enumerate(lab): 
    if x == -100: 
        pred[i] = -100

[bozheng-hit]

I am able to reproduce the results after adding the above codes.
I strongly suggest the contributors at least run the codes for one time before committing.

[jgmf-amazon]

Hi @bozheng-hit , our apologies for the extra debugging that you had to perform, and thanks for proposing a solution. I am able to reproduce this problem, namely that the validation engine gives bad results for encoder-only models when using the new evaluation code. PR here: #22

[jgmf-amazon]

Please re-open if you still see the problem. Thank you!