Specifying dataProvider and successPercentage causes test to always pass

[Yharooer]

TestNG Version

7.10.2

Expected behavior

Expected that the following tests fail because 75% of the tests fail but successPercentage is 99.

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

import static org.testng.Assert.assertEquals;

public class TestTest {
    @Test(dataProvider = "test", successPercentage = 99)
    public void sampleTest(String string) {
        assertEquals(string, "1");
    }

    @DataProvider(name = "test")
    public Object[][] testProvider() {
        return new Object[][]{{"1"}, {"2"}, {"3"}, {"4"}};
    }
}

Actual behavior

All tests pass.

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

import static org.testng.Assert.assertEquals;

public class TestTest {
    @Test(dataProvider = "test", successPercentage = 99)
    public void sampleTest(String string) {
        assertEquals(string, "1");
    }

    @DataProvider(name = "test")
    public Object[][] testProvider() {
        return new Object[][]{{"1"}, {"2"}, {"3"}, {"4"}};
    }
}

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

[Yharooer]

Note that this behaviour changed in TestNG 7.

In TestNG 6.14.3, for the tests above, only "3" and "4" fail, which is still not the correct behaviour.

This changes again at TestNG 6.8.15. AT TestNG 6.8.17, "3" and "4" fail but at TestNG 6.8.15, they all pass.

[Yharooer]

It seems like when using a data provider and successPercentage, TestNG is reporting all failed tests as SUCCESS_PERCENTAGE_FAILURE even when the failure rate is breaching.

When running this:

@Listeners({DebugTestListener.class})
public class TestTest {
    @Test(dataProvider = "test", invocationCount = 100, successPercentage = 99)
    public void sampleTest(String string) {
        assertEquals(string, "1");
    }

    @DataProvider(name = "test")
    public Object[][] testProvider() {
        return new Object[][]{{"2"}};
    }
}

all tests report SUCCESS_PERCENTAGE_FAILURE.