TestNG Success Percentage Below 100
ID |
java.testng_success_percentage |
Severity |
low |
Remediation Complexity |
trivial |
Remediation Risk |
low |
Remediation Effort |
low |
Resource |
Reliability |
Language |
Java |
Tags |
reliability, testing |
Rationale
TestNG’s successPercentage attribute lets a test keep passing in the suite even when a share of its invocations fail — it is meant as a temporary tolerance for a test known to be flaky, run together with invocationCount. Left in place, it hides the underlying instability: the test suite reports green while some runs are silently failing, and nobody is on the hook to fix the flakiness because the test never turns red.
@Test(successPercentage = 80, invocationCount = 10)
public void reconcilesWarehouseStock() {
// allowed to fail twice out of every ten runs
}Remediation
Find and fix the source of the flakiness — timing assumptions, shared mutable state, ordering between tests — then drop successPercentage (or set it back to 100) so the test must pass every time.
@Test(invocationCount = 10)
public void reconcilesWarehouseStock() {
// must pass on every run
}