Unstable Test
ID |
javascript.unstable_test |
Severity |
high |
Remediation Complexity |
medium |
Remediation Risk |
low |
Remediation Effort |
medium |
Resource |
Reliability |
Language |
JavaScript |
Tags |
reliability, testing |
Description
Reports a test or hook that draws on an unpredictable value: a random number (Math.random(), crypto.randomUUID()) or the current clock (Date.now(), new Date(), performance.now()).
Scoped to code that runs as part of a test — a test callback or a lifecycle hook. Randomness elsewhere in a spec file, in a fixture factory called with an explicit seed or in the module’s own setup, is left alone. new Date(2024, 0, 1) is not reported: a date built from arguments is a fixed instant, and it is the no-argument form that reads the clock.
Deliberately out of scope: a fixed wait, which makes a test flaky for a different reason — a race rather than an unpredictable input.
Rationale
// Bad
it('discounts the order', () => {
const order = buildOrder(Math.random() * 100); // a different order every run
expect(discount(order)).toBeLessThan(order.total);
});
it('expires the day-old token', () => {
const token = issue({ at: Date.now() - DAY }); // crosses a date boundary once a day
expect(isExpired(token)).toBe(true);
});-
Such a test does not test one thing; it tests a different thing on each run, so a pass says nothing about the case that matters and a failure cannot be repeated.
-
It passes for months and then fails once in CI on a value nobody can reproduce. The usual response — re-run the job — teaches the team to ignore the suite’s verdict, which costs far more than the test was worth.
-
A random value is not extra coverage: one arbitrary case per run is fewer cases than a written table of them, and the boundary cases a reviewer would pick are the ones randomness almost never hits.
-
Clock-dependent tests fail on a schedule — at a month end, in another timezone, during a daylight-saving shift — which is when the least attention is available to diagnose them.
Remediation
Fix the input: a literal, a fixture, a seeded generator, or an injected clock.
// Good — the cases are written down, so a failure names one
it.each([[10, 1], [100, 15], [0, 0]])('discounts an order of %i', (total, expected) => {
expect(discount(buildOrder(total))).toBe(expected);
});
// Good — the clock is an input, so the boundary can be tested on purpose
it('expires the day-old token', () => {
const now = new Date('2024-01-02T00:00:00Z');
const token = issue({ at: new Date('2024-01-01T00:00:00Z') });
expect(isExpired(token, now)).toBe(true);
});Where a test really is about a range of values, a property-based generator with a recorded seed keeps the failure reproducible; an unseeded Math.random() does not.