Field note
Tracing flaky tests to environment drift
A test that passes locally and fails in CI—or only on Tuesdays—often gets labelled flaky and muted. Before muting, spend twenty minutes proving whether the environment moved under the spec.
Compare fingerprints
Log language runtime versions, timezone, locale, feature-flag snapshots, and container image digests for both environments. Drift in any of those can change formatted dates, sort order, or async timing without a code change.
Data parity
Shared staging databases accrue orphaned rows from manual QA. A test that assumes an empty cart may fail when someone left a session open. Seed data per job or namespace transactions instead of reusing a communal schema without resets.
External dependencies
DNS TTL changes, throttled sandboxes, and corporate proxies show up as intermittent connection resets. Capture tcp dumps or structured HTTP logs once, then replace live calls with fixtures if the vendor is outside your control.
Clock and concurrency
Tests that sleep for two seconds assume the CI worker is not oversubscribed. Prefer awaiting explicit conditions. If load is the issue, reduce parallelism for that job rather than increasing sleeps everywhere.
Document the verdict
When drift is the root cause, file a ticket against infrastructure with reproduction steps. When the test is genuinely nondeterministic, quarantine it with an owner and expiry date. Ambiguous muting is how suites lose trust.
We walk through this checklist during CI Pipeline Advisory weeks when failure logs are the primary artifact.