Failure analysis & flaky-test classifier
The problem
Every night 40 tests fail; someone spends the morning sorting real bugs from flaky noise.
Workflow
- CI resultsfailures + logs
- Groupby error signature
- LLM + historyreal bug / flaky / env / data
- Reportper group, with evidence
- Owneracts on 'real bug' first
Real-world example
38 failures collapse into 4 groups: 31 share 'ECONNREFUSED auth-service' (environment), 4 are known flaky timeouts, 2 assertion failures on discount price (real bug), 1 expired test user (data).
Starter prompt
Classify each failure group as REAL_BUG, FLAKY, ENVIRONMENT or TEST_DATA. Use the error, the last 10 runs history and recent commits. Give evidence for each. Return a table sorted by REAL_BUG first.
Tools
CI (GitHub Actions/Jenkins), test reports, LLM
Human check
Re-run 'flaky' tests before ignoring; a wrong 'flaky' label hides real bugs.
Measure success
Morning triage time; real bugs found before standup; flaky-test count trend.