Imported from Disqus
Marcin Raczkowski • 4 years ago
Most flaky tests I’ve seen deal either with time/date or external/network services.
Tests will fail inconsistently when staging/testing enviornments of vendors are flaky. This is a notorious problem with PayPal for example, which can have their testing API down for hours.
We’re dealing with this in various ways (depending on level of flakiness), with either:
- Rerunning tests with increased delays until it succeedes or fails (and does not for example throw network error).
- Skipping flaky tests after certain amount of tries.
It’d be good to see extra status: ‘Warning’ similar to ‘skip’ that allows a full suite to ‘Pass with warnings’, then you might decide that this ‘Pass with warnings’ status does not allow for automatical deployments but then you get ‘approve’ button that allows you to deploy.
Similarilly ‘Pass with warnings’ could be defined as ‘Pass’ for integrations like github and allow merging.