Flaky Tests: Are You Sure You Want to Rerun Them?

Different teams have different approaches to dealing with flaky tests. Some even go as far as using the “Let’s run each test 10 times and if it passes at least once, it passed” approach.

I personally think that rerunning failed tests is poisonous — it legitimizes and encourages entropy, and rots the test suite in the long run.

Imported from Disqus

Marcin Raczkowski4 years ago

Most flaky tests I’ve seen deal either with time/date or external/network services.

Tests will fail inconsistently when staging/testing enviornments of vendors are flaky. This is a notorious problem with PayPal for example, which can have their testing API down for hours.

We’re dealing with this in various ways (depending on level of flakiness), with either:

  1. Rerunning tests with increased delays until it succeedes or fails (and does not for example throw network error).
  2. Skipping flaky tests after certain amount of tries.

It’d be good to see extra status: ‘Warning’ similar to ‘skip’ that allows a full suite to ‘Pass with warnings’, then you might decide that this ‘Pass with warnings’ status does not allow for automatical deployments but then you get ‘approve’ button that allows you to deploy.
Similarilly ‘Pass with warnings’ could be defined as ‘Pass’ for integrations like github and allow merging.