Software is a binary affair. Zeros and ones. True or false. So-called “flaky” tests contradict this simple principle: they are tests that produce different results for the same tasks. With a new plug-in, Gradle wants to discreetly get rid of these troublemakers.
Time is a factor that should not be neglected in software development. Deadlines, customer demands and security gaps make it necessary to deliver or update software on a regular basis. Of course, the whole thing should, if possible, not cause any new problems due to errors and bugs, which is why testing is necessary. But what to do if a test is sometimes successful and sometimes not, without anything having been changed? So-called flaky tests can clog up entire continuous delivery or continuous integration pipelines and delay everything.
One solution is to have several successive build processes – you just try until one goes through. Since this is a rather suboptimal approach, Gradle has now provided a new plug-in to help identify, prioritize and fix flaky tests. Unlike other tools, this does not repeat the entire build process to find out if a test is “flaky”. Instead, only the faulty tests are repeated to identify the “flakiness”.
The advantage of all this is that no changes to the source code are required. The plug-in even allows the default to automatically wave through or fail builds in which flaky tests are undoubtedly present. Also, how often a test should be repeated can be set individually via maxFailures parameters. Last but not least, tests are performed at the method level or even more finely granulated, so that entire classes do not have to be retested each time. By default (this is also customizable), all builds are waved through where the tests succeed after a certain number of retries, making the build successful.
The plug-in is available for Eclipse, NetBeans and IntelliJ IDEA, and has also been successfully used with JUnit 4 (4.12), JUnit 5 (5.5.2), Spock (1.3-groovy-2.5) and TestNG (7.0.0). It is compatible with all versions starting with Gradle 5.0 and can be obtained from the Gradle plug-in portal with the ID org.gradle.test-retry.
More information is available on the project’s GitHub page and on Gradle’s blog.