Flaky Test Management

By enabling test retries, Cypress Cloud can detect, flag, and track flaky tests from your recorded Cypress test runs in your CI/CD pipeline.

Flake Detection

One way to battle flaky tests is to detect and monitor them as they occur in an organized and methodical manner such that you can assess their severity to assist with prioritizing their fix.

Flagging Flaky Tests

Test runs with flaky tests will be flagged with the number of flaky of tests within Cypress Cloud "Latest runs" page. Flaky tests runs can also be filtered in and out via the "Flaky" filter within this page.

Flagging flaky tests runs in Cypress Cloud

Any failure across multiple test run attempts triggered by test retrying will result in a given test case to be flagged as flaky.

Flaky Test Analytics

The flaky tests analytics page provides a birds-eye-view on the state of flake within your project by showing:

  • Plot of the number of flaky tests over time.
  • Overall flakiness level of the entire project.
  • Number of flake tests grouped by their severity.
  • Filterable log of all flaky test cases ordered by severity.
Flaky tests analytics

Selecting any of the flaky test cases in the analytics page will reveal a test case details panel that shows:

  • Historical log of latest flaky runs
  • Most common errors across the runs of the test case
  • Related test case changelog
  • Plot of failure rate and flaky rate over time.

All these test case level details provide deeper context around the occurrences of flake over time to assist with debugging the root cause.

Flaky tests analytics details panel

Failure Rate vs Flake Rate

It is important to understand the distinction between failure rate and flake rate when test retries is enabled. These two metrics are tracked over time for each flaky test case within the flaky tests analytics page:

flake rate vs fail rate

A test case flagged as flaky could have still passed after multiple test retry attempts. The test result status of individual test retry attempts is separate and distinct from the final test status.

For example, a project is configured to retry failing tests up to 3 times. The first two attempts fail, but the last and third attempt passes, resulting in a final status of passing.

With this concept in mind, it is possible to always have zero final failure rate while exhibiting flake as demonstrated below:

flake rate vs fail rate

Flake Alerting

Cypress Cloud can provide alerts via GitHub and Slack integrations, to further assist with staying on top of flake occurrences.

GitHub

Flake alerting via GitHub PR comments and status checks can be enabled within a project's GitHub integration settings:

GitHub flake alert settings

After enabling GitHub flake alerting, GitHub PR comments will show the number of flaky tests associated with the PR within the test summary, and include a "Flakiness" section highlighting the specific tests that flaked.

GitHub flake alert pr comment

Slack

Flake alerting via Slack can be enabled within Slack integration settings:

Slack flake alert settings

After enabling Slack alerts, Cypress Cloud will send Slack messages whenever flaky tests are detected:

Slack flake alert

See Also