Smart Orchestration
Cypress Cloud provides four distinct Smart Orchestration features for use in CI to speed up test runs, accelerate debugging workflows, and reduce costs:
- Parallelization: Reduce test duration by running multiple test spec files simultaneously across available CI machines.
- Load Balancing: Optimize CI resources and minimize test duration by intelligently prioritizing running of test spec files during parallelization.
- Spec Prioritization: Quickly verify that your latest changes fixed a build by prioritizing the specs that failed in the previous Cypress run.
- Auto Cancellation: Save on CI resources by automatically canceling a Cypress run upon the first test failure.
Spec Prioritization
It is often helpful to be aware of test failures earlier within a CI test run so that debugging and iterations can resume and progress much faster. Being able to catch issues sooner within the CI process can save valuable time in troubleshooting failures and deploying fixes.
Consider the Scenario
For example, you have a test suite consisting of several spec files that take 30 minutes to execute, but failures start to occur within one spec at the 20 minute marker. A subsequent code change aimed at resolving the issue will have to be validated after running the CI build and waiting 20 minutes. This process and waiting continues until the issue is fixed.
By running failed specs first, the issues can be surfaced earlier and reduce that 20 minute wait to potentially a few minutes. Considering that a test suite has multiple spec files, this orchestration strategy will always save time.
Activate Spec Prioritization in Cypress Cloud
Premium Cypress Cloud Feature
Spec Prioritization is a Smart Orchestration feature available to users with a Business or Enterprise tier Cypress Cloud plan. Wondering if upgrading your Team tier subscription is worth it? Check out the recommendations on the run overview page to see how much time we predict these features would save you.
Spec Prioritization is a Smart Orchestration feature that is managed within a project's settings.
To enable or disable this feature at the Project level (must be an admin user):
- Select the desired project within Projects view to visit Cypress Cloud.
- Click "Project Settings" with the right-hand sidebar.
- Scroll to the Smart Orchestration section within Project Settings page.
- Toggle "Spec Prioritization".
CI Tip: Reduce Costs
Note that Spec Prioritization alone will not save time as it purely alters the order in which specs are run. However, when used in tandem with Auto Cancellation it can result in shorter test runs due to cancellation being triggered earlier, leading to significantly reduced CI costs.
Auto Cancellation
Continuous Integration (CI) pipelines are typically costly processes that can demand significant compute time. When a test failure occurs in CI, it often does not make sense to continue running the remainder of a test suite since the process has to start again upon merging of subsequent fixes and other code changes. When Auto Cancellation is enabled, once the number of failed tests goes over a preset threshold, the entire test run is canceled. Note that any in-progress specs will continue to run to completion.
Benefits
Canceling an entire test run, even if parallelized, upon the first test failure will:
- Save time. Resolve test outcomes faster.
- Reduce CI costs. These cost savings can be significant for large test suites.
- Free-up CI resources for validating fixes, and helping other users.
Activate Auto Cancellation in Cypress Cloud
Premium Cypress Cloud Feature
Auto Cancellation is a Smart Orchestration feature available to users with a Business or Enterprise tier Cypress Cloud plan. Wondering if upgrading your Team tier subscription is worth it? Check out the recommendations on the run overview page to see how much time we predict these features would save you.
Auto Cancellation is a Smart Orchestration feature that is managed within a project's settings. Coming soon, you will also have the option to enable the feature on a per-run basis with the Cypress CLI.
Consideration for Teams
If your development, testing, or QA teams operate in a highly collaborative workflow where multiple people are working on various test failures at the same time, it may be helpful to increase the failures threshold or even disable Auto Cancellation altogether, so multiple failing tests can be surfaced for each test run.
To enable or disable this feature at the Project level (must be an admin user):
- Select the desired project within Projects view to visit Cypress Cloud.
- Click "Project Settings" with the right-hand sidebar.
- Scroll to the Smart Orchestration section within Project Settings page.
- Toggle "Auto Cancellation".
- Set the "failures before auto canceling" threshold (or just leave it at the default value of 1).
CI Tip: Save Time
In addition to Auto Cancellation, consider reducing CI costs further by enabling Spec Prioritization to surface test failures earlier in the run, and reduce the time it takes to fix issues and deploy changes.