docs: 5.0.0 release docs test retries

source/_data/sidebar.yml

@@ -28,6 +28,7 @@ guides:
     module-api: module-api.html
     debugging: debugging.html
     network-requests: network-requests.html
+    test-retries: test-retries.html
     continuous-integration: continuous-integration.html
     parallelization: parallelization.html
     environment-variables: environment-variables.html

source/_partial/allowed_test_config.md

@@ -8,6 +8,7 @@
 - `env`
 - `requestTimeout`
 - `responseTimeout`
+- `retries`
 - `viewportHeight`
 - `viewportWidth`
 - `waitForAnimations`

source/guides/core-concepts/retry-ability.md

@@ -12,6 +12,10 @@ title: Retry-ability
 
 A core feature of Cypress that assists with testing dynamic web applications is retry-ability. Like a good transmission in a car, it usually works without you noticing it. But understanding how it works will help you write faster tests with fewer run-time surprises.
 
+{% note info "Test Retries" %}
+If you are looking to retry tests a configured number of times when the test fails, check out our official guide on {% url "Test Retries" test-retries %}.
+{% endnote %}
+
 # Commands vs assertions
 
 There are two types of methods you can call in your Cypress tests: **commands** and **assertions**. For example, there are 6 commands and 2 assertions in the test below.
@@ -436,3 +440,4 @@ Watch the short video below to see this example in action
 - You can add retry-ability to your own {% url "custom commands" custom-commands %}; see {% url 'this pull request to cypress-xpath' https://github.com/cypress-io/cypress-xpath/pull/12/files %} for an example.
 - You can retry any function with attached assertions using this 3rd party plugin {% url cypress-pipe https://github.com/NicholasBoll/cypress-pipe %}.
 - See retry-ability examples in the {% url "Cypress should callback" https://glebbahmutov.com/blog/cypress-should-callback/ %} blog post.
+- To learn how to enable Cypress' test retries functionality, which retries tests that fail up to the configured number, check out our official guide on {% url "Test Retries" test-retries %}.

source/guides/core-concepts/writing-and-organizing-tests.md

@@ -331,33 +331,39 @@ specify(name, config, fn)
 
 {% partial allowed_test_config %}
 
-### Suite of test configuration
+### Suite configuration
 
-You can configure the size of the viewport height and width within a suite.
+If you want to target a suite of tests to run or be excluded when run in a specific browser, you can override the `browser` configuration within the suite configuration. The `browser` option accepts the same arguments as {% url "`Cypress.isBrowser()`" isbrowser %}.
 
 ```js
-describe('page display on medium size screen', {
-  viewportHeight: 1000,
-  viewportWidth: 400
-}, () => {
-  it('does not display sidebar', () => {
-    cy.get('#sidebar').should('not.be.visible')
+describe('When in Chrome', {  browser: '!chrome' } () => {
+  it('Shows warning', () => {
+    cy.get('.browser-warning')
+      .should('contain', 'For optimal viewing, use Chrome browser')
   })
 
-  it('shows hamburger menu', () => {
-    cy.get('#header').find('i.menu').should('be.visible')
+  it('Links to browser compatibility doc', () => {
+    cy.get('a.browser-compat')
+      .should('have.attr', 'href')
+      .and('include', 'browser-compatibility)
   })
 })
 ```
 
 ### Single test configuration
 
-If you want to target a test to run or be excluded when run in a specific browser, you can override the `browser` configuration within the test configuration. The `browser` option accepts the same arguments as {% url "`Cypress.isBrowser()`" isbrowser %}.
+You can configure the number of times to retry a test if they fail during `cypress run` or `cypress open` separately. See {% url "Test Retries" test-retries %} for more information.
 
 ```js
-it('Show warning outside Chrome', {  browser: '!chrome' }, () => {
-  cy.get('.browser-warning')
-    .should('contain', 'For optimal viewing, use Chrome browser')
+it('should redirect unauthenticated user to sign-in page', {
+    retries: {
+      runMode: 3,
+      openMode: 2
+    }
+  } () => {
+    cy.visit('/')
+    // ...
+  })
 })
 ```
 

source/guides/guides/test-retries.md

@@ -0,0 +1,200 @@
+---
+title: Test Retries
+---
+
+{% note info %}
+# {% fa fa-graduation-cap %} What you'll learn
+
+- What are test retries?
+- Why is test retries important?
+- How to configure test retries
+{% endnote %}
+
+# Introduction
+
+End-to-end (E2E) tests excel at testing complex systems. However, there are still behaviors that are hard to verify and make tests flaky (i.e., unreliable) and fail sometimes due to unpredictable conditions (eg., temporary outages in external dependencies, random network errors, etc.).  Some other common race conditions that could result in unreliable tests include:
+
+- Animations
+- API calls
+- Test server / database availability
+- Resource dependencies availability
+- Network issues
+
+With test retries, Cypress is able to retry failed tests to help reduce test flakiness and continuous integration (CI) build failures. By doing so, this will save your team valuable time and resources so you can focus on what matters most to you.
+
+# How It Works
+
+By default, tests will not retry when they fail. You will need to {% urlHash "enable test retries in your configuration" Configure-Test-Retries %} to use this feature.
+
+When test retries is on, tests will automatically be retried up to X additional times based on your desired configuration. For example, if test retries is set to `2`, Cypress will retry tests up to 2 additional times (for a total of 3 attempts) before potentially being marked as a failed test.
+
+When each test is run again, the following {% url "hooks" writing-and-organizing-tests#Hooks %} will be re-run also:
+
+- `beforeEach`
+- `afterEach`
+
+{% note warning %}
+However, failures in `before` and `after` hooks will not trigger a retry.
+{% endnote %}
+
+**The following is a detailed step-by-step example of how test retries works:**
+
+Assuming we have set test retries to `2` (for a total of 3 attempts), the following is how the tests would run.
+
+1. A test runs for the first time. If the {% fa fa-check-circle green %} test passes, Cypress will move forward with any remaining tests as usual.
+
+2. If the {% fa fa-times red %} test fails, Cypress will tell you that the first attempt failed and will to run the test a second time.
+
+{% img /img/guides/test-retries/attempt-2-start.png %}
+
+3. If the {% fa fa-check-circle green %} test passes after the second attempt, Cypress will continue with any remaining tests.
+
+4. If the {% fa fa-times red %} test fails a second time, Cypress will make the final third attempt to re-run the test.
+
+{% img /img/guides/test-retries/attempt-3-start.png %}
+
+5. If the {% fa fa-times red %} test fails a third time, Cypress will mark the test as failed and then move on to run any remaining tests.
+
+{% img /img/guides/test-retries/attempt-3-fail.png %}
+
+The following is a screen capture of what test retries looks like on the same failed test when run via {% url "`cypress run`" command-line#cypress-run %}.
+
+{% img /img/guides/test-retries/cli-error-message.png %}
+
+During {% url "`cypress open`" command-line#cypress-open %} you will be able to see the number of attempts made in the {% url "Command Log" test-runner#Command-Log %} and expand each attempt for review and debugging if desired.
+
+{% video local /img/guides/test-retries/attempt-expand-collapse-time-travel.mp4 %}
+
+# Configure Test Retries
+
+## Global Configuration
+
+Typically you will want to define different retry attempts for `cypress run` versus `cypress open`. You can configure this in your {% url "configuration file" command-line#cypress-open-config-file-lt-config-file-gt %} (`cypress.json` by default) by passing the `retries` option an object with the following options:
+
+- `runMode` allows you to define the number of test retries when running `cypress run`
+- `openMode` allows you to define the number of test retries when running `cypress open`
+
+```jsx
+{
+  "retries": {
+    // Configure retries for `cypress run`
+    // Default is 0
+    "runMode": 2,
+    // Configure retries for `cypress open`
+    // Default is 0
+    "openMode": 0
+  }
+}
+```
+
+### Configure specific retries for all modes
+
+If you want to change the number of attempts being made for all tests run for both `cypress run` and `cypress open`, you can configure this in your {% url "configuration file" command-line#cypress-open-config-file-lt-config-file-gt %} (`cypress.json` by default) by defining the `retries` property and setting the desired number of retries.
+
+```jsx
+{
+  "retries": 1
+}
+```
+
+## Custom Configurations
+
+### Individual Test(s)
+
+If you want a specific test to run a custom number of retries, you can set this by using the {% url "test's configuration" writing-and-organizing-tests#Test-Configuration %}.
+
+```jsx
+// Customize retries for an individual test
+describe('User sign-up and login', () => {
+  // `it` test block with no custom configuration
+  it('should redirect unauthenticated user to sign-in page', () => {
+    // ...
+  })
+
+  // `it` test block with custom configuration
+  it('allows user to login', {
+    retries: {
+      runMode: 2,
+      openMode: 1
+    }
+  }, () => {
+    // ...
+  })
+})
+```
+
+### Test Suite(s)
+
+If you want a suite of tests to re-run a custom number of times, you can do this by setting the suite's configuration.
+
+```jsx
+// Customizing retries for a suite of tests
+describe('User bank accounts', {
+  retries: {
+    runMode: 2,
+    openMode: 1,
+  }
+}, () => {
+  // Individual tests will be run normally
+  it('allows a user to view their transactions, () => {
+    // ...
+  }
+
+  it('allows a user to edit their transactions, () => {
+    // ...
+  }
+})
+```
+
+You can find more information about custom configurations here: {% url "Test Configuration" configuration#Test-Configuration %}
+
+# Screenshots
+
+When a test retries, Cypress will continue to take screenshots for each failed attempt or {% url "`cy.screenshot()`" screenshot %} and suffix each new screenshot with `(attempt n)`, corresponding to the current test attempt number.
+
+With the following test code, you would see the below screenshot filenames when all 3 attempts fail:
+
+```js
+describe('User Login', () => {
+  it('displays login errors', () => {
+    cy.visit('/')
+    cy.screenshot()
+    // ...
+  })
+})
+```
+
+```js
+// screenshot filename from cy.screenshot() on 1st attempt
+'User Login -- displays login errors.png'
+// screenshot filename on 1st failed attempt
+'User Login -- displays login errors (failed).png'
+// screenshot filename from cy.screenshot() on 2nd attempt
+'User Login -- displays login errors (attempt 2).png'
+// screenshot filename on 2nd failed attempt
+'User Login -- displays login errors (failed) (attempt 2).png'
+// screenshot filename from cy.screenshot() on 3rd attempt
+'User Login -- displays login errors (attempt 3).png'
+// screenshot filename on 3rd failed attempt
+'User Login -- displays login errors (failed) (attempt 3).png'
+```
+
+# Dashboard
+
+If you are using the {% url "Cypress Dashboard" dashboard %}, information related to test retries is not currently shown. Showing this information and other analytics related to test retries is on our {% url "product roadmap" https://cypress-io.productboard.com/roadmap/1238172-product-roadmap %}.
+
+# Tips and Strategies
+
+While test retries are great for helping to avoid false negatives from failing an entire test run, it is not a good replacement for writing good tests. Here are some tips and strategies to keep in mind in order to maximize the effectiveness of your tests with test retries:
+
+- If you are noticing that you need to increase the number of retries, the cause is more likely due to how the tests are written and is worth spending the time to investigate.
+
+# Frequently Asked Questions (FAQs)
+
+## Will retried tests be counted as more than one test recording in my billing?
+
+No. Tests recorded during `cypress run` with the `--record` flag will be counted the same with or without test retries.
+
+We consider each time the `it()` function is called to be a single test for billing purposes. The test retrying will not count as extra test recordings in your billing.
+
+You can always see how many tests you've recorded from your organization's Billing & Usage page within the {% url "Dashboard" https://on.cypress.io/dashboard %}.

source/guides/references/configuration.md

@@ -22,6 +22,7 @@ Option | Default | Description
 `port` | `null` | Port used to host Cypress. Normally this is a randomly generated port
 `reporter` | `spec` | The {% url 'reporter' reporters %} used during `cypress run`
 `reporterOptions` | `null` | The {% url 'reporter options' reporters#Reporter-Options %} used. Supported options depend on the reporter.
+`retries` | `{ "runMode": 0, "openMode": 0 }` | The number of times to retry a failing test. Can be configured to apply to `cypress run` or `cypress open` separately. See {% url "Test Retries" test-retries %} for more information.
 `watchForFileChanges` | `true` | Whether Cypress will watch and restart tests on test file changes
 
 ## Timeouts
@@ -216,19 +217,21 @@ The configuration values passed in will only take effect during the suite or tes
 
 ### Suite configuration
 
-You can configure the size of the viewport height and width within a suite.
+You can configure the number of times to retries a suite of tests if they fail during `cypress run` and `cypress open` separately.
 
 ```js
-describe('page display on medium size screen', {
-  viewportHeight: 1000,
-  viewportWidth: 400
+describe('login', {
+  retries: {
+    runMode: 3,
+    openMode: 2
+  }
 }, () => {
-  it('does not display sidebar', () => {
-    cy.get('#sidebar').should('not.be.visible')
+  it('should redirect unauthenticated user to sign-in page', () => {
+    // ...
   })
 
-  it('shows hamburger menu', () => {
-    cy.get('#header').find('i.menu').should('be.visible')
+  it('allows user to login', () => {
+    // ...
   })
 })
 ```
@@ -395,6 +398,7 @@ Run GC cleanup before every 3rd test during {% url "`cypress run`" command-line#
 IntelliSense is available for Cypress while editing your configuration file. {% url "Learn how to set up Intelligent Code Completion." IDE-integration#Intelligent-Code-Completion %}
 
 {% history %}
+{% url "5.0.0" changelog %} | Added `retries` configuration.
 {% url "5.0.0" changelog %} | Renamed `blacklistHosts` configuration to `blockHosts`.
 {% url "4.1.0" changelog#4-12-0 %} | Added `screenshotOnRunFailure` configuration.
 {% url "4.0.0" changelog#4-0-0 %} | Added `firefoxGcInterval` configuration.

themes/cypress/languages/en.yml

@@ -48,6 +48,7 @@ sidebar:
     screenshots-and-videos: Screenshots and Videos
     guides: Guides
     command-line: Command Line
+    test-retries: Test Retries
     continuous-integration: Continuous Integration
     parallelization: Parallelization
     environment-variables: Environment Variables

themes/cypress/languages/es.yml

@@ -48,6 +48,7 @@ sidebar:
     screenshots-and-videos: Capturas de pantalla y videos
     guides: Guías
     command-line: Línea de comandos
+    test-retries: Test Retries
     continuous-integration: Integración Continua
     parallelization: Paralelización
     environment-variables: Variables de entorno

themes/cypress/languages/ja.yml

@@ -48,6 +48,7 @@ sidebar:
     screenshots-and-videos: スクリーンショットとビデオ
     guides: ガイド
     command-line: コマンドライン
+    test-retries: Test Retries
     continuous-integration: 継続的インテグレーション
     parallelization: Parallelization
     environment-variables: 環境変数

themes/cypress/languages/pt-br.yml

@@ -48,6 +48,7 @@ sidebar:
     screenshots-and-videos: Screenshots e Vídeos
     guides: Guias
     command-line: Linha de Comando
+    test-retries: Test Retries
     continuous-integration: Integração Contínua
     parallelization: Paralelização
     environment-variables: Variáveis de Ambiente

themes/cypress/languages/ru.yml

@@ -48,6 +48,7 @@ sidebar:
     screenshots-and-videos: Скриншоты и видео
     guides: Руководства
     command-line: Командная строка
+    test-retries: Test Retries
     continuous-integration: Непрерывная интеграция
     parallelization: Параллелизация
     environment-variables: Переменные среды

themes/cypress/languages/zh-cn.yml

@@ -50,6 +50,7 @@ sidebar:
     screenshots-and-videos: 截图和视频
     guides: 指南
     command-line: 命令行
+    test-retries: Test Retries
     continuous-integration: 持续集成
     parallelization: 并行化
     environment-variables: 环境变量