test_system.qmd

# System tests {#sec-tests-system}


```{r}
#| eval: true 
#| echo: false 
#| include: false
source("_common.R")
library(testthat)
library(gt)
```

```{r}
#| label: co_box_tldr
#| echo: false
#| results: asis
#| eval: true
co_box(
  color = "b",
  look = "default", hsize = "1.10", size = "1.05",
  header = "TLDR   ![](images/shinytest2.png){width='50' height='55'}",
  fold = TRUE,
  contents = "
<br>
  
<h5>**System tests**</h5>
  
`shinytest2` streamlines and enhances system testing for Shiny applications, making it easier to ensure your app works flawlessly inside your app-package:

- Use `record_test()` to capture user interactions and generate tests without writing code manually.
  
- Leverage `AppDriver$new()` for scripting and automating tests, making it simpler to test complex user scenarios.
  
- Apply `exportTestValues()` and `test.mode` to access and assess internal app states, offering a deeper testing perspective.
  
**BDD functions**
  
  - Use `describe()` and `it()` to add features and sceanrios in test files.\n   
  
    \`\`\`r
    describe('Feature...', code = {
        it('Scenario...', code = { ... })
    })
    \`\`\`


  "
)
```

---

This chapter covers using `shinytest2` and `testthat` to perform system tests on the features and scenarios in your app-package. 

:::: {.callout-tip collapse='true' appearance='default'}

## [Accessing the code examples]{style='font-weight: bold; font-size: 1.15em;'}

::: {style='font-size: 0.95em; color: #282b2d;'}

I've created the [`shinypak` R package](https://mjfrigaard.github.io/shinypak/) In an effort to make each section accessible and easy to follow:
  
Install `shinypak` using `pak` (or `remotes`):

```{r}
#| code-fold: false 
#| message: false
#| warning: false
#| eval: false
# install.packages('pak')
pak::pak('mjfrigaard/shinypak')
```

Review the chapters in each section:
  
```{r}
#| code-fold: false 
#| message: false
#| warning: false
#| collapse: true
library(shinypak)
list_apps(regex = 'system')
```

Launch an app: 

```{r}
#| code-fold: false 
#| eval: false
launch(app = "14_tests-system")
```

::: 

::::

> "*Failure to allow enough time for system test, in particular, is peculiarly disastrous. Since the delay comes at the end of the schedule, no one is aware of schedule trouble until almost the delivery date. Bad news, late and without warning, is unsettling to customers and to managers.*" - ['The Mythical Man-Month', Frederick P. Brooks Jr.](https://www.goodreads.com/en/book/show/13629)

System (or end-to-end) tests simulate real user interactions in a 'pre-production' environment to verify the whole application (or system) works.[^tests-prod] Approaches to system testing vary, but in general, we'll want to run a system test for each feature in our application before a release. 

If we've been documenting our unit and integration tests with BDD feature and scenario descriptions, the system tests can be used to confirm the functional requirements for the primary execution path (or user experience) before release.

[^tests-prod]: System tests should strive to replicate the production *conditions*, even when/if it's not possible to perfectly replicate the environment.

## Current tests

```{r}
#| label: git_box_13_tests-modules
#| echo: false
#| results: asis
#| eval: true
git_margin_box(
  contents = "launch",
  fig_pw = '70%', 
  branch = "13_tests-modules", 
  repo = 'sap')
```

The current files in our tests folder are below: 

```{bash}
#| eval: false 
#| code-fold: false
tests
├── testthat
│   ├── fixtures
│   │   ├── make-tidy_ggp2_movies.R
│   │   └── tidy_ggp2_movies.rds
│   ├── helper.R
│   ├── test-mod_scatter_display.R
│   ├── test-mod_var_input.R
│   └── test-scatter_plot.R
└── testthat.R

3 directories, 7 files
```

The output from `devtools::test()` is below:

```{r}
#| eval: false 
#| code-fold: false
devtools::test()
```

```{verbatim}
ℹ Testing sap
✔ | F W  S  OK | Context
⠏ |          0 | mod_scatter_display  
Loading required package: shiny
INFO [2024-05-20 04:19:55] [ START display = selected_vars initial values]
⠋ |          1 | mod_scatter_display                                    
INFO [2024-05-20 04:19:55] [ END display = selected_vars initial values]
INFO [2024-05-20 04:19:55] [ START display = scatterplot[['alt']] = 'Plot object']
⠙ |          2 | mod_scatter_display                                            
INFO [2024-05-20 04:19:56] [ END display = scatterplot[['alt']] = 'Plot object']
INFO [2024-05-20 04:19:56] [ START display = inputs() creates ggplot2 object]
INFO [2024-05-20 04:19:56] [ END display = inputs() creates ggplot2 object]
✔ |          3 | mod_scatter_display
⠏ |          0 | mod_var_input                                            
INFO [2024-05-20 04:19:56] [ START var_inputs = initial returned()]
INFO [2024-05-20 04:19:56] [ END var_inputs = initial returned()]
INFO [2024-05-20 04:19:56] [ START var_inputs = updated returned()]
INFO [2024-05-20 04:19:56] [ END var_inputs = updated returned()]
✔ |          2 | mod_var_input
⠏ |          0 | scatter_plot                                               
INFO [2024-05-20 04:19:56] [ START fixture = tidy_ggp2_movies.rds]
INFO [2024-05-20 04:19:56] [ START fixture = tidy_ggp2_movies.rds]
INFO [2024-05-20 04:19:56] [ START data = sap::movies]
⠙ |          2 | scatter_plot                                 
INFO [2024-05-20 04:19:56] [ END data = sap::movies]
✔ |          2 | scatter_plot
══ Results ═══════════════════════════════════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 7 ]

Nice code.
```


## Setting up [`shinytest2`]{style="font-size: 1.05em;"} {#sec-tests-system-shinytest2}

`shinytest2` requires a few steps to get up and running (most notably the [`chromote` package](https://rstudio.github.io/chromote/)), but you'll find excellent documentation on the [package website](https://rstudio.github.io/shinytest2/index.html).[^shinytest2-start] 


[^shinytest2-start]: A great place to start is the [Getting Started](https://rstudio.github.io/shinytest2/articles/shinytest2.html) vignette. 


The `shinytest2::use_shinytest2()` performs the following setup steps: 

-   ✔ Adding `shinytest2::load_app_env()` to `tests/testthat/setup-shinytest2.R`

-   ✔ Adding `*_.new.png` to `.gitignore`

-   ✔ Adding `_\\.new\\.png$` to `.Rbuildignore`

-   ✔ Adding `shinytest2` to `Suggests` field in `DESCRIPTION`

We also get some advice on using `shinytest2` functions in our code: 

```{verbatim}
#| eval: false 
#| code-fold: false
• In your package code, use `rlang::is_installed("shinytest2")` or
  `rlang::check_installed("shinytest2")` to test if shinytest2 is installed
• Then directly refer to functions with `shinytest2::fun()`
```

*After setting up `shinytest2`, be sure you can create a new chromote session like the one below:*

```{r}
#| eval: false 
#| code-fold: false
library(chromote)
b <- ChromoteSession$new()
b$view()
```

:::{#fig-tests_sys_chromote_new}

![Chromium headless browser](images/tests_sys_chromote_new.png){#fig-tests_sys_chromote_new width='80%' align='center'}

A new `Chromote` session
:::

## Record a test {#sec-tests-system-recording-tests}

If we launch the test recorder with `shinytest2::record_test()`, change the inputs in our application, click on **Expect Shiny values** and **Save test and exit**, a test is recorded to a new test file: `tests/testthat/test-shinytest2.R`


![Creating a test with `shinytest2::record_test()`](images/tests_sys_record_test.png){width='100%' align='center'}


The test runs and saves the PNG snapshot and test values to the `tests/testthat/_snaps/` folder: 

``` sh
[ FAIL 0 | WARN 2 | SKIP 0 | PASS 1 ]

Warning (test-shinytest2.R:23:3): {shinytest2} recording: sap-feature-01
Adding new file snapshot: 'tests/testthat/_snaps/sap-feature-01-001_.png'

Warning (test-shinytest2.R:23:3): {shinytest2} recording: sap-feature-01
Adding new file snapshot: 'tests/testthat/_snaps/sap-feature-01-001.json'

[ FAIL 0 | WARN 2 | SKIP 0 | PASS 1 ]
```

```{r}
#| label: co_box_system_tests
#| echo: false
#| results: asis
#| eval: true
co_box(
  color = "r", 
  look = "default", hsize = "1.10", size = "1.05",
  fold = TRUE, 
  header = "loadSupport warning with shinytest2",
  contents = "
After setting up `shinytest2`, the `tests/testthat/setup-shinytest2.R` file contains a call to `shinytest2::load_app_env()`. This runs automatically with `shinytest2` tests and produces a familiar warning: 
  
\`\`\`bash
Warning message:
In shiny::loadSupport(app_dir, renv = renv, globalrenv = globalrenv) :
  Loading R/ subdirectory for Shiny application, but this directory appears
  to contain an R package. Sourcing files in R/ may cause unexpected behavior.
\`\`\`
  
We covered this warning message in the [Launch](launch.qmd) chapter, and it's being addressed in a future release of [`shinytest2`](https://github.com/rstudio/shinytest2/issues/308)
  
  "
)
```

If we view the contents `test-shinytest2.R` we find each action we performed in the test recorder has a corresponding call in the test:

```{r}
#| eval: false 
#| code-fold: false
library(shinytest2)
test_that("{shinytest2} recording: sap-feature-01", {
  app <- AppDriver$new(name = "sap-feature-01", height = 887, width = 1241) # <1>
  app$set_inputs(`vars-y` = "imdb_num_votes") # <2>
  app$set_inputs(`vars-x` = "critics_score") # <3>
  app$set_inputs(`vars-z` = "genre") # <4>
  app$set_inputs(`vars-alpha` = 0.7) # <5>
  app$set_inputs(`vars-size` = 3) # <6>
  app$set_inputs(`vars-plot_title` = "New plot title") # <7>
  app$expect_values() # <8>
})
```
1. Initialize the `AppDriver$new()` with the name of the test and the dimensions of the Chromium browser.  
2. Change the y axis (`vars-y`) to 'IMBD number of votes' (`imdb_num_votes`)   
3. Change the x axis (`vars-x`) to 'Critics Score' (`critics_score`)   
4. Change the color by (`vars-z`) to 'Genre' (`genre`)    
5. Change the point opacity (`vars-alpha`) to '0.7'    
6. Change the point size (`vars-alpha`) to '3'    
7. Change the plot title to (`vars-plot_title`) to 'New plot title' 
8. Export the test values 

We'll use this initial test as a template for writing the steps in our test `Scenario`s. 

## [`shinytest2`]{style="font-size: 1.05em;"} and BDD {#sec-tests-system-shinytest2-bdd}

There are multiple ways to approach your test layout with `testthat`'s `describe()`, `it()` and/or `test_that()` functions. Below is an example with dedicated `Feature` and `Scenario` descriptions, a reference to the feature number in the `it()` (or `test_that()`) call: 

```{r}
#| eval: false 
#| code-fold: false
describe("Feature 1: Scatter plot data visualization dropdowns
           As a film data analyst
           I want to explore variables in the movie review data
           So that I can analyze relationships between movie reivew sources", {
             
describe("Scenario: Change dropdown values for plotting
            Given the movie review application is loaded
            When I choose the variable [ ] for the x-axis
            And I choose the variable [ ] for the y-axis
            And I choose the variable [ ] for the color
            And I choose the size of the points to be [ ]
            And I choose the opacity of the points to be [ ]
            And I enter '[ ]' for the plot title
           
            Then the scatter plot should show [ ] on the x-axis
            And the scatter plot should show [ ] on the y-axis
            And the points on the scatter plot should be colored by [ ]
            And the size of the points on the scatter plot should be [ ]
            And the opacity of the points on the scatter plot should be [ ]
            And the title of the plot should be '[ ]'", {
              
              it("Feature 01", {
                    app <- AppDriver$new(name = "feature-01", 
                                       height = 800, width = 1173)
                  app$set_inputs(`vars-y` = "imdb_num_votes")
                  app$set_inputs(`vars-x` = "critics_score")
                  app$set_inputs(`vars-z` = "genre")
                  app$set_inputs(`vars-alpha` = 0.7)
                  app$set_inputs(`vars-size` = 3)
                  app$set_inputs(`vars-plot_title` = "New plot title")
                  app$expect_values()
              })
      })
})
```

With this approach you can create the test file as soon as you have a `Feature` description (and come back later to fill in the `Scenario`s and tests).

An alternative approach is to use nested `describe()` functions and include each of the `Scenario`'s `Then` steps in the `it()` or `test_that()` call (these are what will actually be tested):

```{r}
#| eval: false 
#| code-fold: false
library(shinytest2)
describe("Feature 1: Scatter plot data visualization dropdowns
           As a film data analyst
           I want to explore variables in the movie review data
           So that I can analyze relationships between movie reivew sources", {
             
  describe("Scenario A: Change dropdown values for plotting
             Given the movie review application is loaded
             When I choose the variable [ ] for the x-axis
             And I choose the variable [ ] for the y-axis
             And I choose the variable [ ] for the color", {
        it("Then the scatter plot should show [ ] on the x-axis
             And the scatter plot should show [ ] on the y-axis
             And the points on the scatter plot should be colored by [ ]", {
              app <- AppDriver$new(name = "feature-01-senario-a",  # <1>
                                     height = 800, width = 1173)
                app$set_inputs(`vars-y` = "imdb_num_votes")
                app$set_inputs(`vars-x` = "critics_score")
                app$set_inputs(`vars-z` = "genre")
                app$expect_values()
       })
     })
             
  describe("Scenario B: Change dropdown values for plotting
              Given the movie review application is loaded
              When I choose the size of the points to be [ ]
              And I choose the opacity of the points to be [ ]
              And I enter '[ ]' for the plot title", {
         it("Then the size of the points on the scatter plot should be [ ]
              And the opacity of the points on the scatter plot should be [ ]
              And the title of the plot should be '[ ]'", {
              app <- AppDriver$new(name = "feature-01-senario-b", # <2>
                                     height = 800, width = 1173)
                app$set_inputs(`vars-alpha` = 0.7)
                app$set_inputs(`vars-size` = 3)
                app$set_inputs(`vars-plot_title` = "New plot title")
                app$expect_values()
        })
      })
})
```
1. Test for scenario A  
2. Test for scenario B

An important note with this approach is the different names for each `AppDriver$new()` (otherwise we'd be overwriting the previous snapshot/values).

### Testing apps in `inst/`

If we want to test a feature for one of the alternative applications in `sap`, we can pass their location to the `app_dir` argument of `AppDriver$new()`. In the test below, the scenario describes changing inputs for x, y, and color, and removing the missing values from the graph: 

```{r}
#| eval: false 
#| code-fold: false
library(shinytest2)
describe(
  "Feature 1: Scatter plot data visualization dropdowns
     As a film data analyst
     I want to explore movie review variables from IMDB (ggplot2movies::movies data)
     So that I can analyze relationships between movie attributes and ratings", {
  describe(
    "Scenario: Change dropdown values for plotting
        Given the movie review application is loaded
        When I choose the variable ['Length'] for the x-axis
        And I choose the variable ['Rating'] for the y-axis
        And I choose the variable ['Genre'] for the color
        And I click the ['Remove missing'] checkbox", code = {
    it("Then the scatter plot should show ['Length'] on the x-axis
        And the scatter plot should show ['Rating'] on the y-axis
        And the points on the scatter plot should be colored by ['Genre']
        And the missing values should be removed from the plot", {
            
    test_logger(start = 'ggp2movies-feat-01', msg = "update x, y, z, missing")
      
    app <- AppDriver$new(app_dir = system.file("dev", package = "sap"), # <1>
                         name = "ggp2movies_app-feature-01", 
                         wait = FALSE, timeout = 30000, # <2>
                         height = 800, width = 1173)
      app$set_inputs(`vars-y` = "length")
      app$set_inputs(`vars-x` = "rating")
      app$set_inputs(`vars-z` = "genre")
      app$set_inputs(`plot-missing` = TRUE)
      app$expect_values()
          
      test_logger(end = 'ggp2movies-feat-01', msg = "update x, y, z, missing")
      
      })
   })
})
```
1. Use `system.file()` to access application in `inst/dev`  
2. Adjust the `wait` and `timeout` so the test will run  


Not that I've changed the `wait` and `timeout` arguments in `AppDriver$new()` because this tests takes over 10 seconds to complete (which I can see with my `test_logger()` output): 

``` sh
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ]
INFO [2024-05-21 12:39:03] [ START ggp2movies-feat-01 = update x, y, z, missing]
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ]
INFO [2024-05-21 12:40:11] [ END ggp2movies-feat-01 = update x, y, z, missing]
```

When I confirm this in the output `png` file, I can see the x, y, and color values have been changed (and the missing values have been removed).

![File saved in `tests/testthat/_snaps/ggp2movies_app-feature-01-001_.png'`](images/ggp2movies_app-feature-01-001_.png){width='100%' align='center'}


## The [`test.mode`]{style="font-size: 1.05em;"} option  {#sec-tests-system-test-mode}

If you recall, we've included an argument in both of our standalone app functions to allow for `options` to be passed to `shinyApp()`. 

```{r}
#| eval: false 
#| code-fold: false
launch_app(options = list())
```

If we're testing our application, we can include the `test.mode = TRUE` option, which will return any values passed to `exportTestValues()`:

```{r}
#| eval: false 
#| code-fold: false
launch_app(options = list(test.mode = TRUE), run = 'p')
```

We can also include this in our `.Rprofile` as:[^cover-profile]

[^cover-profile]: We covered the `.Rprofile` in @sec-dev-rprofile.

```{r}
#| eval: false 
#| code-fold: false
options(shiny.testmode = TRUE)
```


To export values, place the name of exported reactive values in curly brackets (`{}`). Below is an example using the `inputs()` reactive object in the `mod_scatter_display_server()`:

```{r}
#| eval: false 
#| code-fold: false
exportTestValues(
   x = { inputs()$x },
   y = { inputs()$y },
   z = { inputs()$z },
   alpha = { inputs()$alpha },
   size = { inputs()$size },
   title = { inputs()$plot_title }
  )
```

In our test, we can create the `AppDriver$new()` object, extract the values with `get_values()`, then write tests against any of the exported values:

```{r}
#| eval: false 
#| code-fold: false
app <- AppDriver$new(name = "test-values",
                     height = 800, width = 1173,
                     wait = FALSE, timeout = 300000)

test_values <- app$get_values()

test_values[['export']]
```

``` sh
$`plot-alpha`
[1] 0.5

$`plot-size`
[1] 2

$`plot-title`
[1] ""

$`plot-x`
[1] "imdb_rating"

$`plot-y`
[1] "audience_score"

$`plot-z`
[1] "mpaa_rating"
```

## Test [`_snaps/`]{style="font-size: 1.05em;"}

After writing our system tests and running `devtools::test()`, the `tests/testthat/_snaps/` folder contains the follow folders and files:

``` sh
tests/testthat/_snaps/
├── app-feature-01
│   ├── feature-01-senario-a-001.json
│   ├── feature-01-senario-a-001_.png
│   ├── feature-01-senario-b-001.json
│   └── feature-01-senario-b-001_.png
├── ggp2_app-feature-01
│   ├── ggp2launch_app-feature-01-001.json
│   └── ggp2launch_app-feature-01-001_.png
└── shinytest2
    ├── sap-feature-01-001.json
    └── sap-feature-01-001_.png

4 directories, 8 files
```

These outputs correspond to the three system test files in the `tests/testthat/`:

``` sh
tests/testthat
├── fixtures/
│   ├── make-tidy_ggp2_movies.R
│   └── tidy_ggp2_movies.rds
├── helper.R
├── setup-shinytest2.R
├── test-app-feature-01.R
├── test-ggp2_app-feature-01.R
├── test-mod_scatter_display.R
├── test-mod_var_input.R
├── test-scatter_plot.R
└── test-shinytest2.R

6 directories, 18 files
```

```{r}
#| label: co_box_snaps
#| echo: false
#| results: asis
#| eval: true
co_box(
  color = "g", 
  look = "default", hsize = "1.25", size = "1.05",
  header = "The `_snaps` folder",
  contents = "
I've removed the `tests/testthat/_snaps/` folder from the GitHub repo, so users can run the tests and generate the snapshots in their local environment.
")
```

 

```{r}
#| label: git_box_12e_tests-system
#| echo: false
#| results: asis
#| eval: true
git_margin_box(
  contents = "launch",
  fig_pw = '75%', 
  branch = "14_tests-system", 
  repo = 'sap')
```

## Recap {.unnumbered}

Behavior-driven development (or behavior-driven testing) fills the gulf between non-technical stakeholders and developers by encouraging natural, descriptive language (often complete sentences) to define and communicate the application requirements.

Capturing the application's desired behaviors in `Feature`s (*As a *, *I want*, *So that*) and `Scenario`s (*Given*, *When*, *Then*) provides a testing script that's clear and easy to follow. Using the BDD format also makes system tests easier to update if the features and scenarios change. 

```{r}
#| label: co_box_app_recap
#| echo: false
#| results: asis
#| eval: true
co_box(
  color = "g",
  header = "Recap &emsp; ![](images/shinytest2.png){width='8%'}",
  fold = FALSE,
  contents = "
  
<h5>**System tests**</h5>
  
- `record_test()` simplifies the creation of system tests by recording interactions with your Shiny app. 
  - Using `record_test()` accelerates test creation and ensures our tests accurately reflect user behavior, making it easier to catch issues that could affect user experience.
  
- Use `AppDriver$new()` to manually create tests for more control and customization. 
  - Scripting tests with `shinytest2` allows for detailed user interactions and testing specific functionalities in isolation or under unique conditions, offering a granular approach to system testing.
  
- Use `exportTestValues()` in tandem with `test.mode` to expose and verify the internal state of a Shiny app during tests. 
  - This technique is crucial for testing the logic and data behind your app's UI, ensuring that the app looks right and operates correctly under various scenarios. 
  
**BDD functions**
  
  - Using `testthat`'s BDD functions makes system tests easier to update if the features and scenarios change.
  
    - Capturing the application’s desired behaviors in `Features` (_As a_ , _I want_, _So that_) and `Scenarios` (_Given_, _When_, _Then_) provides a testing script that’s clear and easy to follow.
  
  - use `describe()` and `it()` to add features and sceanrios in test files.\n   
  
    \`\`\`r
    describe('Feature...', code = {
        it('Scenario...', code = { ... })
    })
    \`\`\`

"
)
```


```{r}
#| label: git_contrib_box
#| echo: false
#| results: asis
#| eval: true
git_contrib_box()
```