civicrm · seancolsen · Nov 2, 2017 · Oct 8, 2017 · Oct 8, 2017 · Oct 8, 2017
diff --git a/docs/api/interfaces.md b/docs/api/interfaces.md
@@ -63,7 +63,7 @@ These requests are different to doing a Chained API call as they are making simu
 
 #### Tests
 
-Qunit tests for `CRM.api3` can be found in [/tests/qunit/crm-api3](https://github.com/civicrm/civicrm-core/tree/master/tests/qunit/crm-api3).
+[QUnit](/testing/qunit.md) tests for `CRM.api3` can be found in [/tests/qunit/crm-api3](https://github.com/civicrm/civicrm-core/tree/master/tests/qunit/crm-api3).
 
 You can run the tests within a web browser by visiting `/civicrm/dev/qunit/civicrm/crm-api3` within a CiviCRM [development installation](/tools/civibuild.md).
 

diff --git a/docs/basics/planning.md b/docs/basics/planning.md
@@ -44,7 +44,7 @@ Make sure to check [the contribution guidelines](/core/contributing.md) for more
 
 **Install the [buildkit](https://github.com/civicrm/civicrm-buildkit)**, ideally as a [vagrant virtual-machine](https://github.com/civicrm/civicrm-buildkit-vagrant) or using one of available `docker` images ([progressivetech](https://github.com/progressivetech/docker-civicrm-buildkit) or [EricBSchulz](https://github.com/ErichBSchulz/dcbk)). The buildkit is not an absolute requirement but it is definitely the fastest path to a good development experience!
 
-**From the outset, [automate testing](/testing/setup.md)**. In the current climate of rapid evolution of not just CiviCRM, but also it's myriad of dependencies, automated testing of PHP code with `phpunit` and javascript with tools like `karma` and `jasmine` is essential. Start all your work by considering how you will provide automated testing for it. Starting with the buildkit will make this much simpler for you to set up. Getting started with unit-testing may seem daunting and onerous when you start, but you will soon come to love the freedom it gives you. If you are unsure how to proceed with testing ask the [community](/basics/community.md).
+**From the outset, [automate testing](/testing/index.md)**. In the current climate of rapid evolution of not just CiviCRM, but also it's myriad of dependencies, automated testing of PHP code with `phpunit` and javascript with tools like `karma` and `jasmine` is essential. Start all your work by considering how you will provide automated testing for it. Starting with the buildkit will make this much simpler for you to set up. Getting started with unit-testing may seem daunting and onerous when you start, but you will soon come to love the freedom it gives you. If you are unsure how to proceed with testing ask the [community](/basics/community.md).
 
 **Create a native [extension](/extensions/index.md)**. If you have new functionality to add to CiviCRM, it probably belongs in an extension.  "Native" extensions will install into all CiviCRM sites regardless of the  underlying CMS used (Drupal, WordPress, Joomla or Backdrop), making it easy to share your extension with the CiviCRM community.
 

diff --git a/docs/core/contributing.md b/docs/core/contributing.md
@@ -21,7 +21,7 @@ Your changes might require documentation updates. Read about [when to document](
 
 Before you write any code you should write tests. You can use your plain-language description of how things should work to create automated tests for the feature.
 
-CiviCRM comes with a variety of [testing tools](/testing/setup.md) that help ensure that changes don't break existing functionality. Since CiviCRM [doesn't release code with failing tests](/tools/jenkins.md), your bug or improvement must not be covered in the existing tests.  Maybe there are incomplete tests, maybe the tests aren't valid measures of the functionality, or maybe your feature lacks test coverage.  Either way, you will need to write them to make sure your work doesn't get undermined by future changes.
+CiviCRM comes with a variety of [testing tools](/testing/index.md) that help ensure that changes don't break existing functionality. Since CiviCRM [doesn't release code with failing tests](/tools/jenkins.md), your bug or improvement must not be covered in the existing tests.  Maybe there are incomplete tests, maybe the tests aren't valid measures of the functionality, or maybe your feature lacks test coverage.  Either way, you will need to write them to make sure your work doesn't get undermined by future changes.
 
 Use your documentation to identify tests that can be run, and then write them.  If you are adding functionality, you may not have the code that the test will call, but you can write your tests as if all the pages and functions exist, defining them later.
 

diff --git a/docs/framework/api-architecture.md b/docs/framework/api-architecture.md
@@ -16,7 +16,7 @@ are supported for accessing the api functions
 
 -   There are a number of (fairly new) helper functions you should use
     in the test - please check [Writing a PHPUnit testcase
-    HOWTO](/testing/setup.md)
+    HOWTO](/testing/index.md)
 -   API functions should match the relevant BAO names & have the
     functions 'get', 'create' and delete'
 -   All functions should receive $params as an array(not a reference)

diff --git a/docs/framework/backbone.md b/docs/framework/backbone.md
@@ -297,7 +297,7 @@ emails.save(...);
 
 ## Unit Tests
 
-The CiviCRM Backbone plugins are tested with qUnit. To run the
+The CiviCRM Backbone plugins are tested with [QUnit](/testing/qunit.md). To run the
 unit-tests, use a web-browser to connect to a CiviCRM installation
 (`http://local.example.com`) and request the following:
 

diff --git a/docs/standards/javascript.md b/docs/standards/javascript.md
@@ -155,17 +155,14 @@ When your script file is being added to the page by `CRM_Core_Resources` it will
 
 CiviCRM ships with a number of UI widgets and plugins to create standardized "look and feel" More information can be found in [UI Elements Reference](/framework/ui.md).
 
-## Test Tools
+## Automated testing
 
-CiviCRM has uses a couple of main tools to test Javascript Code when a pull request is made against code. This happens along side the [code style checks](/standards/javascript.md#coding-standards) which are as mentioned above against JsHint
+CiviCRM's testing regimen includes:
 
-### Qunit
-
-CiviCRM has a small set up of Qunit which is an automated testing system. It's generally inspired by the xUnit family (JUnit, PHPUnit, etc) and allows you to test low- and mid-level component functionality. It's most suitable for testing pure Javascript components. For a general introduction, see the [README](https://github.com/civicrm/civicrm-core/blob/master/tests/qunit/README.txt) and [example test](https://github.com/civicrm/civicrm-core/tree/master/tests/qunit/example) that ship with CiviCRM. It is currently only run manually through a web broweser.
-
-### Karma
-
-For testing CiviCRM's Angular implementation in Core, CiviCRM has intergrated some Karma tests into the standard test suite that is run against each Pull request. More information on karma can be found in the [Javascript testing](/testing/javascript.md) section of this guide
+ * (Linting) [JSHint](/standards/javascript.md#coding-standards)
+ * (Unit testing) [Karma and Jasmine](/testing/karma.md)
+ <!-- * (End-to-end testing, for AngularJS) [Protractor and Jasmine](/testing/protractor.md) -->
+ * (Deprecated; end-to-end testing) [QUnit](/testing/qunit.md)
 
 ## Javascript in Markup
 

diff --git a/docs/testing/codeception.md b/docs/testing/codeception.md
@@ -0,0 +1,6 @@
+!!! tip "Setup"
+
+    The test suites require a small amount of [setup](/testing/index.md#setup).  If your system was created via [buildkit](/tools/buildkit.md) and
+    [civibuild](/tools/civibuild.md), then it was handled automatically.
+
+TODO
diff --git a/docs/testing/index.md b/docs/testing/index.md
@@ -0,0 +1,183 @@
+## Overview
+
+The CiviCRM suite spans the full stack -- it includes low-level helpers and user-facing applications; it includes
+server-side PHP and client-side Javascript; it touches multiple business areas (such as contributions, mailings, and
+cases); it includes a common core project and an ecosystem of add-ons.
+
+Depending on the scope of the test and the relevant language, one chooses among these tools:
+
+<table>
+  <thead>
+    <tr>
+      <th></th>
+      <th>Description</th>
+      <th>PHP</th>
+      <th>Javascript</th>
+      <th>Other</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <th>
+        Unit Test
+      </th>
+      <td>
+        Basic testing of a single function or class<br/>
+        <b>Trade-off</b>: <em>Fast but minimal/synthetic environment</em><br/>
+        <b>Good for testing</b>: <em>Helpers, libraries</em>
+      </td>
+      <td>
+        <a href="phpunit">PHPUnit</a>
+      </td>
+      <td>
+        <a href="karma">Karma</a>
+      </td>
+      <td>
+      </td>
+    </tr>
+    <tr>
+      <th>
+        Headless Test
+        </th>
+      <td>
+        Mid-level testing with headless database<br/>
+        <b>Trade-off</b>: <em>Mid-level speed and realism</em><br/>
+        <b>Good for testing</b>: <em>Data management APIs, portable backend services</em><br/>
+      </td>
+      <td>
+        <a href="phpunit">PHPUnit</a>
+      </td>
+      <td>
+      </td>
+      <td>
+      </td>
+    </tr>
+    <tr>
+      <th>
+        End-to-End Test (E2E)
+      </th>
+      <td>
+        Full-stack testing with CMS, HTTPd, etc<br/>
+        <b>Trade-off</b>: <em>Slow but realistic environment</em><br/>
+        <b>Good for testing</b>: <em>Screens, pageflows, integrations</em><br/>
+      </td>
+      <td>
+        <a href="phpunit">PHPUnit</a><br/>
+        <!-- <a href="codeception">Codeception</a><br/> -->
+        <s><a href="selenium">Selenium</a></s>
+      </td>
+      <td>
+        <!-- <a href="protractor">Protractor</a><br/> -->
+        <s><a href="qunit">QUnit</a></s>
+      </td>
+      <td>
+        <a href="upgrades">Upgrades</a><br/>
+        <a href="manual">Manual</a><br/>
+      </td>
+    </tr>
+  </tbody>
+</table>
+
+!!! caution "Be mindful of test types"
+
+    Each style of testing (unit, headless, E2E) has distinctive practices.  For example, headless tests periodically reset the entire database, but
+    E2E tests don't. These concepts are discussed in more detail below.
+
+## Setup
+
+Many test suites require information about your local development environment.  For example, headless tests may require credentials for an extra
+MySQL database, and end-to-end tests may require credentials for a CMS.
+
+The test tools obtain this information via [cv](https://github.com/civicrm/cv).  If your build was created by [buildkit](/tools/buildkit.md) and
+[civibuild](/tools/civibuild.md), then `cv` can fetch *all* the information automatically. Other builds may require some manual configuration.
+
+To inspect the configuration, run:
+
+```bash
+$ cd /path/to/civicrm
+$ cv vars:show
+```
+
+This should display a number of properties, including:
+
+* Credentials for an empty test database (`TEST_DB_DSN`)
+* Credentials for an administrative CMS user (`ADMIN_USER`, `ADMIN_PASS`, `ADMIN_EMAIL`)
+* Credentials for a non-administrative CMS user (`DEMO_USER`, `DEMO_PASS`, `DEMO_EMAIL`)
+
+If these are missing or blank, then you need to fill them in. Initialize and edit the configuration file:
+
+```bash
+$ cv vars:fill
+$ vi ~/.cv.json
+```
+
+!!! tip "Tip: Database snapshots"
+
+    Many tests interact with the database. In case they mess up the database, you should retain a snapshot of your baseline DB.
+
+    If you used `civibuild`, it automatically retained a DB snapshot when you last (re)installed the site.
+    See [civibuild restore](/tools/civibuild.md#rebuild) for more information.
+
+!!! tip "Tip: Multi-user systems"
+
+    By default, `cv` uses the configuration file `~/.cv.json`. However, if this build is accessed by many user accounts, then
+    you can use a shared configuration file. Run `export CV_CONFIG=/path/to/shared/file.json` and then call `cv vars:fill`.
+
+## Architecture
+
+Most automated tests require access to some mix of **resources** -- such as source-code files, databases, or URLs.  To access these resources, test
+frameworks should build on the [cv](https://github.com/civicrm/cv) command.
+
+There are three types of tests. Each handles these resources differently.
+
+### Minimal unit test {:#unit}
+
+A minimal **unit test** focuses on testing a discrete technical artifact, such as a function, class, or file.  Minimal unit tests are **loosely**
+**coupled**, and they generally shouldn't require an external service (such as a database or web-server).  The narrow scope and loose-coupling makes
+these **fast** -- so you can execute a large suite of tests in a short time.  However, it also makes them **less representative**.  This type of
+testing is ideal for helpers, utilities, and libraries which have innately low coupling.  They're also useful if you need to test a large range of
+possible inputs/permutations.
+
+!!! tip "Unit tests and safety"
+
+    In a minimal unit-test, running the test should have no side-effects -- ie it should not change data-files or database tables.
+
+!!! tip "Unit tests and bootstrap"
+
+    In a framework like `phpunit` or `karma`, a minimal unit test might call `cv php:boot --level=classloader`
+    or `cv path -d [civicrm.root]` to gain access to Civi's source code.
+
+### Headless test {:#headless}
+
+The most common kind of automated test in `civicrm-core` is a **headless test**.  Headless tests use a *nearly complete* CiviCRM environment;
+however, there is no CMS or web-server, and all data is stored on a private, headless database.  This is ideally suited to testing data-management
+APIs (where the DBMS is an important part of the system) and other portable services (where you wouldn't expect the CMS to influence behavior).
+
+!!! tip "Headless tests and safety"
+
+    In a headless unit-test, the test takes ownership over a private, test-only, *headless* copy of CiviCRM.  Headless tests commonly take steps to
+    keep the database at a clean baseline, such as (a) rolling back a SQL transaction or (b) truncating a few data tables or (c) resetting the entire
+    database.
+
+!!! tip "Headless tests and bootstrap"
+
+    In a framework like `phpunit` or `codeception`, a headless test might call `cv php:boot --level=full`. To avoid bootstrapping a full CMS, it
+    would need an environment variable `CIVICRM_UF=UnitTests`.
+
+### End-to-end test {:#e2e}
+
+An **end-to-end test** (E2E) focuses on testing a full stack, including some combination of CRM, CMS, web-server, file-system, database, and web-browser.
+These tests are the most **representative** of real-world usage, because they tie together so many components.  However, this **high coupling** also makes
+them brittle (because a design-change or fault in any part of the system can disrupt the expected flow).  This type of testing is ideal for screens,
+pageflows, and integrations which have innately high coupling.
+
+!!! tip "End-to-end tests and safety"
+
+    In an E2E test, the test runs against an active, local CiviCRM installation.  By default, each test makes **persistent** changes, which can
+    create side-effects for other tests. Be conscientious about (a) checking pre-conditions (in case other tests have side-effects) and
+    (b) cleaning up to prevent side-effects (that might break other tests).
+
+!!! tip "End-to-end tests and bootstrap"
+
+    In a framework like `phpunit` or `codeception`, an E2E test might call `cv php:boot --level=full`. However, it would not need to explicitly
+    set `CIVICRM_UF`. Depending on the active CMS, this value is determined automatically.
diff --git a/docs/testing/javascript.md b/docs/testing/javascript.md
diff --git a/docs/testing/karma.md b/docs/testing/karma.md
@@ -0,0 +1,28 @@
+!!! tip "Setup"
+
+    The test suites require a small amount of [setup](/testing/index.md#setup).  If your system was created via [buildkit](/tools/buildkit.md) and
+    [civibuild](/tools/civibuild.md), then it was handled automatically.
+
+[Karma] is a Javascript testing tool which executes [Jasmine] tests on the command-line.
+It was introduced  in Civi v4.6 in tandem with several AngularJS-based UIs.
+
+[Buildkit](/tools/buildkit.md) includes a copy of `karma`. Alternatively,
+you can download it by running `npm install` in the `civicrm` directory.
+
+## Running Karma
+
+If you're actively working on Javascript files or Karma tests, then you can
+start `karma` in a *watch* mode.  Any time you save a change to disk, it
+will automatically re-execute the tests.
+
+```bash
+$ cd /path/to/civicrm
+$ karma start
+```
+
+## Running Karma (Other ways)
+
+You can also run the karma tests as they would be run by [Jenkins](/testing/continuous-integration.md) using [civi-test-run](/tools/civi-test-run.md).
+
+[Karma]: https://karma-runner.github.io/1.0/index.html
+[Jasmine]: https://jasmine.github.io/2.1/introduction.html