Merge pull request #43 from w3c/sideshowbarker/doc-updates-more

Updated docs/usage.rst to make it more helpful.

docs/usage.rst

@@ -28,57 +28,115 @@ environment created as above::
   pip install -e ./
 
 In addition to the dependencies installed by pip, wptrunner requires
-a copy of the web-platform-tests. This can be located anywhere on
-the filesystem, but the easiest option is to put it in a sibling
-directory of the wptrunner checkout called `tests`::
+a copy of the web-platform-tests repository. That can be located
+anywhere on the filesystem, but the easiest option is to put it within
+the wptrunner checkout directory, as a subdirectory named ``tests``::
 
   git clone https://github.com/w3c/web-platform-tests.git tests
 
-It is also necessary to generate the ``MANIFEST.json`` file for the
-web-platform-tests. It is recommended to put this file in a separate
-directory called ``meta``::
+It is also necessary to generate a web-platform-tests ``MANIFEST.json``
+file. It's recommended to put that within the wptrunner
+checkout directory, in a subdirectory named ``meta``::
 
   mkdir meta
-  cd web-platform-tests
+  cd tests
   python tools/scripts/manifest.py ../meta/MANIFEST.json
 
-This file needs to be regenerated every time that the
+The ``MANIFEST.json`` file needs to be regenerated each time the
 web-platform-tests checkout is updated. To aid with the update process
 there is a tool called ``wptupdate``, which is described in
 :ref:`wptupdate-label`.
 
 Running the Tests
 -----------------
 
-A test run is started using the ``wptrunner`` command. By default this
-assumes that tests are in a subdirectory of the current directory
-called ``tests`` and the metadata is in a subdirectory called
-``meta``. These defaults can be changed using either a command line
-flag or a configuration file.
+A test run is started using the ``wptrunner`` command.  The command
+takes multiple options, of which the following are most significant:
 
-To specify the browser product to test against, use the ``--product``
-flag. If no product is specified, the default is ``firefox`` which
-tests Firefox desktop. ``wptrunner --help`` can be used to see a list
-of supported products. Note that this does not take account of the
-products for which the correct dependencies have been installed.
+``--product`` (defaults to `firefox`)
+  The product to test against: `b2g`, `chrome`, `firefox`, or `servo`.
 
-Depending on the product, further arguments may be required. For
-example when testing desktop browsers ``--binary`` is commonly needed
-to specify the path to the browser executable. So a complete command
-line for running tests on firefox desktop might be::
+``--binary`` (required)
+  The path to a binary file for the product (browser) to test against.
 
-  wptrunner --product=firefox --binary=/usr/bin/firefox
+``--metadata`` (required only when not `using default paths`_)
+  The path to a directory containing test metadata. [#]_
 
-It is also possible to run multiple browser instances in parallel to
-speed up the testing process. This is achieved through the
-``--processes=N`` argument e.g. ``--processes=6`` would attempt to run
-6 browser instances in parallel. Note that behaviour in this mode is
-necessarily less deterministic than with ``--processes=1`` (the
-default) so there may be more noise in the test results.
+``--tests`` (required only when not `using default paths`_)
+  The path to a directory containing a web-platform-tests checkout.
 
-Further help can be obtained from::
+``--prefs-root`` (required only when testing a Firefox binary)
+  The path to a directory containing Firefox test-harness preferences. [#]_
 
-  wptrunner --help
+.. [#] The ``--metadata`` path is to a directory that contains:
+
+  * a ``MANIFEST.json`` file (the web-platform-tests documentation has
+    instructions on generating this file)
+  * (optionally) any expectation files (see :ref:`wptupdate-label`)
+
+.. [#] Example ``--prefs-root`` value: ``~/mozilla-central/testing/profiles``.
+
+There are also a variety of other command-line options available; use
+``--help`` to list them.
+
+The following examples show how to start wptrunner with various options.
+
+------------------
+Starting wptrunner
+------------------
+
+To test a Firefox Nightly build in an OS X environment, you might start
+wptrunner using something similar to the following example::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+    --binary=~/mozilla-central/obj-x86_64-apple-darwin14.0.0/dist/Nightly.app/Contents/MacOS/firefox \
+    --prefs-root=~/mozilla-central/testing/profiles
+
+And to test a Chromium build in an OS X environment, you might start
+wptrunner using something similar to the following example::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+    --binary=~/chromium/src/out/Release/Chromium.app/Contents/MacOS/Chromium \
+    --product=chrome
+
+--------------------
+Running test subsets
+--------------------
+
+To restrict a test run just to tests in a particular web-platform-tests
+subdirectory, use ``--include`` with the directory name; for example::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+    --binary=/path/to/firefox --prefs-root=/path/to/testing/profiles \
+    --include=dom
+
+-------------------
+Running in parallel
+-------------------
+
+To speed up the testing process, use the ``--processes`` option to have
+wptrunner run multiple browser instances in parallel. For example, to
+have wptrunner attempt to run tests against with six browser instances
+in parallel, specify ``--processes=6``. But note that behaviour in this
+mode is necessarily less deterministic than with ``--processes=1`` (the
+default), so there may be more noise in the test results.
+
+-------------------
+Using default paths
+-------------------
+
+The (otherwise-required) ``--tests`` and ``--metadata`` command-line
+options/flags be omitted if any configuration file is found that
+contains a section specifying the ``tests`` and ``metadata`` keys.
+
+See the `Configuration File`_ section for more information about
+configuration files, including information about their expected
+locations.
+
+The content of the ``wptrunner.default.ini`` default configuration file
+makes wptrunner look for tests (that is, a web-platform-tests checkout)
+as a subdirectory of the current directory named ``tests``, and for
+metadata files in a subdirectory of the current directory named ``meta``.
 
 Output
 ------