Skip to content

Writing tests

Jump to bottom
Leopold Talirz edited this page Jan 16, 2020 · 15 revisions

Setting up your test environment

  1. Install extra dependencies: pip install -e .[testing]
  2. (optional) Set up a test profile (starting with test_) for faster testing:
    • with quicksetup: verdi quicksetup test_profile_1
    • with setup: verdi -p test_profile_1 setup
    • Note: The profile name, the database name, and the repository folder need to start with test_ . This is to avoid accidental data loss (tests modify the database).

Running existing tests

  • Run all tests: pytest
    • By default this will use pgtest to set up a temporary postgres cluster that is used to run the tests.
    • If you have a test profile set up (see above), you can run the tests directly there by setting: 
      export AIIDA_TEST_PROFILE=test_profile_1
export AIIDA_TEST_BACKEND=django
  • Run subset of tests: pytest tests/path/to/test

Transport tests

For transport tests, you need to be able to ssh into your local machine (localhost) without password, i.e.

  • Make sure to have a ssh server running.
  • Configure a ssh key for your user on your machine, and add your public key to the authorized keys of localhost.

Linux (Ubuntu)

  • sudo apt install openssh-server
  • Create an ssh key (if you don't have one already)
  • Add it to ~/.ssh/authorized_keys, e.g. using ssh-copy-id localhost
  • For security reasons, you might want to disallow ssh connections from outside your local machine: Edit /etc/ssh/sshd_config, changing #ListenAddress 0.0.0.0 to ListenAddress 127.0.0.1 (note the missing #).
  • ssh localhost should now work

Testing philosophy

  • Whenever you encounter a bug, add a (failing) test for it. Then fix the bug.
  • Whenever you modify or add a feature, write a test for it! Writing tests before writing the actual implementation helps keeping your API clean.
  • Make unit tests as atomic as possible. A unit test should run in the blink of an eye.
  • Document why you wrote your test - developers will thank you for it once it fails.

Writing a new test

There are three types of tests in AiiDA:

  1. Tests that do not require the database (for example, testing the creation of paths in k-space, the functionality of a transport plugin, ...)
  2. Tests that require the database, but do not require submission of jobs (for example, verifying that node attributes can be correctly queried, that the transitive closure table is correctly generated, ...)
  3. Tests that require the submission of jobs

We now describe how to write each of these type of tests.

1. Tests that do not need the database

  • Create a test_MODULENAME.py in the appropriate subfolder of the tests directory.
  • For each group of tests create a new subclass of unittest.TestCase and then create the tests as methods using the unittest module.

For an example, see tests.common.test_utils.

2. Tests that need the database

These tests use the testing functionality of Django, adapted to run smoothly with AiiDA.

  • Create a test_MODULENAME.py in the appropriate subfolder of the tests directory.
  • For each group of tests create a new subclass of aiida.backends.testbase.AiidaTestCase.

Note:

  • The AiidaTestCase takes care of
    • creating a temporary database
    • populating the database with default data for each test class (in particular, a computer and a user; see the AiidaTestCase.setUpClass() method).
    • cleaning the database when all tests of the class have finished
  • Run these tests only using verdi devel tests - or you risk corrupting your database.

3. Tests that require the submission of jobs

These tests require an external engine to submit the calculations and then check the results at job completion. We use for this a continuous integration server, and the best approach is to write suitable workflows to run simulations and then verify the results at the end.

Reusing test code

Some test modules contain dedicated routines to simplify the creation of new tests.

For example, any method defined in tests.transport.test_all_plugins that is decorated by @run_for_all_plugins will automatically run for all available transport plugins (testing e.g., to copy files, remove folders, etc.) - in addition to the plugin specific tests inside tests.transports.test_ssh and aiida.transports.test_local.

Clone this wiki locally