TestSuite

Overview of the CellBlender Test Suite:

The CellBlender Test Suite consists of a somewhat simple add-on designed to help developers quickly run through a series of CellBlender tests. As of this writing, the test suite contains 22 tests and it takes less than 2 minutes to run through all of them.

The test suite works by invoking the various Blender/CellBlender operators and setting various Blender/CellBlender properties. Eventually, each test generates an MDL file which is checked against a "known good" MDL file for that test. If they match exactly (comparison of SHA1 hash), then the test passes and a green check mark appears with the word "Pass". If they don't match exactly, Blender exits immediately ... so there is little chance of not noticing a failure!!

The test suite starts each test by completely reinitializing the CellBlender add-on and then invoking the operators and setting the properties needed for the test. That means that CellBlender should NOT typically be enabled when running the test suite. The test suite will do that automatically for each test.

Resetting after each test is done by reloading the startup file. In other words, the startup file should contain the desired pre-test configuration of Blender. That will be the configuration of windows, panels, etc that you want to have for the start of each test. All objects in the scene will be deleted by each test anyway, so it's suggested that all objects (cube, camera, lamp...) be removed before saving the startup file to eliminate any confusion. The CellBlender Test Suite panel has convenient buttons for "Load Startup" and "Save Startup" to make this easier.

The primary goal of the CellBlender Test Suite is to make quick checks of CellBlender ... not MCell. The focus is on ensuring that a known sequence of CellBlender operations will produce the proper set of MDL files. For that reason, it is typically not necessary to run MCell to verify the passing of each test. However, it's sometimes helpful to run MCell anyway to observe the results. For that purpose, the CellBlender Test Suite contains a "Run MCell" check box. If that box is checked, then MCell will be launched for each test, and the test suite will wait for the MCell run to complete (Note: at this time the test suite waits a fixed amount of time for MCell to complete since it doesn't contain code to check for completion). If a series of tests are to be run, it's a good idea to set the status of the "Run MCell" box to the desired value and save it in the startup file so it will be set properly after the "Load Startup" done before each test.

Running a series of Tests:

Start by preparing to run the CellBlender Test Suite as described below (that's the hardest part). Then open the panel for the tests that you want to run ("Simple Geometry Tests" for example). Since you'll be reloading the startup file after each test, and since you don't want to have to reopen the test panel each time, simply save the startup file with the panel open. Then when you reload the startup file, it will still be open. The same is true with the "Run MCell" check box - set it and then click the "Save Startup" button.

Once all that is done, the testing is very easy. Just click each test and wait for the result (question mark turns to a green check mark ... or Blender exits). After each test has passed, click the "Load Startup" button and then run another test. When you've run all the tests in one group, click "Load Startup" one more time, close the panel you've been testing, open another testing panel, and save the startup file again. Then repeat the process of "Load Startup", run a test, observe results, "Load Startup", run a test, observe results, ...

Preparing and Running the Test Suite:

This is a step by step recipe for setting up CellBlender for running a series of tests. Each step contains a series of actions that should be done for that step. Menu/submenu commands are separated by a forward slash, and actions are separated by a comma.

1. Create a directory (folder) for your testing files and start Blender from that location.

2. Start with a fresh startup file ( File/Load Factory Settings, File/Save Startup File, File/User Preferences/Save User Settings, close )

3. Enable the CellBlender Test Suite: ( File/User Preferences, Add-ons, Cell Modeling, CellBlender Test Suite, Save User Settings, close )

4. Properties Panel, Scene tab, collapse all sub-panels except "CellBlender Test Suite"

5. Set Path To MCell Binary, collapse "Setup Panel", click "Save Startup" button

6. Configure windows and other features/panels as desired for testing, click "Save Startup" button again.

7. Open a testing group and click the "Save Startup" button

8. Click each test and wait for the result (question mark turns to a green check mark ... or Blender exits). After each test has passed, click the "Load Startup" button and then run another test. When you've run all the tests in one group, click "Load Startup" one more time, close the panel you've been testing, open another testing panel, and save the startup file again. Then repeat the process of "Load Startup", run a test, observe results, "Load Startup", run a test, observe results, "Load Startup", run a test, observe results, ...

Other Uses for the CellBlender Test Suite:

In addition to running a series of tests, the CellBlender Test Suite can be used for demonstrations and to generate starting models for additional experimentation. For example, the E.coli MinD/MinE system is available as one of the test cases. To use that model as a "baseline", simply run that case (following the steps listed above) with the "Run MCell" button disabled. That will build the entire model and it will be ready to run. Just save the blend file in the location of your choice, and open the normal CellBlender tab to make changes or run.

The CellBlender Test Suite contains many specialized tests designed to test specific items. But it also contains Complete Model Tests including the Lotka Volterra model, the Organelle model, and the E.coli MinD/MinE system. These complete models may serve as starting points for building additional models.

Source Code:

The entire CellBlender Test Suite is currently contained in the single file: "test_suite/cellblender_test_suite.py". This file contains two kinds of code:

1. Testing Infrastructure
2. Actual Test Cases

The purpose of the infrastructure code is to make the individual test cases as clear and easy to write as possible. The infrastructure code is mostly contained in the "CellBlender_Model" class, and each test will typically start by creating a "cb_model" object of type "CellBlender_Model". The cb_model is then customized by adding molecule species, adding release sites, adding reactions, adding model objects, etc. Eventually, the "run_model" method is called which will generate the MDL for the model and run MCell (if the RunMCell button is checked). After the run, the "compare_mdl_with_sha1" will check the results and provide the green check mark (if they match) or exit Blender (if they don't match). Also after the run, a few other operations may be performed (like changing the view distance and playing the animation). Currently, because molecules don't exist until after the run, all molecule display changes must be made after the run and there are some unresolved problems in that process. Also, in order for the compare_mdl_with_sha1 function to work properly, the MDL should be exported as a single unified file. You can export as modular MDL files, but the checking will only be done (currently) on the main file.

Adding a new test case:

The easiest way to add a new test case is to simply copy one that already exists. The test cases are separated from each other by a long line of "hash marks" (#####). Simply duplicate one of those blocks and make the following changes:

1. Change the "group_name" to match the group you want the test to be in. A new name will create a new group.

2. Change the "test_name" to a unique name within that group. This is the name that will appear on the button.

3. Change the "operator_name" to a unique name (generally follow the naming pattern of other tests).

4. Change the class name (follows the word "class") to be consistent with the test name you've chosen.

5. If your test is going to call the "compare_mdl_with_sha1" function, be sure to change the "test_name" argument to match the "test_name" variable you chose in step 2.

6. Make any other changes to the model as needed (molecules, reactions, view distance, etc).

7. If you're using the "compare_mdl_with_sha1" function, run your test to get the "SHA1" of the resulting MDL files. It won't likely match the value you passed to the function, so Blender will exit after printing the SHA1 that it was expecting and the SHA1 that it computed. Copy the value that it computed and put it into the function call (assuming that you're satisfied that it's generating the proper MDL). One handy technique is to comment out the "compare_mdl_with_sha1" function when developing a test. That will allow you to run the test without Blender exiting. When you're convinced that the test is correct and it's generating the correct MDL, then you can uncomment the compare line and run it. Of course it will fail, but it will show you the proper value to copy into the call statement. Once that's done, the test will only pass if CellBlender generates MDL that matches the SHA1 that you've copied into the code. So any changes to CellBlender that break your model will fail to pass the test.

The Future:

The Test Suite code currently requires a "Load Startup" before each test. If this isn't done, then something from the previous test will generally corrupt the subsequent test. This is probably a bug somewhere in CellBlender, and it will hopefully be found at some point in the future. At this time, the focus is on using the Test Suite for the major code refactoring taking place in the development branch, so refinement of the Test Suite is lower on the priority list. Right now, 22 tests can be completed in less than 2 minutes, and that's just what's needed to support the code refactoring effort. However, in the future it would be nice to have the Test Suite run a series of tests without requiring any intervention at all. Another potential application would be to define a CellBlender API for power users who might want to generate models through Python. Blender already offers a rich interface for creating meshes in Python. Adding a convenient capability for defining an MCell model results in a complete solution for those with the skills to use it. Note that this is NOT the same as PyMCell, but it could migrate in that direction.