Extensive, automated unit tests of Leaf queries

[artgoldberg]

In my view, extensive unit tests of Leaf queries would be extremely valuable. For example, they would have caught the failure reported in issue #447 and likely identified the time-out problem reported in issue #429 .

I would like Leaf to support automated, stochastic testing with the following characteristics.

• Tests are generated automatically, not by using the Leaf GUI. Testing conducted by using Leaf itself has some serious limitations: very few tests can be performed, the tests cannot be easily rerun when updates are made to the LeafDB or the Leaf software, and results of the tests cannot be automatically recorded, so future tests are difficult to compare with previous tests.
• Tests would be generated automatically from the query facets specified in the LeafDB. Many query facets can be easily tested. E.g., condition, gender, race, or ethnicity queries could be simply executed. Some facets, such as for age or lab values would require that a reasonable value be added by the testing apparatus.
• Since Leaf supports an infinite set of possible queries, all possible queries cannot be tested. But a stochastic selection of possible queries could be generated. For example, a small percentage of the numerous (about 100K in our system) conditions queries could be stochastically selected and executed. This would have caught the failure reported in issue "Illegal SQL CRUD operation(s) found in SQL" error #447. Logical combinations of queries could also be generated and executed. This approach has been well studied. Extensive work has developed methods for stochastic testing of SQL systems [Slutz 1998, Khalek and Khurshid 2010, and more recent work that cites them]. Another body of work indicates that just combining pairs of facets could provide good coverage while reducing testing costs [Cohen et al. 1996, Lejay 2004, Colbourn et al. 2006].
• Leaf queries return sets of patients (clinical DW persons). Ideally, my proposed testing system would compare the output of a test Leaf query with an independent query executed against the clinical data. But I believe that's not necessary. Instead, testing can simply report success if a query runs successfully, or failure otherwise. This would find many Leaf failures that might otherwise not be detected.
• Tests are executed by as much of the Leaf system as possible. I.e., ideally they should be injected into Leaf's code at the point where the execution of GUI queries begins.
• Since each Leaf installation has a customized configuration in its LeafDB, this testing system should be easy for each installation to use.
• Tests and their results should be automatically stored in a permanent repository, such as a version control system like git, and tracked over time. As suggested above, the results can be success, or failure and the failure's details. Leaf deployments can then cycle through the process of make changes - run tests - review test results and reliably know whether the changes fix existing problems or create new problems.
• Testing should be deterministic for a particular instantiation of the LeafDB. Although this testing must be stochastic, it should be driven by a deterministic random number generator, which can be initialized by a configurable seed. This enables the iterative process in the previous bullet.

References
D. M. Cohen, S. R. Dalal, J. Parelius and G. C. Patton, "The combinatorial design approach to automatic test generation," in IEEE Software, vol. 13, no. 5, pp. 83-88, Sept. 1996, doi: 10.1109/52.536462. https://ieeexplore.ieee.org/document/536462
Lejay, L. V., Shasha, D. E., Palenchar, P. M., Kouranov, A. Y., Cruikshank, A. A., Chou, M. F., & Coruzzi, G. M. (2004). Adaptive combinatorial design to explore large experimental spaces: approach and validation. Systems biology, 1(2), 206-212. https://doi.org/10.1049/sb:20045020
Colbourn, C. J., Martirosyan, S. S., Mullen, G. L., Shasha, D., Sherwood, G. B., & Yucas, J. L. (2006). Products of mixed covering arrays of strength two. Journal of Combinatorial Designs, 14(2), 124-138. https://doi.org/10.1002/jcd.20065
Slutz, D.R., 1998, August. Massive stochastic testing of SQL. In VLDB (Vol. 98, pp. 618-622)
Abdul Khalek, S. and Khurshid, S., 2010, September. Automated SQL query generation for systematic testing of database engines. In Proceedings of the IEEE/ACM international conference on Automated software engineering (pp. 329-332)

Regards
Arthur

[ndobb]

These are really great points @artgoldberg and I appreciate you writing this up. I'm working through the implications here and trying to consider concretely what a solution for this would look like. Perhaps this could be a command-line driven tool of some sort (for example, run as a cron job perhaps after database maintanence if sites choose to do so).

I'm reading through the references as well.

Best,
-nic

[artgoldberg]

Hi @ndobb

You’re welcome and thank you.

A command line tool would be a great place to start. It would be fine to require that the tool be executed on the server that runs the Leaf API and website.

It could use a small library (or language) of Leaf functions that constructs queries from individual concepts, their optional parameters, and logical operators. The architectural point that I don’t know how to design is to connect it into the Leaf call stack as close as possible to the point where Leaf accepts queries in the UI.

With this foundation users with needs could expand the tool with other features, such as:

1. Run it automatically when changes to the Leaf code are checked in or changes to the LeafDB are made.
2. Parameterize it with initial random number seed, number of queries to run, etc.
3. Play with different algorithms to stochastically generate test queries.
4. Instrument the Leaf code to report line and branch coverage obtained by a test run.
5. Save the results of test runs to appropriate repos, such as perhaps coveralls.io.

Regards
Arthur