From 2ff044145c204cdc0b310339c89ac5e1bac4bec3 Mon Sep 17 00:00:00 2001 From: RokuRnD Date: Mon, 5 Mar 2018 14:29:48 +0200 Subject: [PATCH] Added Setup and TearDown functions description --- docs/unit-test-framework.md | 796 +++++++++++++++++++----------------- 1 file changed, 417 insertions(+), 379 deletions(-) diff --git a/docs/unit-test-framework.md b/docs/unit-test-framework.md index 25c7fa0..0644e3e 100644 --- a/docs/unit-test-framework.md +++ b/docs/unit-test-framework.md @@ -1,379 +1,417 @@ -## Overview - -This document describes the architecture and usage of the Roku unit test framework. This framework allows Roku channel developers to instrument their code with predefined or custom test cases in order to unit test their code. - -## System Architecture - -### System diagram - -![System architecture diagram](../images/system-architecture-diagram.png) -**Figure 1:** System architecture diagram - -### Sequence diagram - -![Flow diagram](../images/flow-diagram.png) -**Figure 2:** Flow diagram - -## Framework Components - -### Test Runner -TestRunner is main object in the unit test framework. This object has 6 properties: -* testsDirectory – path to directory which contains all tests. -* testFilePrefix - string to identify all test files in testsDirectory and subdirectories. -* testSuitePrefix - string identifying all test suites in test files. -* testSuiteName – string identifying specific test suite. -* testCaseName - string identifying specific test case. -* logger – object which formats and outputs logs. - -TestRunner searches for .brs files with the testFilePrefix in testsDirectory and subdirectories. When the list of files is complete, TestRunner parses these files and extracts all test suites identified by the prefix testSuitePrefix. Every test suite has a list of test cases. TestRunner compiles a single list of all test cases and then runs them one at a time. Set testSuiteName to specify exact test suite to run or testCaseName to specify exact test case. - -During a test run TestRunner collects the statistic into a statistic object. After the test run TestRunner passes a statistics object to a Logger instance which outputs a results to a console accessible via telnet port 8085. The verbosity level can be set by calling the SetVerbosity method of the Logger instance. - -### BaseTestSuite -A collection of one or more test cases is called a test suite. Developers can implement their own test suite types which must be derived from BaseTestSuite. BaseTestSuite implements a set of assertion methods. All these methods have one common argument “msg”. This is an error message returned if the assertion fails. - -**Table 1. Assertions list** - -| Method Name | Arguments | Description | -| ------------- |:-------------:| -----:| -| Fail | msg (string) An error message. Default value: "Error". | Fail immediately, with the given message. | -| AssertFalse | expr (dynamic) An expression to evaluate. | Fail the test if the expression is true. | -| AssertTrue | expr (dynamic) An expression to evaluate. | Fail the test if the expression is false. | -| AssertEqual | first (dynamic) A first object to compare. second (dynamic) A second | Fail if the two objects are unequal as determined by the '<>' operator. | -| AssertNotEqual | first (dynamic) A first object to compare. second (dynamic) A second object to compare. | Fail if the two objects are equal as determined by the '=' operator. | -| AssertInvalid | value (dynamic) A value to check. | Fail if the value is not invalid. | -| AssertNotInvalid | value (dynamic) A value to check. | Fail if the value is invalid. | -| AssertAAHasKey | array (dynamic) A target array. key (string) A key name. | Fail if the array doesn't contain the specified key. | -| AssertAANotHasKey | array (dynamic) A target array. key (string) A key name. | Fail if the array contains the specified key. | -| AssertAAHasKeys | array (dynamic) A target associative array. keys (object) A key names array. | Fail if the array doesn't contain the specified set of keys. | -| AssertAANotHasKeys | array (dynamic) A target associative array. keys (object) A key names array. | Fail if the array contains the specified set of keys. | -| AssertArrayContains | array (dynamic) A target array. value (dynamic) A value to check. key (object) A key name for associative array. | Fail if the array doesn't contain the specified item. | -| AssertArrayNotContains | array (dynamic) A target array. value (dynamic) A value to check. key (object) A key name for associative array. | Fail if the array contains the specified item. | -| AssertArrayContainsSubset | array (dynamic) A target array. subset (dynamic) An items array to check. | Fail if the array doesn't contains the specified subset of items. | -| AssertArrayNotContainsSubset | array (dynamic) A target array. subset (dynamic) A items array to check. | Fail if the array contains the specified subset of items. | -| AssertArrayCount | array (dynamic) A target array. count (integer) An expected array items count. | Fail if the array item count is not equal to the specified value. | -| AssertArrayNotCount | array (dynamic) A target array. count (integer) An expected array items count. | Fail if the array item count is equal to the specified value. | -| AssertEmpty | item (dynamic) An array or string to check. | Fail if the item is not empty. | -| AssertNotEmpty | item (dynamic) An array or string to check. | Fail if the item is empty. | -| AssertArrayContainsOnly | array (dynamic) A target array. typeStr (string) An item’s type name. | Fail if the array contains items of types other than the specified type. | - -### Item Generator - -ItemGenerator is an object that generates random items according to a specified scheme. ItemGenerator is used to create random test data for use as function parameters, etc. A scheme could be one of simple types (integer, string or float), array or associative array. - -For simple types, ItemGenerator generates a random value of this type. - -For arrays, the scheme specifies that types of each of the individual array items. - -For example: - -**scheme:** [“int”, “string”, “string”, “boolean”] - -**result:** [5, “ghn56f”, “7sb2td”, true] - -An associative array scheme has the following structure: - -``` -{ - propertyName1: "propertyType1" - propertyName2: "propertyType2" - ... - propertyNameN: "propertyTypeN" -} -``` - -ItemGenerator will return an associative array object with the specified property names and random values of the specified property types. - -For example: - -**scheme:** - -``` -{ - id: "integer" - name: "string" - address: {city: “string”, street: “string”} - active: “boolean” -} -``` - -**result:** - -``` -{ - id: 75 - name: "ht7d9nt5zp" - address: {city : “9gbst5mpw7”, street : “nfdt7ns5p2”} - active: false -} -``` - -### Logger - -The Logger object is used to output test results. It takes information from a TestRunner, formats it according to selected verbosity level and outputs to the console. TestRunner calls printStatistic function to print logs. This method has one argument which is a statistic object. The printStatistic function can be overwritten to generate custom log output. - -## Usage - -###General - -To use the unit test framework in your channel, place the framework files in the directory pkg:/source /testFramework/. - -To run your unit tests, follow these instruction: - -1) Create new folder “tests” under source directory of your project: “pkg:/source/tests”. - * This will be the root folder for all your unit tests. In this folder you can also create subfolders for every test suite collection. - -2) Create test suite files in “tests” folder or subfolders. - -Create new .brs files for each test suite. The default prefix for test suite files is “Test\__”. You can use any prefix you want just don’t forget to specify it in the next step. In this new file define a function TestSuite__Main(). This function will return the test suite object. Create a new test suite object from BaseTestSuite: - -_this = BaseTestSuite()_ - -then set its name - -_this.Name = "MainTestSuite"_ - -Then add test cases: - -this.addTest("CheckDataCount",TestCase\__Main_CheckDataCount). -TestCase\__Main_CheckDataCount is a test function. -Function TestCase__Main_CheckDataCount() as String -return m.assertArrayCount(m.mainData, 15) -End Function - -3) Run all your tests. - -To run your tests: -* create instance of TestRunner object (Runner = TestRunner()) and call it’s method Run (Runner.Run()). Embrace this code with such if statement – “args.RunTests = "true" and type(TestRunner) = "Function"”. -* If needed set TestRunner’s properties by calling its setter methods before calling Run method. -* Deploy your channel to the device -* POST the following command to the device via ECP: -http://{Roku_box_IP_address}:8060/launch/dev?RunTests=true -* View the test results by opening a telnet console on the device on port 8085: -telnet {Roku box IP address} 8085. - -To set verbosity level call Runner.logger.SetVerbosity(level). There are four verbosity levels in this framework: -* “0” – basic level -* “1” – normal level -* “2” – detailed only for failed tests, normal for others -* “3” – detailed level. - -You can overwrite printStatistic if you like, such as - -Runner.logger.printStatistic = customPrintFunction - -With basic verbosity level the framework outputs total number of tests, number of passed tests, number of failed tests, number of tests that caused a crash and time spent for running the tests: - -![Basic verbosity level](../images/Basic-verbosity-level.png) - -**Figure 3:** Basic verbosity level - -Normal level includes the list of all tests with their names and results: - -![Normal verbosity level](../images/normal-verbosity-level.png) - -**Figure 4:** Normal verbosity level - - -Detailed level shows verbose statistics for every test suite and any error messages for failed tests: - -![Detailed verbosity level](../images/detailed-verbosity-level.png) - -**Figure 5:** Detailed verbosity level - -###RSG - -If you want to test RSG nodes you should: - -1) Create new folder "tests" under components directory of your project: "pkg:/components/tests". - * This will be the root folder for RSG unit tests. In this folder you can also create subfolders for every test suite collection. - -2) Create test suite files in "tests" folder or subfolders. - -Create new .xml file for each node you want to test. This .xml file should be a node and it must extend the node you want to test. The default prefix for test nodes is "Test\__". You can use any prefix you want just don’t forget to specify it. File name must match node name. For each node you may create as many test suites as you wish. Details on how to create test suites you can find in the previous section. The only difference is that every test suite should have following prefix: TEST\_FILE\_PREFIX + NODE\_NAME + "\__" + TEST\_SUITE\_NAME, where default TEST\_FILE\_PREFIX is "Test\__", NODE\_NAME may be "MyNode" and TEST\_SUITE\_NAME can be "TestSuite1". So the name of a test suite file may look like this: "Test\__MyNode\__TestSuite1". In every test node (.xml file) you should: -* Add interface function for running tests. This function will be used by framework and must not be addressed in any other way: - - - - - -* Import unit test framework: - -