-
Notifications
You must be signed in to change notification settings - Fork 2.9k
Automated UI Tests
This wiki is to describe the state of test automation in Firefox for iOS.
In master branch, every check in is verified with unit test suite upon merge. It is also enabled by default on Buddybuild when a build is triggered via a PR request. However, UI tests are scheduled to execute on master branch once per day due to its size. It takes about 2 hours to complete on buddybuild.
The tests are listed in XCode under the Test navigator tab. Tests that are not enabled for the selected scheme would be greyed out.
Currently, Fennec_Enterprise_UITests scheme is executed on Buddybuild for UI Testing, and this scheme can be also used locally to execute tests on simulators. This scheme has additional libraries (Namely, KIFTest and EarlGrey) which are required to run tests in UITests package.
If you would like to run UITests on Buddybuild for your PR, please change your PR's scheme to Fennec_Enterprise_UITests scheme. (Click the cogwheel icon next to the title of your PR) But be prepared to wait for next 2 hours.
Because Fennec_Enterprise_UITests scheme requires a separate license, XCode will complain if one tries to run the test on the actual device. One workaround is to enable UITests/XCUITests in Fennec scheme, as it does not require the enterprise provisioning profile in the Client setting. However, to run UITests, one must include additional parameters in the Test section of the scheme.
- On 'Arguments Passed On Launch', add 'FIREFOX_CLEAR_PROFILE' and 'FIREFOX_TEST'
- On 'Environmental Variables', add 'DYLD_INSERT_LIBRARIES' : @executable_path/EarlGrey.framework/EarlGrey
Tests under UITests package are considered as 'grey-box' tests, where tests can inject specific configuration or values into the target app prior to testing. It uses a mix of KIFTest and EarlGrey.
- KIFTest link: https://github.com/kif-framework/KIF
- Earl Grey link: https://github.com/google/EarlGrey
Initially, KIFTest was used, but Earl Grey was used later to cover the areas where KIF failed on timing/element detection.
One downside of UITests is that the app does not reset between tests - the same instance of Firefox iOS is used for the entire duration of the test execution. So each test has their own setup/cleanup method. So far it appears there is no easy way to reset the app between tests, because this would make the running db to lose connection, and cause failures. Any solution to this issue will be welcomed.
One another thing is that because the tests technically execute as unit tests, one cannot pause the execution and manipulate the UI of the app. The UI will be frozen, and it will interact only when the test is running.
Tests under this package are considered complete black-box tests, with no visibility to the innards of the Firefox iOS app state. This is also the test suite that takes most time, with slightly over an hour of execution time.
One downside of XCUITests is that it appears the SimplePageServer (to prop up mock websites) does not start properly when the XCUITest script calls them. Any solution to this issue will be welcomed.
The UI Test automation status is being updated here: http://bit.ly/2sxBw7Z You are welcome to take one from our to-do lists and make contributions.
Please send questions to npark@mozilla.com or ping :njpark in mozilla irc.