Magento Automated Testing

Magento Automated Testing

Magento Automated Testing is a testing infrastructure that allows to run the set of all required Magento Automated Tests against code in Pull Requests. It automatically triggers the tests against the changes on Pull Request creation or new commits pushes into the Pull Request. The running tests are represented both as Pull Request statuses and as Pull Requests checks.

Below there's information about all kinds of tests available currently.

Semantic Version Checker

Semantic Version Checker is a test that compares the changed code against the mainline and detects levels of code changes. It shows a list of changes grouped by the class (level) of changes (MAJOR, MINOR, PATCH) with explanation of the change (Class was removed, [public] Method implementation changed, [private] Property has been removed etc.) The allowed level is PATCH. Please see the Backward Compatibility Development Guide for more information

Functional Tests

Functional Tests are PHPUnit-based tests that check Magento functional behavior from the perspective of a user. They press buttons and links in browser, fill and submit forms, look and check what is displayed in the browser's window like real users.

Sample Data Tests

Sample Data Tests are functional tests that run on Magento with Sample Data included. Currently they include two kinds of tests:

• Sample Data functional tests which check Magento functional behavior with Sample Data installed
• Installation Functional tests which check Magento installation flow with Sample Data installed

Unit Tests

Unit tests are PHPUnit-based (for PHP) and Jasmine-based (for JS) tests which check the behavior of single methods of Magento classes. The purpose is to validate that each unit of Magento performs as designed.

Integration Tests

Integration Tests are PHPUnit-based tests that check the behaviour of group of Magento components. They include three types of tests:

• Standard Integration tests which check Magento behavior on installed instance.
• Setup Integration tests which check Magento behavior during the installation process.
• Integration Integrity tests which check the consistency of installed Magento instance (configuration files, DB schemes etc.)

WebAPI Tests

WebAPI Tests are functional tests that check the behavior of Magento Web API endpoints. They send requests to endpoints of installed Magento instance and and then check the response received and the state of the instance. Currently they include three kinds of tests:

• SOAP Web API tests
• REST Web API tests
• GraphQL WEB API tests

Static Tests

Static Tests are set of different checks that analyze each single file of Magento code base separately. Most of them analyze the whole Magento code base whereas some run only against the change set. They include:

• Static Integration tests (PHPUnit-based) which check the consistency of Magento instance without installation (configuration files, DB schemes etc.)
• Static Legacy Tests which check Magento code doesn't contain any deprecated approaches and patterns.
• Static PHP, LESS, JS, TS Code Style tests which check Magento source code against the best practices of programming style (Code style, Mess detector, Copy/Paste detector)
• Static Sanity Check which tests Magento code against dictionary of forbidden words.
• Static Conflicts Check which tests Magento code doesn't contain duplicate file paths between Magento editions and bundle extensions.

Database Compare

Database Compare is a test which compares DB schema and data between freshly installed Magento instance (from the PR code) and another instance which was upgraded to the PR code from the pre-previous latest minor release Magento version. Usually it runs several separate scenarios:

• Upgrade from the latest pre-previous minor release directly to the PR code (e.g. 2.1.x => 2.3-based PR code)
• Upgrade from the latest pre-previous minor release to previous minor release, and the to the PR code (e.g. 2.1.x => 2.2.x => 2.3-based PR code)
• Upgrade from the latest pre-previous minor release directly to the PR code with sample data (e.g. 2.1.x => 2.3-based PR code)
• Upgrade from the latest pre-previous minor release to previous minor release, and the to the PR code with sample data (e.g. 2.1.x => 2.2.x => 2.3-based PR code)

Magento Component Health Index

Magento Component Health Index is a test which analyses Magento code quality and calculates weighted health index (100 is great, lower is worse) per Magento Component (module). The index is calculated based on 3 groups of indicators:

• General code quality indicators (GCQI). The metrics described here:
  • Inheritance rate
  • Coupling Between Objects
  • Cyclomatic Complexity Number
  • Class Size
  • NPath Complexity
  • Code Rank
  • Number Of Methods
  • Number Of Child Classes
• Magento-specific code quality indicators (MSCQ):
  • Inline JS usage (<script> tag without attribute x-magento-init or x-magento-template)
  • PHP Static method calls (except unit tests)
  • PHP global state usage (global variables usage & static local variable usage, registry object usage)
  • PHP usage of protected modifiers for methods & properties (except unit tests)
  • PHP and JS 'fixmeComments' usage (TODO, FIXME, HACK, XXX, @SuppressWarnings etc.)
• Legacy API usage indicators (LAI):
  • Legacy method calls in action classes (loadLayout(), renderLayout() etc.)
  • Legacy classes extension (\Magento\Backend\Block\Widget\Grid\Extended, \Magento\Backend\Block\Widget\Form, \Magento\Backend\Block\Widget\Form\Generic, \Magento\Payment\Model\Method\AbstractMethod, \Magento\Payment\Model\Method\Cc, \Magento\Backend\Block\Widget\Grid etc.)
  • Legacy layout block classes usage
  • Direct class calls instead of an interface that has DI preference
  • Comparison of composer dependencies against etalon file