Support for CI/CD pipelines

[mchrominski]

Version used

• AET 2.1.7

Context of the issue/feature

With the introduction of Shared Patterns (#121) and later fix for using latest available 'patternSuite' we now have the higher comfort of AET usage within the development lifecycle.

However, it is still unnatural to use AETs within the CI pipeline. When a change is introduced and accepted on a single environment, if there are more environments in the pipeline, same change needs to be accepted on all the subsequent environments.

From my point of view, the test results (similar to expectations in the unit test methods) are closely linked to the source code and can be treated as a code signature. Surely, there are other factors like the environment configuration or the content that influence the tests, but I think it's reasonable to introduce a relation between a code (in given version) and the expected test results.

Similar use case appear in a hotfix process for an already release version. Currently, if a change is applied to an old release, the user first need to (blindly) accept the results of release retesting, then apply the change and look for any regression. Ideally, the first test outcome could be reused from the history making sure the change assessment is accurate.

After conducting couple of design sessions below are findings:

• The test results shouldn't be stored inside the repository, due to the large size
• Keeping the GIT graph (either code or commits) inside AET seems an abuse, due to tight coupling to this idea
• The feature should be an option, in some development lifecycles this isn't a real problem
• Quite often, there isn't a single GIT repository that is being deployed to the web server, due to the multi-module architectures
• Current Shared Patterns implementation is not enough, as the same pattern will be used for all the environments, but the expectation is that the environment (AET started for the environment) will detect the required pattern in a smart way
• The solution shouldn't base on commit hashes but rather on the state of files in the repository, due to merges being a separate commit
• If a task A has been accepted on the first environment, then task B, and the workflow allows task B to be merged before task A, it is expected that there will be no matching suite while testing the second environment, because such configuration was not tested before. Once task A is merged, the suite should be found
• There will be issues with regards to comparing results between environments, because the test outcome includes environment-specific content (like domain in URLs). This will be treated separately, with a goal to have an environment-free patterns
• Current implementation of pattern cleaner service is going to punish this solution so we'd need to have a smarter way of finding unused patterns, and the same time a way to mark a pattern as something to not to clean at all (because it's related to a LTS release)
• If a change is also to be introduced to the AET suite configuration, the repository should be part of the codebase meaning a change in suite also changes the state of repository binary state

Steps to reproduce

1. Have two environments, being built from same repository but different branches
2. Introduce code change A into the branch used for environment 1
3. Accept any changes required
4. Merge code change A into the branch used for environment 2

It would be useful to automatically update the pattern, as the expectation has been set up while accepting the change on environment 1. If the change flows through multiple environments, there shouldn't be a need to manually approve the change, if it fits already accepted pattern.

Proposed solution

Let's introduce a function hash(code)=hash(repo1, repo2, repo3, ...) which is a hash of all the repositories returning a unique result for every binary state of the repository.

While running the AET suite, let's calculate that hash and push it as a parameter to the AET endpoint.

While looking for a pattern to compare to, if the parameter is available, let's first look for patterns that have the hash assigned already, in the database.

If the pattern is accepted, and a hash attribute was provided, let's save it in the database next to the pattern.

If the hash has been reported, but wasn't found in the database, let's find the previously accepted pattern on that environment. To recognise the environment, another parameter (or domainURL) should be provided.

Let's add a new button (or a technial endpoint) that will be clicked (or used while releasing) to mark a pattern we want to save for the future.

Let AET know what is our final-environment that we consider the end of the development pipeline. If a change reaches this environment, it outdates the previous change on that environment making it available to free up.

Other aspects / bucket list for considerations

• reverting a change
• reducing the volume of database
• environment-free tests
• how to find patterns to clear

[malaskowski]

After today's discussion with @mchrominski the flow of the AET processing with this idea implemented would look like this:

The next step should be researching a mechanism that will enable counting hash based on parameters that @mchrominski described above.
It looks that overall change in the AET will be not a huge effort and will mainly depend on implementing the flow form the diagram above.

[waldemarGr]

Calculation of the checksum for a project containing 55,000 files (SHA-250/CRC32).

Calculation in the standard way of the whole project takes about 60 seconds
Due to the long time, let's consider the incremental calculation of the checksum.
The incremental calculation of the checksum in most cases should not last longer than 5s

The first checksum calculation will be performed for all individual files (it will be executed only once during the implementation of the solution)
The result will be file paths and a checksum for each file and saved in the FilesChecksum.chk file.
Next, the checksum of the FilesChecksum.chk( contains 55 000 schecksums) file will be calculated and saved in ProjectChecksum.chk (1 checksum)

An incremental computed chcecksum can be initialized from Jenkins, post-merge hook or Maven / Gradle.

• In the case of initiation via Jenkins / post-merge hook, the calculation will start after completing the merge feature branch to the master. New values ​​of the checksum will be written to the master branch (amend ProjectChecksum.chk and FilesChecksum.chk) .
• In the case of mvn / gradle, the calculation will take place each time the project is built.

The incremental calculation of the checksum consists of:

1. checking which files have been changed
2. calculation of the checksum only for changed files
3. update of the checksum of changed files in FilesHashcode.chk
4. calculating the checksum for FilesHashcode.chk and saving the result in ProjectHashcode.chk

To ensure the possibility of providing information about which files have been changed in the project, consider using Git \ CVS or java service that detects changes in the files (implement the WatchService interface).

[waldemarGr]

Update
The upper scenario calculates all project files. If we consider ".igorefile" the number of files to be calculated has dropped to 8382. For this reason, the calculation time is from 7 to 10 seconds.

[radeklawgmin]

One more case worth considering:

• there are two (or even more) different streams of one project,
• one stream develops a group of features in one environment (e.g. DEV1),
• the other stream develops some other group of features, in the other environment (e.g. DEV2),
• two (or even more) streams share some part of the code and content which are synhcronized from time to time,
• these two (or even more) streams do not know much about each other but would like to check if a change introduced in one stream affects the other but at the same time does not introduce any unwanted change,
• so in fact there will be two (or more) suites of tests and each suite will be run in a different environment by a different team,
• in these two suites there will be some common set of patterns that one stream would like to take from the other stream and the other way round, which means that each stream would like to verify pages with components created within this stream but as for the other part of the suite they would refer to patterns tested by the other stream in a different environment.