Dockerized End-To-End Tests (Proof of Concept)

[lethuillierg]

Describe the contribution

A tool to perform dockerized end-to-end tests of cFS.

This tool simulates network-based interactions between containerized cFS executables and a pseudo-ground system software acting as a test runner.

(General context: #56)

Quick Overview

The tool makes a containerized cFS executable (cfs service) and a test runner (gsw service) interact. The following screen captures illustrate this mechanism by comparing a passing test with a failing one.

1. gsw service: the test runner
2. cfs service: the system under test (i.e. a containerized cFS executable)
3. Zero exit code (the test passed)
4. Error voluntarily triggered for demonstration purposes: here, the NOOP command does not emit the expected event
5. Non-zero exit code (the test failed)

Quick Start

Requirements

• Docker 19 or higher
• Docker Compose compatible with compose file format 2.0 or higher

Usage

From the cFS top-level directory, run the following command:

docker-compose up --abort-on-container-exit --exit-code-from gsw

When executed for the first time, this command builds the cFS with the options set in docker-compose.yml (by default: Ubuntu 18.04, unit tests disabled, simulation native, debug build, and deprecated omitted) as well as a pseudo-ground system software that attempts to enable the telemetry and, if successful, runs a simple assertion.

The default build-time options can be found (and, if needed, modified) in docker-compose.yml (lines 7-12):

dockerfile: ./tools/e2eTests/platforms/Ubuntu/18.04/Dockerfile
args:
  - ENABLE_UNIT_TESTS=false
  - SIMULATION=native
  - BUILDTYPE=debug
  - OMIT_DEPRECATED=true

For more details, please refer to the README file located in tools/e2eTests subdirectory.

Goal

The purpose of this working proof of concept is to demonstrate the potential as well as the limitations of using Docker to perform end-to-end tests.

This approach has several advantages.

First, it aims at enabling the future incorporation of end-to-end tests in the continuous integration process, automatically ensuring that the executable is working as expected in the scope of these tests. In this regard, it should be noted that Travis can run Docker Compose commands (see: https://docs.travis-ci.com/user/docker/#using-docker-compose).

Second, it allows for more granular control of the testing environment by pinning the operating systems and the required dependencies to a specific version. Therefore, it makes the tests highly reproducible. It follows that the versions of the packages are explicitly specified in all cFS Dockerfiles. For instance (tools/e2eTests/platforms/Ubuntu/18.04/Dockerfile, lines 19-27):

RUN apt-get -qy update                              \
    && apt-get -y install --no-install-recommends   \
        ca-certificates=20190110~18.04.1            \
        git=1:2.17.1-1ubuntu0.7                     \
        cmake=3.10.2-1ubuntu2.18.04.1               \
        make=4.1-9.1ubuntu1                         \
        gcc=4:7.4.0-1ubuntu2.3                      \
        g++=4:7.4.0-1ubuntu2.3                      \
    && rm -rf /var/lib/apt/lists/*

Third, it facilitates the compilation of cFS executables using different combinations of built-time options. In other words, it potentially permits the automatic generation of matrixed cFS executables.

Limitations

The purpose of this pull request, which introduces a working proof of concept, is not per se to provide a complete end-to-end testing solution. As a consequence, this tool is not yet integrated into the existing Travis configuration, and the test runner performs a simple assertion.

This contribution is also limited by the technical characteristics of Docker. In particular:

• Broadly speaking, Docker does not natively and/or straightforwardly support "non-standard" operating systems (e.g. RTOS). That explains why, for instance (and unfortunately), VxWorks is presently not supported by this solution. Alternative or complementary approaches will be needed to extend automated end-to-end to other platforms.
• Unit tests systematically fail when the cFS images are being built (see below, Additional context). This is one of the reasons why they are disabled by default.

It should also be noted that an Alpine image is included (./tools/e2eTests/platforms/Alpine/3/Dockerfile) but is currently not operational, mainly because of the following error: /usr/include/sys/signal.h:1:2: error: #warning redirecting incorrect #include <sys/signal.h> to <signal.h> [-Werror=cpp] (see: nasa/osal#438 (comment)). I have decided to keep it because it is conceivable that it becomes operational in the future.

Testing performed

In the context of this proof of context, a simple flow is tested by the gsw test runner (tools/e2eTests/gsw/gsw.py):

• Try to enable the telemetry
• If the telemetry cannot be enabled, the test fails
• Otherwise, send a NOOP command (for the SAMPLE app) packet to the cFS
• If a SAMPLE: NOOP command event appears in the telemetry message sent by the cFS, the test passes
• Otherwise, the test fails

Expected behavior changes

This contribution does not modify the existing code. Therefore, no behavior changes are expected.

System(s) tested on

Hosts

Ubuntu

• Ubuntu 18.04
• VirtualBox 6.0
• Docker 19.03.6
• Docker-compose 1.17.1

macOS

• iMac Retina 4K, 2019
• macOS 10.15.5
• Docker 19.03.8
• Docker-compose 1.25.5

Docker

• Alpine 3 (currently not operational)
• CentOS 7
• Ubuntu 18.04, and 20.04

Additional context

Components

The tool consists of:

• Multiple Dockerfiles (one for each version of an operating system) to compile and run the cFS
• A Dockerfile to build and run a pseudo-GSW container to test the cFS executable files
• A docker-compose.yml file to make them interact. This Docker Compose file makes reference to two services: cfs (the system under test) and gsw (the test runner).

cFS Dockerfiles - cfs service

The cFS Dockerfiles have a multistage structure. The first stage builds the core-cpu1 executable file. The second one corresponds to the runtime environment for this executable.

Pseudo-GSW (test runner) - gsw service

The pseudo-GSW is a Python program, running in its own container, that, at this stage, performs a simple assertion and returns a zero (success) or non-zero (failure) exit code.

It is a proof of concept which, obviously, and if the present proposal is accepted, will have to be reworked and replaced by a more robust test runner.

cFS<>GSW interactions

cfs and gsw services are interacting on their own internal network managed by Docker Compose (docker-compose.yml, lines 27-29):

networks:
  default:
    internal: true  

Their IPs are dynamic. In order for gsw to get cfs IP as well as its own, it uses the names of the corresponding Docker Compose services, benefiting therefore from the Docker DNS lookup feature (tools/e2eTests/gsw/gsw.py, lines 35 and 42):

CFS_IP = socket.gethostbyname('cfs')
. . .
GSW_IP = socket.gethostbyname('gsw')

Unit Tests Failures

As evoked above, the unit tests fail during the building stage (i.e. at compile-time, but not at runtime). This is due to a Docker limitation vis à vis POSIX message queues that is explainable by the fact that Docker disables, among other things, the CAP_SYS_RESOURCE Linux capability. When a cFS image is being built with unit tests enabled, the unit tests that fail are osal-core-test (OS_QueueCreate, OS_QueueDelete, OS_QueueGetIdByName and OS_QueueGetInfo) as well as queue-timeout-test.

There are four possible approaches to overcome this issue:

1. Disable unit tests at build time. In the context of continuous integration testing, that would make sense as the unit tests would continue to be performed in a preliminary and separate job.

2. Allow unit tests to fail at build time.

3. Modify the unit tests to make them fully compatible with Docker. Not necessarily desirable as it would affect the characteristics of the system under test.

4. Run unit tests at runtime. Indeed, the solution requires the Docker containers to run with capability CAP_SYS_RESOURCE (It should be noted that privileged: true would have the same effect but with a more extensive scope) (docker-compose.yml, lines 13-14):

   cap_add: 
     - CAP_SYS_RESOURCE

   Such an approach would however blur the line between unit testing and end-to-end testing. For this reason, I would mostly recommend against it.

Suggestions for Future Improvements

• A more complete and robust test runner
• Travis integration
• A solution to generate matrixed cFS executables and compare their respective test results

Third party code

None

Contributor Info - All information REQUIRED for consideration of pull request

Guillaume Lethuillier
Personal, individual CLA submitted

[astrogeco]

@lethuillierg thank you for this very detailed contribution. I'm excited for our team to review it and play with it!

[skliper]

@lethuillierg note you don't need a cFS CLA at this point. It may considered for 7.0 (which would be a new CLA) or fall under the cFE CLA, but we will figure it out. Thanks!

[dzbaker]

CCB 16 March 2023: Discussed today. There are a lot of really good ideas in this PR. At the moment, there isn't a strong pull for Docker configs, and GitHub Workflows are currently used for CI testing. Will revisit in the future if need arises.