[draft] test: integration test framework #20

danielskinstad · 2024-10-16T13:22:37Z

This does not run in any pipeline, nor does it use any docker-images as of now
To run the sample test, navigate to the integration directory and run e.g. pytest -s. You must set the following variables in manually inside the test:

tentant_token
auth_token
device_id

The test is configured to run using hosted.mender.io, as I couldn't get the demo servers to work (possibly due to the self-signed certificate)

Note that this test also uses the native_sim board, so it will also require you to setup the network - and the current README doesn't entirely cover it.

lluiscampos

Impressive work!

I haven't looked into the server parts (this we will totally move from integration repo based to mender-server based). The rest I read enough to get an idea of the design and I dropped some comments where I saw fit.

The next step I think is for us to sit together and break it down into small tasks for the upcoming springs 📝

lluiscampos · 2024-10-21T08:13:46Z

tests/integration/src/main.c

+}
+
+int
+main(void) {


The main() is the same as for the reference application, right? Then I think we should refactor it so that we have one file for the main() that is shared and one separate file for the callbacks that can be different.

This is to avoid pitfalls of modifying the code in only one of the tests.

Alternatively, we could mark the callback as weak function attributes in the reference app, and for the integration tests only override these. I like this second approach more, as it can also be easier extended by users if they want to reuse our reference app for quick tests.

The main() is the same as for the reference application, right?

Except for the test_update_module_register(), of course. Then, I don't know what could be a good design to share this code 🤔

Yes, it's the same but with the callbacks defined and test_update_module_register(), as you said. I guess we could guard the test update module with #ifdef CONFIG_INTEGRATION_TEST in the reference application main

tests/integration/test_mender_mcu.py

tests/integration/helpers.py

tests/integration/device.py

lluiscampos · 2024-10-21T08:29:33Z

tests/integration/device.py

+                "-DCONFIG_LOG_MODE_IMMEDIATE=y",
+                "-DCONFIG_MENDER_LOG_LEVEL_OFF=n",
+                "-DCONFIG_MENDER_LOG_LEVEL_DBG=y",
+                f'-DINTEGRATION_TESTS="y"',


This one should also be CONFIG_...

lluiscampos · 2024-10-21T08:30:50Z

tests/integration/device.py

+                "-DCONFIG_MENDER_LOG_LEVEL_DBG=y",
+                f'-DINTEGRATION_TESTS="y"',
+                f'-DCONFIG_MENDER_SERVER_HOST="{self.server_host}"',
+                f'-DCONFIG_MENDER_SERVER_TENANT_TOKEN="{self.server_tenant}"',


Just FYI, the CONFIG_ prefix is for defines injected by KConfig. We are free to chose any prefix that we like. Maybe we should use a different one for easily distinguish what is coming from KConfig and what from the test framework?

tests/integration/device.py

lluiscampos · 2024-10-21T08:34:54Z

CMakeLists.txt

 target_sources(app PRIVATE
    src/main.c
    src/utils/netup.c
    src/utils/certs.c
 )
+endif()



This we will need to rework after my refactor in mendersoftware/mender-mcu#97 🙈

vpodzime

Looks good to me otherwise.

vpodzime · 2024-10-21T08:37:39Z

tests/integration/device.py

+        # Defaults to `https://docker.mender.io`
+        self.set_host()
+
+    def set_host(self, host="https://docker.mender.io"):


What is docker.mender.io?

The URL of the demo server

Would be nice to add a comment on what it is and how to get (access to) it.

tests/integration/device.py

vpodzime · 2024-10-21T08:41:14Z

tests/integration/helpers.py

+
+def stdout(device):
+    line = device.proc.stdout.readline()
+    if device.stdout:


Maybe has_stdout would be a better name?

tests/integration/helpers.py

vpodzime · 2024-10-21T08:46:08Z

tests/integration/server.py

+URL_DEPLOYMENTS_STATUS = "/deployments/{id}/status"
+
+
+class Server:


Isn't there a class like this in the mender_integration repo?

Not with as explicit methods as abort_deployment etc., and not where you could easily connect to hosted.mender.io, at least as far as I could tell. But this server part was really only done this way to get something working. It will need to be reworked anyways, because the server-part is being moved from integration to mender-server (as Lluis said), plus the demo-server doesn't work

Cool, thanks for the extra explanation.

tests/integration/helpers.py

tests/integration/src/main.c

tests/integration/src/modules/test-update-module.c

tests/integration/test_mender_mcu.py

Integration test framework for testing mender-mcu client with native-sim. Still requires work - e.g. making the demo server work, and probably more work on the testing framework itself. It uses `integration` as a submodulel, so you need to init it. You have to set tenant token, auth_token etc. manually, and I've only tested it with 'hosted.mender.io'. I went with the compile-in approach: You can modify the callbacks directly in the test. You do this by utilizing `set_callback()` from `helpers`. The available callbacks you can modify are stored in `callbacks.py`. The callbacks have an #ifdef, and the python script creates a header-file which is imported by the test-update-module. The test-update-module and the main-file for the tests are located in the `src/` in the integration-tests directory. Ticket: MEN-7599 Signed-off-by: Daniel Skinstad Drabitzius <daniel.drabitzius@northern.tech>

And some small other adjustments, like * setting up the server as a fixture etc., * allow you to pass e.g. `hosted.mender.io` as a flag when running pytest We only use some functions from integration now, but that is related to logging Ticket: MEN-7675 Signed-off-by: Daniel Skinstad Drabitzius <daniel.drabitzius@northern.tech>

vpodzime

Looks good to me other than those nitpicks.

vpodzime · 2024-11-18T09:35:28Z

tests/integration/device.py

+
+    def compile(self, pristine=False, extra_variables=None):
+        if extra_variables is None:
+            extra_variables = []


You can do extra_variables = extra_variables or [], but your version is more explicit.

vpodzime · 2024-11-18T09:35:48Z

tests/integration/device.py

+                command_output = " ".join(command)
+                pytest.fail(f"Failed to compile with command: {command_output}")
+
+    def start(self, compile=True, pristine=False, extra_variables=[]):


Another empty list as a default argument.

vpodzime · 2024-11-18T09:36:12Z

tests/integration/helpers.py

+
+from mender_integration.tests.MenderAPI import logger
+
+THIS_DIR = os.path.dirname(os.path.abspath(__file__))


You can drop os. here.

Signed-off-by: Daniel Skinstad Drabitzius <daniel.drabitzius@northern.tech>

danielskinstad force-pushed the poc-integration-framework branch from b0ab323 to 03ab1e2 Compare October 16, 2024 13:28

danielskinstad requested review from lluiscampos and vpodzime October 21, 2024 07:48

lluiscampos reviewed Oct 21, 2024

View reviewed changes

vpodzime reviewed Oct 21, 2024

View reviewed changes

danielskinstad force-pushed the poc-integration-framework branch from 03ab1e2 to 3038797 Compare October 21, 2024 13:27

danielskinstad force-pushed the poc-integration-framework branch from 3038797 to 5ac481c Compare October 21, 2024 14:35

vpodzime approved these changes Nov 18, 2024

View reviewed changes

test: Readme for integration tests locally

ac66352

Signed-off-by: Daniel Skinstad Drabitzius <daniel.drabitzius@northern.tech>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[draft] test: integration test framework #20

[draft] test: integration test framework #20

danielskinstad commented Oct 16, 2024

lluiscampos left a comment

lluiscampos Oct 21, 2024

lluiscampos Oct 21, 2024

danielskinstad Oct 21, 2024

lluiscampos Oct 21, 2024

lluiscampos Oct 21, 2024

lluiscampos Oct 21, 2024

vpodzime left a comment

vpodzime Oct 21, 2024

danielskinstad Oct 21, 2024

vpodzime Oct 22, 2024

vpodzime Oct 21, 2024

vpodzime Oct 21, 2024

danielskinstad Oct 21, 2024

vpodzime Oct 22, 2024

vpodzime left a comment

vpodzime Nov 18, 2024

vpodzime Nov 18, 2024

vpodzime Nov 18, 2024

		URL_DEPLOYMENTS_STATUS = "/deployments/{id}/status"


		class Server:


		from mender_integration.tests.MenderAPI import logger

		THIS_DIR = os.path.dirname(os.path.abspath(__file__))

[draft] test: integration test framework #20

Are you sure you want to change the base?

[draft] test: integration test framework #20

Conversation

danielskinstad commented Oct 16, 2024

lluiscampos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vpodzime left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vpodzime left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment