Clarification on Automated Testing of Zephyr Samples with Interactive User Input

[NamrataGagare-eaton]

Hello Zephyr Team,

I am part of a platform team that supports the same application structure as Zephyr application structure.

I would like to understand how Zephyr tests sample applications that require interactive user input, such as UART echo bots or other examples where the user is expected to type input and observe output.

Specifically:
Are these samples tested automatically in CI, or is manual verification required?
Is there a recommended approach or framework for automating the testing of such interactive samples?
How does Zephyr ensure correctness for samples where the output depends on user interaction (e.g., verifying that UART output matches keyboard input)?
Are there any host-side scripts, hardware setups, or Twister features that facilitate automated interaction testing?
Any guidance or references to documentation on best practices for testing interactive samples in Zephyr would be greatly appreciated.

Thank you!

[kartben]

@PerMac @hakehuang

[hakehuang]

@NamrataGagare-eaton , please take a look at the sample.yaml/testcase.yaml for each application. current status is partial supported. we have one recommended solution which is the shell harness, please take a look at the samples\subsys\testsuite\pytest\shell

[PerMac]

As Hake says, pytest is the way to go. Here are docs. Also links to several samples https://docs.zephyrproject.org/latest/develop/test/pytest.html
To elaborate:

"Are these samples tested automatically in CI, or is manual verification required?"
Yes. Zephyr CI is running interactive tests using native_sim and qemus. The same automated tests can be run on hw. No need for manual input (linked by Hake).

"Is there a recommended approach or framework for automating the testing of such interactive samples?"
You would get the best milage with twister and pytest harness.

"How does Zephyr ensure correctness for samples where the output depends on user interaction (e.g., verifying that UART output matches keyboard input)?"
Check the linked tests/samples and docs. TL;DR: pytest allows for such tests. We provides setup fixtures and methods to open a connection with shell, write data defined in a test, read output and assert a match.

"Are there any host-side scripts, hardware setups, or Twister features that facilitate automated interaction testing?" Yes. In linked docs. E.g. https://github.com/zephyrproject-rtos/zephyr/blob/main/tests/net/lib/lwm2m/interop is a full end2end test suite, with virtual server running on a host and a client on simulator/real device.

"Any guidance or references to documentation on best practices for testing interactive samples in Zephyr would be greatly appreciated."
Provided.

[NamrataGagare-eaton]

Thank you for your detailed response and guidance in the discussion. I appreciate your insights and will go through the information further. If I have any additional questions, I’ll reach out.