Unit testing of sof code with zephyr dependencies

[dabekjakub]

Context: There are currently no unit tests for interfaces that need zephyr functionalities.

This is a place where we should share ideas/solutions for what to do about the situation.

I could not get the current unit test system to build with zephyr - maybe someone has better idea how to implement it or has already done it ?. - please comment/try.

For now the idea for "rtos including unit tests" is to create an api that would run series of tests after FW is loaded on RVP and would return the tests error codes into memory window. This would entail either baking the unit tests into the c files or testing only non private member functions in outside api. Idea came up while developing virtual memory heaps since api has to have coverage of some sort of tests but there was no possibility with current implementations. @lyakh Could You share Your commit that you created for VMH ? Just without the vmh part ? :)

Other quick suggestion is Zephyr Test Framework (Ztest) - seems like a tool for the job and it skips the need for a board. It would allow to test virtual heaps and other zephyr dependant apis in future. It needs some work in context of creating a project/zephyr aplication for it - i am not specialist in that field @lgirdwood Do You have any expertise ? As far as i understand Ztest we can put tests in the same repo and build it with west just pointing to different configuration than sof.
Run tests with twister later.

Or maybe there is a third way. Would love to hear from everyone so we can have better tests coverage in the future.

[lgirdwood]

Good proposal @dabekjakub !

Adding @marcinszkudlinski @mmaka1 @marc-hb @kv2019i @abonislawski @cujomalainey @dbaluta @andyross

I think we need 2 features here

1. Ability to run UTs via ztest on host PC via Zephyr POSIX mode. This enables easy stress testing of general SOF application and RTOS code that does not need to interact with IP or DSP environment/behaviours.
2. Ability to run same UTs from 1) and new system UTs on real DSP HW. i.e. less test iterations than 1), but UT focus includes our IP stack around drivers, PM, xtensa, RTOS scheduling, cache.

My understanding is that ztest should support operating modes 1 & 2 above and that this can all be managed via Kconfig to select the target and mode.

[cujomalainey]

@lgirdwood what is the fallback case should the test size exceed the binary size of the DSP?

Also I am betting @andyross has some pointers for posix config

[lgirdwood]

I think we are OK for size as UT mode would be a Kconfig which would only be used by developers/CI and not on end user production devices.

[marc-hb]

Echoing others: I also think ztest+twister is the answer - or at the very least worth a try. I've used ztest but not much and a long time ago.

I wouldn't call that "unit" testing because if SOF code has Zephyr dependencies then it's not really a "unit" anymore. Also, I would keep CMocka for "actual" unit testing of code without dependencies and having two different types of testing named the same defeats the purpose of a name. Naming is hard but not so hard that two different things should be named the same. This could be very simply called "SOF ztests"?

cc: @teburd

[dabekjakub]

@lgirdwood I think that ztest only goes for the first scenario - and solves most of our cases. It provides mocking capabilities so we can even mock HW responses - depending on what we test.
The second one would need a separate implementation like the one @lyakh made for testing my commit - that is why I asked him to share it here if possible. Also, this seemed like a good idea when I knew nothing about ztest. I intended to use this method for defacto unit testing. Now it would be a testsuit for IP/feature testing and my intention here was strictly to look into UT capabilities. Think we have to look into which one of those takes priority since we have CI systems in place and we fall behind on unit tests that have Zephyr as a backbone - so in my opinion first option should take priority.

As for @marc-hb comment - strictly speaking, what we talk about are interoperability tests of a kind. I call them unit tests since the situation I want to cover here is testing a specific SOF code that uses some of the Zephyr-provided API. Ex.:
We could implement our version of memory block management API like mem_blocks or just use zephyrs API. It would be wasteful not to use it. But testing the sof code now becomes difficult since it needs this API to function and would have to pull it into cmoka somehow or mock the API in cmoka itself. Hence the discussion and proposition :)

[lgirdwood]

@dabekjakub we need to support both 1 & 2 above, but you are right that the entry API/mechanism for 1 & 2 will be different. The main thing is that our test payload should be compataible for 1 & 2 for most tests. i.e. we should have a Kconfig to enable 1 or 2 above.
Lets keep it simple with a vmh ztest implementation as our starting point and we can then build from there.

[dabekjakub]

Agreed - do we have someone with the expertise to create a quick POC for ztest ?

[lyakh]

@lyakh Could You share Your commit that you created for VMH ? Just without the vmh part ? :)

@dabekjakub You're probably referring to #7688 and specifically to the second commit in it 4f3ae65 but it isn't quite the same as unit testing - that is testing on the target hardware at initialisation time. And I do want to get that code merged too, including the VMH test

[dabekjakub]

So this would cover scenario 2 as far as I understand @lgirdwood

[lgirdwood]

yes, this is use case 2 and it should reuse as much as possible for the same UT from use case 1.

[lgirdwood]

Btw, UT code reuse is important because UTs that pass on host (1) and fail on DSP (2) are meaningful results. i.e. they tell us where to look for failures.

[teburd]

I'd start by creating a tests directory just like Zephyr has today and creating the simplest test suite in there to do what you want. Then run away on posix/qemu/hardware with twister. It's possible you'll end up with some suite of test helpers to setup/teardown parts of SoF for testing in the process.

E.g. maybe something like...

cd sof
west twister -T tests/volume_module -p native_posix -vvv

Running on the hardware will require a firmware loading + console tool, you are welcome to reuse the cavstool/acetool scripts in the Zephyr repo as that's how Zephyr does it today if useful. I'm also happy to help provide more details on how that works though probably many here already know. The alternative might be reusing the kernel modules but built in a way be a bit minimal in how much they actually do and then wrap the load + log of the firmware with modprobe in a little script.