Practical use of Caraya

[1984-git]

Hi,

I have recently decided to investigate unit testing in LabVIEW. I found Caraya, watched the presentation videos and now I have the feeling that I may have something useful in my hands without actually understanding how can I use it.

I've been doing LV development for 15+ years and did projects of pretty much every size. I have also seen tons of projects written by other people and participated in team developed projects, contracting etc as well. Based on this I guess I'm a relatively experienced developer. Now with that said I have never-ever seen anybody using any sort of unit testing during the development. (of course we tested our VIs but the tests were based on practical hands on tests and code reviews)

I assume that's the reason why however I understand how Caraya works I can't see how can I integrate it into my development tasks.

Could you guys give me a practical example where Caraya would help me finding bugs, helping me to capture problems and therefore making my code safer? I would press the word practical, cause the presentations I have found so far only explain how to add assertions to my VIs without giving a real life example of why should one use Caraya.

Thanks and keep up the good work!

[kosist]

I believe, that you need to investigate how unit testing itself could help during development, and not just Caraya itself.
Because once you realize, why unit testing is useful - then you would just select appropriate tool for testing. That would be Caraya, or VI Tester, or NI Unit Tester, or something else.
I would recommend to watch some presentations (they are available on youtube, I believe) about unit testing in general (maybe even not for LabVIEW, but for any other language), and then check presentations from GLA conference, related to unit testing in LabVIEW. Then, I hope, you will have better picture of how unit testing could be used in practice.

[francois-normandin]

@TMB-HUB There are many example of open source projects that contain Unit Tests, but as @kosist mentions, there are also many reasons why it is useful (and sometimes even mandatory) in commercial applications.

Tests are as diverse as anything else. Basic unit tests grow to integration tests and to system tests. The advantages of using a Unit Test framework are mainly about automation, reporting and traceability. You can run your unit tests manually and watch boolean indicators turn all green, or you can generate automated reports that aggregate a large set of tests.

Even with the best of process, it is extremely hard to declare that you have 100% test coverage on anything that is not trivial. Unit testing is like building a card castle. If you find a serious flaw at the base, the whole architecture can become flimsy. Nowadays, we hear a lot about security flaws in applications, because most of the code that exists is covered with some sort of unit test in the process, so the trivial bugs have probably been addressed by the time a library is adopted by thousands or millions or developers. (I'm talking in general, not about LabVIEW code which generally does not have the user base to allow me to statistically claim that this is the case).

Depending on the project's criticality, there are different approaches one might follow. One of the most tedious one, but which produces the highest quality code (IMO) on the first pass, is called "Test-driven development". That is, you write the unit test before you start writing the code. The test is supposed to fail until the code meets the requirements.

Most people will write a few unit tests to see if functionality of what they've written is working. That can be at development time where you want to make sure you capture the generic use cases, but also after you've received a bug report. Showing that a unit test always fails in a certain situation is a good way to claim that you understand the scope of the problem. You can then make sure that the solution you apply to the problem will not only make this new test pass, but does not break previous functionality. As your application grows, the unit tests become essential to keep backward-compatibility. How many times have we all fixed a bug, just to create another one for one of our existing users.

I`ll give you an example... yesterday morning, I decided to add a feature to a library of mine. All my tests were working the last time I touched the code in 2020. When I added a helper method to my palette, I ran my unit tests to find out that one of the tests now failed. Since I just added a new node, I was confused... It turned out that I wrote and tested the application during the winter and it passed the tests, but yesterday is on Daylight Savings Time and my timestamp was ahead by an hour! How could I have found this out without having an automated test that ran on a virtual machine set up to be on DST? That is not trivial unless you have an automated process that runs this test on a VM every time I make a change to the codebase. I reassure you, I do not have such an automated pipeline and virtual machines in place, but if I were to code for a life-critical application (aerospace, transport, medical device) that relies on this functionality for keeping everyone safe, I would want that pipeline to test for time zones and daylight saving times.

I encourage you to find examples of projects so you can see how different people approach the problem. It might be that you want to know the test coverage of you project to get paid for the work you've done, or you want to achieve/maintain an ISO9000 certification, you want to deliver a piece of code that will be used in life-critical settings, or you simply want to write a piece of code that is maintainable.

Here is a project of mine for which I followed TDD (Test Driven Development) paradigm. It is not full-coverage, but what the project claims to be doing is covered by one of many unit tests. (Concretely, it means that I do not close an issue if there is not a specific unit test that covers the requirement): https://github.com/LabVIEW-Open-Source/LV-MQTT-Broker/tree/master/Unit%20Tests

Browse through these resources to find more ideas:

• https://thelonelyant.com/labview-unit-testing-tools-overview/
• https://kosist.org/
• https://blog.jki.net/topic/unit-testing
• http://www.labviewcraftsmen.com/blog/do-you-unit-test
• http://controlsoftwaresolutions.com/software-development-utf/
• https://www.ni.com/en-us/innovations/white-papers/09/prove-it-works--using-the-unit-test-framework-for-software-testi.html
• http://stry.kr/2018/11/04/conversations-about-unit-testing/
• https://labviewwiki.org/wiki/NIWeek_2019/LabVIEW_Unit_Testing:_Outlook_%26_Tutorial

[1984-git]

Thanks for responding. I'll check these links. Until that... am I right to say that the assertions should be placed before the actual coding and not afterwards?

[kosist]

Not really, because TDD - is just one of the approaches for development. When you follow TDD then yes, you must write tests before actual coding.
But you can also use Caraya assertions for actual code which you already have, that will be fine (and useful).

[francois-normandin]

Because you've emphasized practical, here's an account of how I most often end up doing Unit Tests:

The average project is a project for internal customers that needed to get off the ground quickly, and for which I have virtually no formal unit tests. Eventually, there comes a point where the users start requesting for new features or complaining about a weird bug in such and such circumstances. My threshold to create unit test comes usually with the adoption rate of the code. I'd say this is far from ideal for commercial products, but in all honesty, most of the code I write is for internal projects anyway.

At this point when I decide to start writing unit tests, I almost never create a whole suite. There are two use cases:
1- Someone reported a bug. I write a unit test because when I make a small change, it might affect the cases that already work. I demonstrate that it passes in some circumstances, and fails in others. I then resolve until all tests pass.
2- Someone asks for a new feature, but I'm afraid it will potentially cause problems with the code already written and fully functional. I write the unit tests and code such that it still passes when I release the new feature.

If you do that piece-meal, one test at a time, after some period you end up with a growing series of tests that give you ever more confidence. TDD is really a luxury, which I rarely get to do in a practical way.