testutils: add helpers for running Release Specs

[jia200x]

This PR adds some helpers to help developers run Release Specs more efficiently.

This is still in an alpha state, but can be used for running some tests. As you can see there's a lot of code duplication (among other stuff). This is part of the work to be done. All feedback and contributions are welcome!!

I'm providing 4 scripts for running 05-single-hop-udp tests. It's required before hand to configure TUN/TAP interface for native and to have an IoT-LAB account with already authenticated iotlab-cli tools.

[miri64]

Would be good to flake8 these before merging (also I prefer to have a if __name__ == "__main__": in all the exacutable scripts, so they might be usable as modules in the future)

[jia200x]

ok, I think it might make sense to give it a look now so other maintainers can run this test :)

[miri64]

I think the TODO would make the function more complicated than it needs to be. Keeping it seperated within get_nodes_addresses() makes the API simpler.

[jia200x]

@aabadie @miri64 are we still interested in these tests? Last time saved me a lot of time during release-spec tests.

[miri64]

Yes, but we should maybe make use of RIOT-OS/RIOT#10431 (or something similar that utilizes python's native testing features) to better be able to integrate it into whatever test framework we decide on later.

[aabadie]

@jia200x, I used the scripts provided by this PR to perform some checks on 2019.01-RC1. It is useful indeed but there are problems:

• Locally, I couln't rebase the current branch of this PR on top of master, even though it's not a problem for GitHub. The history of this PR is total mess, to save me time and headache I went to a git reset and manually fix things locally...
• IMHO, the mixin design is bad. It makes things too much hidden, implicit. I would prefer the use of plain functions and maybe a class wrapping the call to the expect process.
• You are using iotlab cli-tools to start/stop experiment but are you aware that it's also a Python package ? You can import the useful functions from its API and use them directly instead of wrapping everything in an external process call:

from iotlabcli.auth import get_user_credentials
from iotlabcli.rest import Api
from iotlabcli.experiment import (submit_experiment, wait_experiment,
                                  stop_experiment, exp_resources)


def submit_iotlab_experiment(nodes, name='test', duration=60):
    api  = Api(*get_user_credentials())
    resources = exp_resources(nodes)
    return submit_experiment(api, name, duration, [resources])['id']


def wait_iotlab_experiment(exp_id):
    ret = wait_experiment(Api(*get_user_credentials()), exp_id)
    return ret


def stop_iotlab_experiment(exp_id):
    ret = stop_experiment(Api(*get_user_credentials()), exp_id)
    return ret

if __name__ == '__main__':
    nodes = ['m3-1.saclay.iot-lab.info', 'samr21-1.saclay.iot-lab.info']
    exp_id = submit_iotlab_experiment(nodes)
    print(wait_iotlab_experiment(exp_id))
    # Do stuff
    stop_iotlab_experiment(exp_id)

• Some task script are not doing what the specs say: example task 4.4 says ICMPv6 echo request/reply exchange between an iotlab-m3 and a samr21-xpro node. but the script books 2 iotlab-m3
• Don't use parenthesis with assert, it's a keyword and it is not meant to used like a function, just use assert a is not None or whatever
• Some setup are almost undocumented when testing on the native platform, calling tapsetup to setup an interface is needed. On my Linux, I have to call make term as root (using sudo), etc

Instead of having all this in a single PR (+1385 lines of pure Python, no comments, no docstring, etc), I would suggest to first refine the initial design and once done, adapt each task incrementally.

Locally, I also had to make some changes that could be useful in the end (added a script for the task 4.9 + adapted a few things).

[miri64]

I think some stuff here can ported and cross-contaminated with what I started in RIOT-OS/RIOT#10431

[jia200x]

@aabadie thanks for the feedback!

Locally, I couln't rebase the current branch of this PR on top of master, even though it's not a problem for GitHub. The history of this PR is total mess, to save me time and headache I went to a git reset and manually fix things locally...

Will fix

IMHO, the mixin design is bad. It makes things too much hidden, implicit. I would prefer the use of plain functions and maybe a class wrapping the call to the expect process.

Roger

You are using iotlab cli-tools to start/stop experiment but are you aware that it's also a Python package ? You can import the useful functions from its API and use them directly instead of wrapping everything in an external process call:

I was not away of the Python package. It makes sense indeed to use it.

Some task script are not doing what the specs say: example task 4.4 says ICMPv6 echo request/reply exchange between an iotlab-m3 and a samr21-xpro node. but the script books 2 iotlab-m3

Good catch

Don't use parenthesis with assert, it's a keyword and it is not meant to used like a function, just use assert a is not None or whatever

👍

Some setup are almost undocumented when testing on the native platform, calling tapsetup to setup an interface is needed. On my Linux, I have to call make term as root (using sudo), etc

Yes, I agree it needs more documentation.

I think some stuff here can ported and cross-contaminated with what I started in RIOT-OS/RIOT#10431

I like that approach and would indeed benefit to have that TestCase wrapper here as well.
Unfortunately I currently have no time to tackle this now. But would be nice to have it at some point.

[miri64]

BTW I just noticed. The tool should also make sure it is on the actual RC that is tested, so that people do not accidentally test master or a more advanced version of the release branch.

[jcarrano]

I was able to run all of the IOT lab tests in 06 automatically. I PR'd my changes against this branch in jia200x#11 .

I strongly believe that we should use a proper testing framework for this (pytest??). Reworking task01.py felt like reinventing the wheel yet again.

[jia200x]

I was able to run all of the IOT lab tests in 06 automatically. I PR'd my changes against this branch in jia200x#11 .

Thanks for the changes!

I strongly believe that we should use a proper testing framework for this (pytest??). Reworking task01.py felt like reinventing the wheel yet again.

Definitively. The original intention was to provide some scripts to automatically run tests (this included the IoTLAB helpers and stuff). But would never replace an actual test framework.

[MrKevinWeiss]

I talked IRL with @jia200x. This is useful automating and I think should be merged in. It needs a bit of love which I am willing to give.

[jcarrano]

Somebody care to approve? My opinion is that these specs are at such an "embryonic" state that there is not much point in working contributions to perfection before merge.

[miri64]

I thought we don't merge because there is parallel work on the testing framework in RIOT upstream happening...

[MrKevinWeiss]

I should clarify I don't want to merge this. Many things are outdated. I just wanted to take it over and Jose seemed to agree. I think it may be a bit of time before RIOT will see the testing framework improvements and sometimes it is better to just get it done then strip it later... (like I am doing with the philip tests in RIOT)

[miri64]

#155 was merged so I guess this can be closed :-).