Document/update integration tests

[sajith]

The tests in fabrictestbed_extensions.tests are mostly undocumented. It is not clear to me how to use the ones that look like integration tests and/or benchmarks. I only know how to run the simple unit tests that I added, and they are already run against PRs and merges by the GitHub Actions.

It would be useful to move the tests to a top-level folder such that they are not included in FABLib packaging. We could also split them into sub-folders, like so:

tests/
├── benchmarks
├── integration
└── unit

And then run them using tox, like so:

$ tox                 # run unit tests
$ tox -e integration  # run integration tests
$ tox -e benchmarks   # run benchmarks

Because of the need for acquiring FABRIC tokens (which requires a bit of human intervention), we might not be able to run the integration tests on GitHub Actions. But they can work on developer machines, and that would be an improvement.

[kthare10]

@sajith - All the tests in fabrictestbed_extensions/tests can be removed as they are obsolete and they cannot be executed standalone in a jenkins environment. I would recommend instead to do the following:

• Take the notebooks from acceptance_testing
• Convert the notebooks into python scripts using nbconvert
• Update the python scripts to use asserts to validate the outcome
• Add these to fabrictestbed_extensions/tests
• Add a script option to fabrictestbed_extensions/pyproject.toml which allows the user to run these tests from the command line in the Jupyter Container.
• This will help automate fablib testing as well as acceptance testing

[sajith]

@kthare10 Thank you for the input! This is very helpful.

Since acceptance testing notebooks are well-exercised, they would be more useful as tests of FABLib than the "mystery" tests that we have in this repository. I think we can add a tox environment to run them, rather than adding them as script entrypoints. That probably is the right thing to do since they do not have to be packaged. Stuff marked as scripts will end up in a $PREFIX/bin upon installation (I think), which may not be desirable.

As for the existing "mystery" tests, I think I sort of have figured out a way to make them runnable -- we'll need to add a main to each of them, update the APIs, and fix any other possible broken-ness. Perhaps it is worth the effort, perhaps not.

In order to determine if resurrecting the mystery tests is worth the trouble, it would be helpful to figure out if mystery tests and acceptance notebooks are different in scope. If that is the case, it probably makes sense to bring them up-to-date? Perhaps we can remove the less useful ones on a case-by-case basis? Turning the acceptance test notebooks to FABLib tests also will require some work anyway.

I am not totally opposed to just removing them altogether. Code can be both an asset and a liability. At the moment I am just not sure which category the mystery tests belong to. :-)

[sajith]

@kthare10 Also, removing them was my initial preference too. But if I understand correctly @paul-ruth is not so sure.

[sajith]

PR #186 merely re-organized tests folder. Re-openining this issue for follow-ups.