Drop integration tests (and migration framework)

[karlnapf]

All of our python examples are currently abused for testing. This works in the way that they return some results (numbers, SGObjects) of the underlying program.
Those results are stored to files. When running the testsuite, all examples are executed once again and it is checked whether the results match the files (planned: up to a certain accuracy).

These files have to be changed very often, basically whenever we touch any related code:

1.) When we change an algorithm or an example, these results might change -- even-though the algorithm still works in a correct way, for example if we optimised an algorithm.

2.) Even worse, if we change an SGObject that is stored as an integration test, for example add a member to the class, then the integration test fails.

There is another downside: Examples are abused as tests. Shogun mostly lacks documentation and examples. However, most python examples do not do a good job in illustrating things since they are just computing a kernel matrix and return it. No help at all. Users are confused by this since it is hard for them to find information they need.
In addition, tests should be done a low level, i.e. basic building blocks should be tested. Currently, if an integration tests fails, we don't know why it fails. This is different if 1/10 unit tests fails.

My proposal (in this order)
1.) Replace all modular examples that are effectively just a test (i.e. most kernels) by unit tests. Those have the advantage that no files are involved. Unit-tests are low-level so they are less likely to fail upon changes in Shogun.

2.) Only keep python examples that properly illustrate a certain use-case.

3.) Disable integration test framework

4.) Now its much easier to modify Shogun code. and we have one testing framework less to keep an eye on.

[karlnapf]

One further nice consequence is: We can drop the migration framework.
It is never used, extremely hard to read/maintain code. It contains bugs that nobody is able to fix in reasonable time.
The reason why we have the migration framework is to not always have to re-submit integration tests when classes change. If we remove integration tests, no need for that.
As for serialised file format changes, we can just ignore newly added parameters in classes when loading from older files. This should be fine. Anything for complicated is totally unfeasible anyways.

[iglesias]

The plan sounds good but I have one fear. We used unit tests to check individually very small pieces, using very simple inputs. However, I believe that the integration tests use larger data sets. I think it is safe to keep tests that are bigger than the unit tests. Maybe we could have unit tests doing equivalent things to the ones that are done currently in the integration tests? Including reading data from input files, etc.

[karlnapf]

I agree on this argument in principle, @sonney2k also made it.

However, this is not what we do. Have a look into the integration tests, most of them do just call 1 function or 2 and then dump all objects to disc. In this state, all existing integration tests can easily replaced by unit.

If we wanted this type of big-picture behaviour, we should
1.) Write tests that do big integration type things
2.) Dump only numerical results of interest rather than the full class

But in fact, I dont even see the need for that since in Shogun, we do not have big parts of framework interacting with each other. Most things can be tested with units, can you think of an example?

[sonney2k]

@karlnapf you know my opinion about this. Nevertheless, please make a table of pros/cons in the wiki with the facts.

[karlnapf]

I can do that in here, everyone can edit this entry:

Pro:

• No need to update tests on parameter changes.
• Simple numerical tests (such as these for kernels) can and should be replaced by unit tests
• Our integration tests do not tell us what is wrong, just that something is wrong. Unit tests tell us what result is expected and what we get. This also provides information about what to expect. Integration tests just assert that there are no changes, which is a less strong check.
• We usually "fix" integration tests via overwriting them without knowing whats going on. Dangerous.
• API problems are detected by unit tests at compile time
• Shogun does not have integration problems, rather most things break around single isolated methods.
• One thing less to keep an eye on when building

Cons:

• Unit tests do not provide big picture integration
• We have to translate integration tests to unit tests before dropping, otherwise dangerous
• There might be cases where unit tests are not applicable. However, we need examples for that.
• We have to write unit tests (instead of just using the example we have to write anyways).
• We don't have tests for the *_modular interfaces - with these tests we could have them for all.
• Most of the issues we are having / had are and were detected by these tests.

[lisitsyn]

I agree our examples are neither tests nor examples thus I agree we should drop them and write some real examples.

[karlnapf]

This was finally done in #2938