Make test suite more random and add add_noise_to_params, move to Github Actions, etc..

[tobiasraabe]

Current behavior

• The tests take too much time and by that waste ressources.
• If we generate random models, we reuse the random models generated by seeds 0-20. Thus, some edge cases might stay undetected. We could use some more randomness.

Solution / Implementation

Random tests

• First, I removed tests test generated by seeds.
• I added pytest-randomly which generates a timestamp based seed for every test session and the random number generator is reset to this seed before every test. Using this seed we have one random model, but different models for every test session, Small win.
• At last, I intervene in pytests generation of tests. For every test using the argument "model_or_seed", I rerun the test five times with the seeds [seed + i for i in range(5)] where seed is the seed generated by pytest-randomly.

Thus, we test multiple random models per session and different models across sessions. Thus, we have a better coverage than ever before. Now, it only depends on how diverse are the random models and how good are the tests.

The tests are still reproducible because every test session outputs --randomly-seed=<value> before each session which can be set as a pytest argument to re-run the tests.

Others

• I also added a function add_noise_to_params() to vary the parameters of an existing model.
• We know have coverage reports.
• We moved from Azure to Github Actions which is less error-prone, has better integration.

[codecov]

Codecov Report

Merging #320 into master will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #320   +/-   ##
=======================================
  Coverage   81.67%   81.67%           
=======================================
  Files          38       38           
  Lines        2467     2467           
=======================================
  Hits         2015     2015           
  Misses        452      452           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee4f0df...ee4f0df. Read the comment docs.

[mo2561057]

I dont get the gain of this function am I missing something or is this style guide stuff?

[tobiasraabe]

Previously we sorted the shocks within respy. Unfortunately, if we optimize with estimagic, estimagic assumes that the shocks are already in the correct format for the constraint before params even reaches respy. Now, we throw an error if the shock matrix is not correctly sorted.

[mo2561057]

Isn't there an easier way to get all combinations permutations of two elements with itertools or a similar package?

[tobiasraabe]

Due to the different labels for diagonal elements and lower triangular elements I would say no :). Using itertools would probably involve generating more than necessary and deleting some elements. I found this to be cleaner because labels are generated in the same order as required by the index.

[mo2561057]

WHy should we have so many cases/choices at such a low level? WOuldnt it be better to make the container of choices uniform at some point?

[tobiasraabe]

Although the function is low-level, it is actually a vectorized replacement for numpy.random.choice. It is only used in two different places, but needs to support ints and dict keys as choices. tuples and lists are added because it does not change anything. I think one could remove the np.ndarray check and the conversion to scalars at the end and defer some steps to where it is applied, but it reduces code and mental effort in other places if we keep it this way.