Create minimal test data for examples and initialize the substitution

[Melkiades]

Fixes #708

I think it does not make sense to create again the data when tern is in Depends: and this would have only one step dependency (which is there anyway)

[github-actions]

Unit Tests Summary

    1 files    33 suites   5s ⏱️
149 tests 149 ✔️ 0 💤 0 ❌
168 runs  168 ✔️ 0 💤 0 ❌

Results for commit b6778f9.

♻️ This comment has been updated with latest results.

[shajoezhu]

hey @Melkiades , @pawelru and I had discussed this previously. the plan was to creaete minimal dataset within each of nest packages, we would like gradually reduce the dependencies between packages to avoid cohort releases, some of these packages will only need subset of these data.

[pawelru]

Please vbump tern dependency to highlight that dev version is required

[Melkiades]

hey @Melkiades , @pawelru and I had discussed this previously. the plan was to creaete minimal dataset within each of nest packages, we would like gradually reduce the dependencies between packages to avoid cohort releases, some of these packages will only need subset of these data.

I understand. It is still a bit difficult to motivate completely the creation of a copy data as there is a >95% overlap and therefore a lot of repetition involved (or only a copy-paste of data which does not look great to me). I would argue that adding the vbump and keeping this minimal test data shared between tern (on which it depends already, i.e. it has a shared namespace (right?)) is the better option. But now thinking about it, I am still wondering how many other packages depend on scda and how many of these you are planning to drop. Idk if it is better copy-and-paste or depending on tern's minimal tests. We can still work on a solution/vignette (as @edelarua vignette) that is not executed and takes in the tern data, adds the remaining cols and dumps it as a minimal test in the package.

Otherwise, I can copy Emily's vignette and add the missing cols. I would not just copy the data in tbh.

Let me know how to proceed. Options:

1. Data is stored in tmc + vignette (not executed) highlighting updates (a. starting from tern's one or b. copying all of the original vignette)
2. Data is stored in tmc (copied) and it is modified wherever needed (or eventually updated -> but this loses track of mods )
3. Data is NOT stored in tmc and we keep the dep on tern test (as it is attached directly) with mods when needed
4. Data is stored (copied) + updates are done in tern
5. data is NOT stored + updates in tern

5 would be my choice if there are not too many other packages to have this worked in. Otherwise, I would go for 1a

My apologies if this rambling of mine is not clear, but I think we have a precise design choice to make here and I am not fully convinced about them.

[pawelru]

I think I lost my nice comment somewhere that nicely answers your concerns @Melkiades so let me write it again

The aim for these activities is to limit package dependencies and make package more encapsulated and self-contained so if there is a need to update anything then we can complete that within a given package. I would like to simplify and loosen dependencies between our packages.

on which it depends already

That's exactly something that I would like to avoid!

To the point - If we use tern datasets then I think we are going to repeat the same mistake as we did with scda - essentially replace scda with tern. Let's suppose we want to change an example then one would have to go to tern and change it there, then change it accordingly here + vbump deps + relay on staged.deps. That's quite a lot and unnecessary complicated.

[shajoezhu]

we won't need adpp data, as it is for PK analysis, and there is no pk modules in tmc currently

[shajoezhu]

briliant! looks good to me @Melkiades ! though cicd is complaining. can you fix that. I will then approve the PR. thanks!

[Melkiades]

@shajoezhu This PR is not finished. I surveyed that we still need to add adaette, admh, adeg, adqs, adex, and advs here.

[Melkiades]

@shajoezhu This PR is not finished. I surveyed that we still need to add adaette, admh, adeg, adqs, adex, and advs here.

@shajoezhu as discussed, I left out these for the next issues where we might introduce these data or replace them with modified versions of other data. I would add them all as they are often specific to their module/tests.

[edelarua]

Lgtm! Don't forget to update the NEWS file too!

[shajoezhu]

lgtm! Thanks @Melkiades