Set up unit testing infrastructure for ATDM Trilinos bash-based scripts #5939

bartlettroscoe · 2019-09-19T20:26:52Z

Description:

The ATDM Trilinos system composed of bash scripts is complex enough that we need to start adding automated testing for the bash functions and scripts.

Tasks:

Find a bash unit test harness and experiment with one ... Selected https://github.com/kward/shunit2/ ... PR ATDM: Switch to true keyword matching for utils/set_build_options.sh and add strong unit tests (#4063, #5939, CDOFA-27) #7202
Create a Triinos TriBITS package called TrilinosATDMConfigTests ... PR ATDM: Switch to true keyword matching for utils/set_build_options.sh and add strong unit tests (#4063, #5939, CDOFA-27) #7202
Write unit tests for atdm/utils/set_build_options.sh ... PR ATDM: Switch to true keyword matching for utils/set_build_options.sh and add strong unit tests (#4063, #5939, CDOFA-27) #7202
Write unit tests for atdm/utils/get_system_info.sh (both supported and custom systems).
Write unit tests for atdm/load-env.sh (both known supported and custom systems).
Run a basic local configure, build, and install for some faster building and running Trilinos packages (e.g. TeuchosCore) using atdm/ctest-s-local-test-driver.sh for 'sems-rhel7' env (only if that env is available) and ensure that the installation happens and has the right parts and they work correctly (i.e. <prefix>/load_matching_env.sh). (This can run in PR testing for one more more of the Trilinos PR builds since these run in RHEH7 machines that have the SEMS env.)

The text was updated successfully, but these errors were encountered:

bartlettroscoe · 2019-09-19T20:28:16Z

This looks pretty close to what we need:

https://github.com/kward/shunit2/

There was bad logic put in to fix the last bug. This script really needs to be put under automated test (see trilinos#5939)

There was bad logic put in to fix the last bug. This break the EMPIRE Trilinos testing/integration process (trilinos#6276). This script really needs to be put under automated test (see trilinos#5939).

There was bad logic put in to fix the last bug. This break the EMPIRE Trilinos testing/integration process (#6276). This script really needs to be put under automated test (see #5939).

There was bad logic put in to fix the last bug. This break the EMPIRE Trilinos testing/integration process (trilinos#6276). This script really needs to be put under automated test (see trilinos#5939).

For now, this just allows for manaul testing of hostname matching. Longer term, it will allow for unit testing of this logic (see #5939).

For now, this just allows for manaul testing of hostname matching. Longer term, it will allow for unit testing of this logic (see trilinos#5939).

bartlettroscoe · 2020-04-17T13:13:35Z

CC: @e10harvey

It just occurred to me how we might be able to do poor-man's unit testing with bash code without needing an elaborate unit test harness (and therefore another dependency). I think I am going to experiment with that quickly.

bartlettroscoe · 2020-04-17T17:08:19Z

I am looking at:

https://github.com/kward/shunit2

some and a few things I see that I don't like:

There a quite a few files. It is not all that small (but they are trying to support a bunch of different shells)
It supports reporting line numbers but not fine names in the error messages
Does not print out the names of vars being checked.
Does not print out the functions being called and checked.
It does not have any tests of its own (how can I know the unit test harness actually works?)

I am going to download this tool and play with this a little and see what I can learn but I think I want to build my own small little unit test harness in bash that will have:

Add asserts that will print the vars being checked and the functions being called.
All asserts will support file name and line number reporting.
It will report the total number of tests run and the number of total asserts and passing asserts (so I can grep for that in CTest).
Some unit testing of itself (raw bash tests).

…-339) This makes getting unit tests up and running very fast but I am not very impressed with the error output when a failure occurs.

bartlettroscoe · 2020-04-17T21:08:23Z

Okay, I have been using shunit2 for a little bit here and it is good enough for what I need and it has some nice features. I am going to just snapshot a version of this under Trilinos/cmake/std/atdm/test/sunit2 and be done with it. It is not that large and it has a very flexible Apache 2 license that should not give any customer a problem.

…339)

… ATDV-339)

…5939, ATDV-339)

…5939)

This now matches more carefully and will not match in the middle of words.

…_arch-and-unit-tests Automatically Merged using Trilinos Pull Request AutoTester PR Title: ATDM: Switch to true keyword matching for utils/set_build_options.sh and add strong unit tests (#4063, #5939, CDOFA-27) PR Author: bartlettroscoe

…inos#5939) This makes it robust in case you already have an ATDM Trilinos env loaded.

…TDV-322)

…hen unit tests (ATDV-322, trilinos#5939)

…s:develop' (62508f2). * trilinos-develop: Tacho - disable tests for kokkos task scheduler TrilinosCouplings: More cleanup to Poisson2D driver MueLu: Avoid throwing exceptions in smoother construction MueLu RefMaxwell: Copy D0 matrix ATDM: Remove some duplicate asserts found in review (trilinos#5939) Fix capitalization of Kokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE in ShyLU dependencies. Update TrilinosFrameworkTests for addition of TrilinosATDMConfigTest package (trilinos#5939) ATDM: van1-tx2: Switch to atdm_match_any_buildname_keyword (trilinos#5939, CDOFA-27) ATDM: van1-tx2: Add unit tests for custom_builds.sh (CDOFA-72) Add TrilinosATDMConfigTests package (trilinos#5939) ATDM: Change to true keyword matching with unit tests for set_build_options.sh (trilinos#4063, trilinos#5939) Add atdm_match_any_keyword (trilinos#5939) Add atdm_match_keyword (trilinos#4063, trilinos#5939, ATDV-339) Allow 'default' to have stuff after it (trilinos#5939, ATDV-339) Automatic snapshot commit from shunit2 at ebc4baa Tacho - export crs matrix in external interface example

…s:develop' (834fcf6). * trilinos-develop: ATDM: van1-tx2: Address problem of 'van1-tx2' matching KOKKOS_ARCH=TX2 (trilinos#4063, CDOFA-27) Tempus: Adding AppAction for IMEX RK and Partition MueLu: adapt doxygen build to changes in Xpetra Xpetra: add build target for doxygen documentation ATDM: Switch cee-rhel6/environment.sh to keyword matching and strengthen unit tests (ATDV-322, trilinos#5939) ATDM: Add unit tests for cee-rhel6/custom_builds.sh (trilinos#5939, ATDV-322) ATDM: Fix atdm/utils/set_build_options.sh on all platforms/envs (trilinos#5939) Make timer unit-tests more robust to prevent failures due to noise. ATDM: cee-rhel6: Remove support for deprecated clang and gnu compilers/mpi (ATDV-322) ATDM: cee-rhel6: fixed clang-5 typo to clang-9 (ATDV-322) ATDM: cee-rhel6: switch from clang-5.0.1 to clang-9.0.1 (ATDV-322) MueLu: fix typos MueLu: adapt doc of non-serializable data MueLu: get dual/primal node mapping from user data MueLu: use Teuchos::null MueLu: remove some trailing white spaces MueLu: make some documentation doxygen-compatible

…s:develop' (834fcf6). * trilinos-develop: (38 commits) ATDM Config: Don't get-platform on Vortex removed second typedef and made explicit set Tpetra_INST_INT_INT ON MueLu DriverCore: Epetra matrix unwrapping Xpetra: Fix CheckRepairMainDiagonal debug check MueLu: Silence warning MueLu: Fix CircNspDependency test MueLu RefMaxwell: Epetra implicit tranpose struggles with explicit zeros MueLu RefMaxwell: Only create timers when asked for ATDM: van1-tx2: Address problem of 'van1-tx2' matching KOKKOS_ARCH=TX2 (trilinos#4063, CDOFA-27) Tempus: Adding AppAction for IMEX RK and Partition MueLu: adapt doxygen build to changes in Xpetra Xpetra: add build target for doxygen documentation ATDM: Switch cee-rhel6/environment.sh to keyword matching and strengthen unit tests (ATDV-322, trilinos#5939) ATDM: Add unit tests for cee-rhel6/custom_builds.sh (trilinos#5939, ATDV-322) ATDM: Fix atdm/utils/set_build_options.sh on all platforms/envs (trilinos#5939) Make timer unit-tests more robust to prevent failures due to noise. ATDM: cee-rhel6: Remove support for deprecated clang and gnu compilers/mpi (ATDV-322) ATDM: cee-rhel6: fixed clang-5 typo to clang-9 (ATDV-322) ATDM: cee-rhel6: switch from clang-5.0.1 to clang-9.0.1 (ATDV-322) ...

…ptions.sh (trilinos#4063, trilinos#5939) This uses the new bash functions atdm_match[_any][_buildname]_keyword to do solid solid keyword matching that removes any case sensitivity in the build name or keywords themselves. As part of this, I also addressed allowing the KOKKOS_ARCH keyword used in the build name to be any keyword and allow it to appear anywhere in build name. This allows matching to what SPARC uses for its build names. (This addresses I also added strong unit tests for this using the newly snapshotted shunit2 unit testing framework for scripting code. (These unit tests are pretty solid.) NOTE: This will technically break backward compatiblity because it would match keywords in names were it should not have before. For example, before, the build name 'gnu-7.2.0-openmpi-4.0.2-opt' wouild match the keyword 'openmp' but now it will not. This may break some user's builds but hopefully they will see the problem and fix their build names.

The ATDM Trilinos Configuration scripts have become complex enough and important enough that they need to be under automated testing.

…5939, CDOFA-27) Now 'arm' will not match non-keywords. I also added this to the ATDMTrilinosConfigTests package tests.

…package (trilinos#5939) Now when there are changes in the directory cmake/std/atdm/, this will trigger the enable of the package TrilinosATDMConfigTests instead of enabling nothing.

@jmgate

Found by @jmgate and @e10harvey. Thanks!

…inos#5939) This makes it robust in case you already have an ATDM Trilinos env loaded.

…TDV-322)

…hen unit tests (ATDV-322, trilinos#5939)

bartlettroscoe · 2020-09-14T23:26:23Z

With the recent merge of PR #8013 and the soon-to-be-merged PR #8026, I think we have some pretty decent unit testing in place.

The last thing to unit test in the atdm/load-env.sh script. I think that will be easy to do with the addition of the test custom env:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/test/unit_tests/dummy_custom_system/

That will ensure that the basic mechanics work as they should.

github-actions · 2021-09-25T12:26:10Z

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

github-actions · 2021-11-06T12:24:10Z

This issue was closed due to inactivity for 395 days.

bartlettroscoe added type: enhancement Issue is an enhancement, not a bug ATDM DevOps Issues that will be worked by the Coordinated ATDM DevOps teams labels Sep 19, 2019

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Nov 14, 2019

ATDM: Fix for non-hostname match systems (ATDV-228)

10d3d28

There was bad logic put in to fix the last bug. This script really needs to be put under automated test (see trilinos#5939)

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Nov 14, 2019

ATDM: Fix for non-hostname match systems (ATDV-228)

8750d28

There was bad logic put in to fix the last bug. This script really needs to be put under automated test (see trilinos#5939)

bartlettroscoe mentioned this issue Nov 14, 2019

ATDM Build Scripts: Error in get_known_system_info.sh #6276

Closed

bartlettroscoe added a commit that referenced this issue Apr 2, 2020

Allow override of real hostname, print hostname on fail (CDOFA-27)

bc05d5b

For now, this just allows for manaul testing of hostname matching. Longer term, it will allow for unit testing of this logic (see #5939).

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 17, 2020

WIP: Flesh out some more compiler parsing tests (trilinos#5939, ATDV-…

997ee0b

…339)

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 17, 2020

WIP: Add unit tests for KOKKOS_ARCH (trilinos#5939, ATDV-339)

d3b05a2

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 17, 2020

Allow 'default' to have stuff after it (trilinos#5939, ATDV-339)

17f6acb

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 17, 2020

WIP: Add starting of unit tests for KOKKOS_ARCH parsing (trilinos#5939,…

f4f2081

… ATDV-339)

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 17, 2020

WIP: Change name of dir from tests to test (trilinos#5939, ATDV-339)

1c476e0

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 17, 2020

WIP: Update for test dir rename and move os shunit2 (trilinos#5939)

b552c91

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 17, 2020

Add atdm_match_keyword (trilinos#4063, trilinos#5939, ATDV-339)

51e7b88

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 17, 2020

Use atdm_match_keyword to match KOKKOS_ARCH (trilinos#4063, trilinos#…

e095017

…5939, ATDV-339)

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 17, 2020

Add unit tests for desired behavior of build type matching (trilinos#…

dfc3075

…5939)

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 17, 2020

Add atdm_match_any_keyword (trilinos#5939)

34f8c30

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 17, 2020

Switch to atdm_match_any_keyword for build type parsing (trilinos#5939)

a7ac4c8

This now matches more carefully and will not match in the middle of words.

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 18, 2020

WIP: Add unit tests for node type (trilinos#5939)

d1dcc5f

bartlettroscoe mentioned this issue Apr 20, 2020

ATDM: Switch to true keyword matching for utils/set_build_options.sh and add strong unit tests (#4063, #5939, CDOFA-27) #7202

Merged

bartlettroscoe removed the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Apr 20, 2020

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 20, 2020

ATDM: Fix atdm/utils/set_build_options.sh on all platforms/envs (tril…

b11eaa2

…inos#5939) This makes it robust in case you already have an ATDM Trilinos env loaded.

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 20, 2020

ATDM: Add unit tests for cee-rhel6/custom_builds.sh (trilinos#5939, A…

c9e95fb

…TDV-322)

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 20, 2020

ATDM: Switch cee-rhel6/environment.sh to keyword matching and strengt…

2623c40

…hen unit tests (ATDV-322, trilinos#5939)

bartlettroscoe mentioned this issue Apr 28, 2020

ATDM/ats1: Add initial ats1 env (ATDV-271) #7269

Merged

kliegeois pushed a commit to kliegeois/Trilinos that referenced this issue May 29, 2020

Allow 'default' to have stuff after it (trilinos#5939, ATDV-339)

a2d1e91

kliegeois pushed a commit to kliegeois/Trilinos that referenced this issue May 29, 2020

Add atdm_match_keyword (trilinos#4063, trilinos#5939, ATDV-339)

4dc242c

kliegeois pushed a commit to kliegeois/Trilinos that referenced this issue May 29, 2020

Add atdm_match_any_keyword (trilinos#5939)

67ed141

kliegeois pushed a commit to kliegeois/Trilinos that referenced this issue May 29, 2020

Add TrilinosATDMConfigTests package (trilinos#5939)

2f01ee0

The ATDM Trilinos Configuration scripts have become complex enough and important enough that they need to be under automated testing.

kliegeois pushed a commit to kliegeois/Trilinos that referenced this issue May 29, 2020

ATDM: van1-tx2: Switch to atdm_match_any_buildname_keyword (trilinos#…

c7c9431

…5939, CDOFA-27) Now 'arm' will not match non-keywords. I also added this to the ATDMTrilinosConfigTests package tests.

kliegeois pushed a commit to kliegeois/Trilinos that referenced this issue May 29, 2020

ATDM: Remove some duplicate asserts found in review (trilinos#5939)

1dcb262

Found by @jmgate and @e10harvey. Thanks!

kliegeois pushed a commit to kliegeois/Trilinos that referenced this issue May 29, 2020

ATDM: Fix atdm/utils/set_build_options.sh on all platforms/envs (tril…

0c5f4e4

…inos#5939) This makes it robust in case you already have an ATDM Trilinos env loaded.

kliegeois pushed a commit to kliegeois/Trilinos that referenced this issue May 29, 2020

ATDM: Add unit tests for cee-rhel6/custom_builds.sh (trilinos#5939, A…

bf01cfd

…TDV-322)

kliegeois pushed a commit to kliegeois/Trilinos that referenced this issue May 29, 2020

ATDM: Switch cee-rhel6/environment.sh to keyword matching and strengt…

2088661

…hen unit tests (ATDV-322, trilinos#5939)

github-actions bot added the MARKED_FOR_CLOSURE Issue or PR is marked for auto-closure by the GitHub Actions bot. label Sep 25, 2021

github-actions bot added the CLOSED_DUE_TO_INACTIVITY Issue or PR has been closed by the GitHub Actions bot due to inactivity. label Nov 6, 2021

github-actions bot closed this as completed Nov 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set up unit testing infrastructure for ATDM Trilinos bash-based scripts #5939

Set up unit testing infrastructure for ATDM Trilinos bash-based scripts #5939

bartlettroscoe commented Sep 19, 2019 •

edited

Loading

bartlettroscoe commented Sep 19, 2019

bartlettroscoe commented Apr 17, 2020

bartlettroscoe commented Apr 17, 2020

bartlettroscoe commented Apr 17, 2020

bartlettroscoe commented Sep 14, 2020

github-actions bot commented Sep 25, 2021

github-actions bot commented Nov 6, 2021

Set up unit testing infrastructure for ATDM Trilinos bash-based scripts #5939

Set up unit testing infrastructure for ATDM Trilinos bash-based scripts #5939

Comments

bartlettroscoe commented Sep 19, 2019 • edited Loading

Description:

Tasks:

bartlettroscoe commented Sep 19, 2019

bartlettroscoe commented Apr 17, 2020

bartlettroscoe commented Apr 17, 2020

bartlettroscoe commented Apr 17, 2020

bartlettroscoe commented Sep 14, 2020

github-actions bot commented Sep 25, 2021

github-actions bot commented Nov 6, 2021

bartlettroscoe commented Sep 19, 2019 •

edited

Loading