Provide feedback on TriBITS package dependencies data-structures

[bartlettroscoe]

@fnrizzi, @MikolajZuzek, @marcinwrobel1986, @keitat

Parent Issue: #63

Description

This story will be to get feedback from NextGen Analytics staff on the basic data-structures and functions for the TriBITS packages dependencies system that is so critical to managing the complex internal and external dependencies of a large CMake project. It is critical to understand these basic data-structures and the code that creates and manipulates them in order to have anything that resembles TriBITS.

NOTE: This code has nothing to do with the internally called commands like tribits_project(), tribits_package(), tribits_add_library() or tribits_add_executable() . That is, this code is completely 100% orthogonal to how targets are created in the CMake project. It only determines the set of external and internal packages that will be enabled and the order and location of the CMakeLists.txt file that need to be included with add_subdirectory() commands.

The package dependency data-structures themselves (which are currently in a state of being refactored as per #63) are described in:

• https://tribits.org/doc/TribitsDevelopersGuide.html#tribits-system-data-structures-and-functions

(NOTE: Some of the variables like ${PACKAGE_NAME}_PACKAGE_STATUS don't exist yet. This documentation is part of the planning for the refactoring to support #63.)

The code that reads in the *.cmake fragment files and creates the internal data-structures for the list of packages and their dependencies is rooted in the macro TRIBITS_READ_ALL_PROJECT_DEPS_FILES_CREATE_DEPS_GRAPH() and its function call graph is given in:

• https://tribits.org/doc/TribitsDevelopersGuide.html#function-call-tree-for-constructing-package-dependency-graph

The <filename>:<line> location of each of these macros and functions are given in the extracted documentation blocks. The bulk of the code in this call graph are defined in just a few *.cmake files (as you can see in the documentation).

A CMake -P utility script that runs this code to create and export the dependency graph for any TriBITS project as an XML file is given in:

• https://github.com/TriBITSPub/TriBITS/blob/master/tribits/ci_support/TribitsDumpDepsXmlScript.cmake

(See documentation at the top of that file. Also see the configure option -D <Project>_DEPS_XML_OUTPUT_FILE=<SOME_FILE_PATH>. And there the Python module TribitsDependencies.py makes it easy to read and process that XML file in Python.)

See some unit tests for TRIBITS_READ_ALL_PROJECT_DEPS_FILES_CREATE_DEPS_GRAPH() in:

• https://github.com/TriBITSPub/TriBITS/blob/master/test/core/TribitsReadAllProjectDepsFilesCreateDepsGraph_UnitTests.cmake

The best way to see these data-structures is to configure as TriBITS-based project (like Trilinos or TribitsExampleProject) with the configure options -D<Project>_DUMP_PACKAGE_DEPENDENCIES=ON and -D<Project>_DUMP_FORWARD_PACKAGE_DEPENDENCIES=ON (and we can add another option to print all the package arch dependency data-structures).

After the package dependency data-structure is created by TRIBITS_READ_ALL_PROJECT_DEPS_FILES_CREATE_DEPS_GRAPH() and some initial set of enables/disables are set by the user, the enable/disable logic is applied by the macro TRIBITS_ADJUST_PACKAGE_ENABLES(). The behaviors of enable/disable logic are spelled out in TriBITS Dependency Handling Behaviors. The unit tests for this logic is given in:

• https://github.com/TriBITSPub/TriBITS/blob/master/test/core/TribitsAdjustPackageEnables_UnitTests.cmake
• https://github.com/TriBITSPub/TriBITS/blob/master/test/core/DependencyUnitTests/CMakeLists.txt

Definition of Done

• Have NGA staff read over all of this code and tests and provide comments on comprehensibility and refactoring suggestions to make more clear.
• Consider factoring out files and code for this into separate subdirectory under tribits/core/ (as per Exposing parts of TriBITS as CMake modules #368).
• Consider creating small TriBITS example project that uses this TriBITS package architecture code but uses 100% raw CMake targets (to prove it can be done).

Tasks

• Set up meeting with NGA staff to walk through code and tests and discuss goals
• ???

[bartlettroscoe]

FYI: I will work to get #274 done so that this code is a little easier to read over.

[marcinwrobel1986]

As agreed on the meeting (18.06.2021) NGA will provide feedback on Function call tree for constructing package dependency graph

[marcinwrobel1986]

I will put just typos in this comment:

• line 71 variouscriteria and line 72 varaible, line 432tribits_read_defined_external_and_intenral_toplevel_packages_lists intenral
• line 77, 87, 91 and 127 tribits_read_defined_external_and_intenral_toplevel_packages_lists intenral, line 98, 111, 123 varaibles, line 248 dependnecy, line 267 alrady
• line 28tribits_read_defined_external_and_intenral_toplevel_packages_lists intenral
• line 56 depenency and varibles, line 67 varaibles, line 69 depenency, line 223 varaible, line 369 depeneency and varaibles, line 420 upstsream, line 421 varaibles and descrdibed, line 424 incrimentally, line 714 varaible

[marcinwrobel1986]

Hello Ross,

I have found something which needs to be completed:

• logic described in section ???.

Regarding complete feedback on 15 TriBITS System Maintainers Guide:

• I was able to understand the flow
• documentation in it's content is clear and precise
• functions names are descriptive (there was just one typo in function name described in previous comment)
• I have found some typos (described in previous comment)
• I would strongly focus on how documentation looks (you mentioned plans for going with RTD and SPHINX)

Regarding documentation (user interface):

• I find it difficult to read, mainly because of:
  • high contrast
  • font is not the best for readability
  • lack of different styling (helps find code faster)
• lack of navigation menu (side menu would help a lot to jump over faster)
• lack of source code built in (showing the source would help as well)

[bartlettroscoe]

@marcinwrobel1986, thanks for the feedback. Can we discuss some at the meeting today or set up a meeting to go over this?

NOTE: I am working on partitioning the document as we discussed last week by working #381.

[bartlettroscoe]

@marcinwrobel1986, could you please put in a PR with your suggested updates?

[marcinwrobel1986]

@bartlettroscoe PR #388 is ready, I have just left the functions/variables names with typos. I could change them if you want Ross, but probably I would not be able to check how they work after changes.

[bartlettroscoe]

I have just left the functions/variables names with typos. I could change them if you want Ross, but probably I would not be able to check how they work after changes.

@marcinwrobel1986, as I said in #388 comment, please go ahead and make any name changes you want. And testing TriBITS locally should be as easy as any CMake project imaginable. Just follow instructions at:

• https://github.com/tribitspub/tribits/#developing-on-tribits

Should take less than 5 mins to set up and run a serial build on any machine that has CMake 3.17+ and any C++ and C compiler.

[marcinwrobel1986]

Ok @bartlettroscoe , I will let you know when done

[marcinwrobel1986]

• line 191 tribits/package_arch/UntiTests/ seems it does not exist

[bartlettroscoe]

With the merge of #388, I guess this is done for now. The real review is when code changes that need to be made by others happens.