A DALiuGE logical graph is representing a computational workflow, where nodes are alternating data and application components. Data components are consumed and produced by applications. Edges are representing events allowing the graph to be executed and also representing the dependencies. Logical graphs can be constructed using the EAGLE visual graph editor and are stored and distributed as JSON files.
+Each graph component (application and data) contains a description of a Python object, which will be instantiated at run-time by the DALiuGE execution engine. These python objects, in general, have a payload which is the actual executable application or data, respectively. We call these objects Drops. Component descriptions are essentially JSON structured versions of the code documentation on a class, method or function level.
+A DALiuGE palette is a collection of such JSON component descriptions typically covering a Python package. That means that all methods and functions of such a package are in a single palette. When constructing a logical graph with EAGLE, users are dragging components from one or multiple palettes into the canvas and connect them. The component descriptions also aid EAGLE to check whether such connections are valid.
+Note: While DALiuGE graph components are bound to be written in Python, these components are merely wrappers or interfaces around the actual applications. DALiuGE has built-in wrapper components for dedicated pure python, general python functions and methods, C/C++ library, MPI, Bash CLI and docker based applications. The latter are limited to inter-application communication based on files, the others also offer the ability to use memory communications between different applications in a graph.
+Palettes can be generated manually using EAGLE, but much more conveniently they can be generated using the dlg_paletteGen tool. The tool supports two main modes of operations, from source code and from an installed module level. In both cases all the details about the actual running code is extracted from that code directly and can thus only be as good as what the developer(s) of that code provided. Thus the quality of the extracted component descriptions is obviously fully depended on what is available in the code. In particular any description of function arguments and keywords, or the function description itself needs to be in the code. If it is not, the component is still available, but without any hint what it is doing, or the documentation of the arguments and keywords.
Fortunately there are good coding standards for Python and C/C++ to guide developers to produce good and useful in-line documentation. For Python code the dlg_paletteGen tool supports Google, rEST and Numpy as well as the domain specific casatask style in order to match the identified arguments and keywords to documentation strings. In addition the tool uses Python introspection (inspect module) and Python type hints, if available, to fix the types of arguments and keywords as well as the return values. This type information in turn is used by EAGLE to check the validity of edges drawn by users when creating a logical graph.
This mode enables to extract component descriptions for almost any installed python module directly. This is extremely powerful and totally generic: If the code can be run in your python environment, DALiuGE can use it as well! In most cases it will provide a very acurate reflection of the module/package and does not require to write any special code or change any code at all. Palette generation for packages even as big as the whole of astropy or numpy takes only a few seconds, but generates very big palettes with thousands of components, or alternatively, a whole set of sub-module level palettes, when using the -S (--split) command line switch. The extraction also supports PyBind11 modules containing virtually no Python code at all.
This was the original approach taken by dlg_paletteGen. It takes a code source directory given as an input parameter, idir and uses doxygen to generate an intermediate XML file. That XML file in turn is then parsed and interpreted to generate the palette and at the end written to a JSON palette file specified on the command line as ofile. Since doxygen is used as the code parser it is possible to use all the more advanced doxygen features.
For dedicated DALiuGE components it is possible to use custom doxygen tagging to inform the tool.
+The tool supports components derived from a number of Python types:
+In practice, during execution, the engine has to deal with objects and not classes. Thus a component representing a class (e.g. DummyClass) is actually exposing the initializer method of the DummyClass (usually __init__ or __call__). This allows to initialize an object by placing the initializer component on the graph and then connect any of the class methods to it, where required. In order to aid this, the self argument of the initializer method is assigned to an output port of the associated component and the self argument of all other class methods is assigned to an input port of those components. The type of those ports is set to Object.DummyClass and that allows EAGLE to check and enforce valid connections. EAGLE users are thus able to construct object oriented graphs without writing a single line of code.
+NOTE: dlg_paletteGen does not expose any private functions and methods starting with '_', except for the initializers __init__ and __call__.
+These are somewhat simpler than the classes and methods and are typically stand-alone functions contained in a set of files.
+The support for direct module inspection was introduced in release 0.2.0 only and is still regarded experimental. In practice dlg_paletteGen enables the usage of a very big pool of Python software for the DALiuGE system. There are a couple of notable exceptions:
+The reason for this limitation comes from an internal Python limitation that does not allow to fully inspect such functions and thus the actual detailed signature of such functions is hidden to the Python interpreter. The behavior of python did change significantly between python3.8 and python3.10, but there are still limitations. Some larger scale packages, e.g. astropy or numpy are using a mixture between pure C extensions, wrapped C code and plain Python code, thus some of the components might show up in a somewhat limited way. We will continue to work in trying to remove these limitations and would appreciate feedback for packages, which experience issues.
+ + + + + + + + + + + + + +First pip install astropy into the virtual environment where dlg_paletteGen is also installed. Then execute:
dlg-paletteGen -rsSm astropy . .
+This will generate quite a bit of screen output, but finally you will see the extraction summary:
+...
+...
+>>>>>>> Extraction summary <<<<<<<<
+Wrote astropy_config.palette with 24 components
+Wrote astropy_conftest.palette with 5 components
+Wrote astropy_constants.palette with 2 components
+Wrote astropy_convolution.palette with 27 components
+Wrote astropy_coordinates.palette with 175 components
+Wrote astropy_cosmology.palette with 11 components
+Wrote astropy_extern.palette with 68 components
+Wrote astropy_io.palette with 131 components
+Wrote astropy_logger.palette with 11 components
+Wrote astropy_modeling.palette with 95 components
+Wrote astropy_nddata.palette with 42 components
+Wrote astropy_samp.palette with 106 components
+Wrote astropy_stats.palette with 50 components
+Wrote astropy_table.palette with 27 components
+Wrote astropy_time.palette with 25 components
+Wrote astropy_timeseries.palette with 96 components
+Wrote astropy_uncertainty.palette with 12 components
+Wrote astropy_units.palette with 206 components
+Wrote astropy_utils.palette with 158 components
+Wrote astropy_visualization.palette with 150 components
+Wrote astropy_wcs.palette with 61 components
+ (cli.py:234)
+
+All of the palettes will be generated in the current directory, but you can call the code from any directory you like.
+NOTE: You can also run the command above with an additional q flag, in which case there is no screen logging produced at all, but the palettes are still generated the same way.
The source code repository of this tool contains a complete set of example code using the supported docstring and type hinting. These examples are also used to test the code itself. Please see the files in the subdirectory tests/data.
After Installation the tool is available on the command line and as a module inside the virtual environment. There are different ways of calling it from the command line.
+$ python -m dlg_paletteGen
+or
+$ dlg_paletteGen -h
+or
+$ dlg-paletteGen -h
+
+For more information please refer to the documentation
+https://icrar.github.io/dlg_paletteGen/
+
+Version: 0.3.1
+
+positional arguments:
+ idir input directory path or file name
+ ofile output file name
+
+optional arguments:
+ -h, --help show this help message and exit
+ -V, --version show tool version and exit
+ -m MODULE, --module MODULE
+ Module load path name
+ -t TAG, --tag TAG filter components with matching tag
+ -c C mode, if not set Python will be used
+ -r, --recursive Traverse sub-directories
+ -S, --split Split into sub-module palettes
+ -s, --parse_all Parse non DAliuGE compliant functions and methods
+ -v, --verbose DEBUG level logging
+ -q, --quiet Only critical logging
+
+Path specification to a directory or a single file to be examined. See also --recursive. NOTE: If a module is specified using --module this is ignored, but still required for backwards compatibility reasons.
Path specification to an output file, which will be used to write the JSON version of the extracted palette to. If --module is specified and ofile is a . the module name will be used as the palette file name. If --split is also specified the value of ofile will be used as a prefix to all palette files.
Show the helptext displayed in the Usage paragraph above.
This allows to load a module to be examined.
+In EAGLE mode, this tag will be used to identify components which should be examined and written to the palette.
+Switches to C mode for the parsing of idir. Python is default.
+If idir is a directory containing sub-directories, this flag will recursively examine all (.py, .h, *.hpp, depending on the -c flag setting) files found and add them to the palette.
For very big packages, like scipy and astropy, the number of components in a single palette would be really big and unmanageable. Setting this switch will trigger the generation of one palette per top-level sub-module. In this case the 'ofile' attribute will be used as a prefix. If . is specified no prefix will be used.
If set, allows to examine functions and methods regardless of whether they contain special DALiuGE doxygen tags or not. Default is that only those special tags are extracted, i.e. -s needs to be specified for everything else. NOTE: We will likely change the default in the future.
Switch to DEBUG output during extraction. This does create quite a lot of output and is usually only really useful when developing the tool further, or to report a bug.
+ + + + + + + + + + + + + +