Extensible reaction rates

[speth]

If applicable, provide an example illustrating new features this pull request is introducing

I figured this should start with an example, before the details of how it works:

Create an input file:

extensions:  # <--- New input file section
- type: python
  name: user_ext

phases:
- name: gas
  thermo: ideal-gas
  species: [{h2o2.yaml/species: all}]
  kinetics: gas
  state: {T: 300.0, P: 1 atm}

reactions:
- equation: H + O2 = HO2
  type: square-rate # <--- user-defined rate "type"
  A: 3.14

With the corresponding module user_ext.py to define the rate type:

import cantera as ct

@ct.extension(name="square-rate")  # <--- name corresponds to rate "type"
class SquareRate(ct.ExtensibleRate):
    def after_set_parameters(self, params, units):
        self.A = params["A"]

    def replace_eval(self, T):  # <--- will be replaced with more generic reaction data
        return self.A * T**2

This input file can now be loaded via any Cantera user interface and will call the user-defined functions for that reaction.

Changes proposed in this pull request

• Add a new ReactionRateDelegator class for delegating methods of ReactionRate to other languages, with ExtensibleRate as the Python base class for this (mirroring the convention established with ReactorDelegator and ExtensibleReactor.
• Add new ExtensionManager base class along with ExtensionManagerFactory and PythonExtensionManager classes that are responsible for loading extensions in other languages (e.g. Python modules) and registering the ExtensibleXYZ objects defined in these modules with the corresponding factories.
• Add a @extension decorator to the Python module that is used to register models with the corresponding factories using the PythonExtensionManager.
• Update the Delegator class to be able to manage the lifetime of external language objects by holding "handles" to them.
• Implement delegation of minimal ReactionRate functionality, namely set_parameters and eval.
• Switch macOS CI builders to use Homebrew Python instead of the setup-python action to fix a weird linking problem related to libintl.

If applicable, fill in the issue number this pull request is fixing

This is a major step in implementing Cantera/enhancements#79.

Work for subsequent PRs

To keep this PR a manageable size, I've deliberately limited the interface for now. In subsequent PRs, I anticipate the following further changes:

• Modify the eval interface to pass in a customizable ReactionData structure rather than just passing the temperature
• Add overloads for methods such as ReactionRate::getParameters to enable serialization and ddTScaledFromStruct to allow derivative calculations
• Modify the passing of input data to enable handling of units and conversions from the Python interface

Checklist

• The pull request includes a clear description of this code change
• Commit messages have short titles and reference relevant issues
• Build passes (scons build & scons test) and unit tests address code coverage
• Style & formatting of contributed code follows contributing guidelines
• The pull request is ready for review

[ischoegl]

@speth ... the interface looks quite promising!

Regarding further extensions: in CustomRate (which is de-facto superseded by this PR, and for good reason), one of the to-do items I have been mulling over (but never realized) was how to cleanly incorporate other properties, e.g. pressure or concentrations. It turns out that for CustomRate you can actually retrieve things from Solution while calculating rates with lambda functions, even if it is somewhat messy. This is not an option here ... what is your general thinking for an interface that goes beyond T? (Obviously this should not be addressed in this PR).

[speth]

Regarding further extensions: in CustomRate (which is de-facto superseded by this PR, and for good reason), one of the to-do items I have been mulling over (but never realized) was how to cleanly incorporate other properties, e.g. pressure or concentrations. It turns out that for CustomRate you can actually retrieve things from Solution while calculating rates with lambda functions, even if it is somewhat messy. This is not an option here ... what is your general thinking for an interface that goes beyond T?

My plan is to provide an interface much like we use in C++: Add a Python ExtensibleRateData class that has an overloadable update method that receives a Solution object, which can extract any store any data needed for rate evaluation (temperature, pressure, species enthalpies, etc), which gets called once during a full set of rate evaluations. This object then gets passed as the argument to ExtensibleRate.replace_eval instead of just the temperature.

At least so far, the biggest complication I see is the need to wrap existing C++ ThermoPhase and Kinetics objects as a Solution object that is owned from the C++ side, but I think it's manageable, and I've already got a simpler version of that implemented here, since the ReactionRateDelegator object has to manage the lifetime of the Python ExtensibleRate object.

[codecov]

Codecov Report

Merging #1382 (d7efc72) into main (c91e9eb) will decrease coverage by 0.18%.
The diff coverage is 84.96%.

@@            Coverage Diff             @@
##             main    #1382      +/-   ##
==========================================
- Coverage   71.04%   70.85%   -0.19%     
==========================================
  Files         359      363       +4     
  Lines       51707    51837     +130     
  Branches    17327    17354      +27     
==========================================
- Hits        36735    36729       -6     
- Misses      12634    12716      +82     
- Partials     2338     2392      +54     

Impacted Files	Coverage Δ
include/cantera/base/ctexceptions.h	56.52% <ø> (ø)
include/cantera/base/global.h	80.95% <ø> (ø)
interfaces/cython/cantera/reactor.pyx	89.41% <ø> (ø)
src/base/application.h	70.45% <ø> (ø)
include/cantera/base/ExtensionManager.h	50.00% <50.00%> (ø)
src/base/ExtensionManagerFactory.cpp	66.66% <66.66%> (ø)
src/base/application.cpp	50.60% <66.66%> (+0.60%)	⬆️
include/cantera/kinetics/ReactionRateDelegator.h	71.42% <71.42%> (ø)
include/cantera/base/Delegator.h	69.71% <80.95%> (+0.29%)	⬆️
interfaces/cython/cantera/delegator.pyx	78.01% <94.87%> (+3.66%)	⬆️
... and 27 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[ischoegl]

@speth ... thank you for this substantial expansion of Cantera capabilities. I see the value not necessarily in the extensible reaction rates themselves, but more in the creation of a machinery to pull in external code extensions natively, which will presumably be expanded to other areas (and languages). The code itself looks good to me; my (nit-picky) in-line comments are mostly about formatting.

As is evident from the C++ test, the availability of the Python extension is not dependent on the installation of Cantera's Python API.

One thing that I believe would be good to have is a routine check that the SCons python_package=minimal works (on that note, we still have the python_package=none option). A dedicated GitHub runner for minimal installation and C++ tests on a ubuntu/Windows/macOS matrix would give some peace of mind (e.g. considering partial Cantera installations like conda's libcantera-devel without the cantera Python API).

[speth]

As is evident from the C++ test, the availability of the Python extension is not dependent on the installation of Cantera's Python API.

This is not quite correct. To implement a Python extension, the user's class must derive from ct.ExtensibleRate, or, as this grows to cover other classes, a Cython class that holds an object derived from the C++ Delegator class. Furthermore, as I mentioned in the plans for ExtensibleRate, the plan is that the ExtensibleRateData class will receive a (Python) Solution object as input, which of course implies access to core pieces of the Cantera Python interface.

[ischoegl]

To implement a Python extension, the user's class must derive from ct.ExtensibleRate, or, as this grows to cover other classes, a Cython class that holds an object derived from the C++ Delegator class.

True (sadly).

Some comments do not apply

[ischoegl]

Thanks for updating CONTRIBUTING.md ... within the context of Sphinx/doxygen directives, there are two other comments based on prior discussion.

[ischoegl]

Thanks, @speth - LGTM!

[ischoegl]

Looks like the changes broke conda-recipes, see Action output ... 😢

undefined symbol: _ZN7Cantera22PythonExtensionManager25registerPythonRateBuilderERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_

[speth]

I don't understand why this should be a problem, given that the Cantera/cantera CI is passing. The referenced function is defined on Line 172 of PythonExtensionManager.cpp. The only "interesting" differences that I can think of are:

• this is a static member function, unlike most functions defined in the library.
• For the conda recipes, we're dynamically linking the Python module against the main library, rather than linking to the static library.

I don't think this should inherently be a problem, but apparently something isn't right here.

[ischoegl]

🤷‍♂️ … I don’t know why this is failing either. Some of the other build scripts that are run upon a merge (MSI, PyPI, macOS) are failing also, but I honestly have never looked at those before, and there may be other things at play.

[speth]

Oh, I see what the problem is. In the conda-recipes build, the shared library is built with the Python module disabled, but that is currently what also determines whether support for Python-based extensions is compiled. I thought that made sense, since the requirements for Python extension support are the same as those for the Python module itself, and using the Python extension support requires the Python module to be present.

The other thing that has changed that isn't accounted for yet in the conda recipe is that libcantera now depends on a specific version of libpython, so we can't provide just one version of libcantera for all Python versions.

[ischoegl]

🎉 … makes sense

the shared library is built with the Python module disabled, but that is currently what also determines whether support for Python-based extensions is compiled.

When I made the (incorrect) comment about the Python API not being required for extensions to work, I was coming from a similar direction. So libcantera should receive Python support … does it even make sense to create the package as a separate entity, especially given that libcantera-devel depends on it also?

This probably should go into an issue report on conda-recipes at this point.