Setuptools integration?

[rainwoodman]

How is the landscape of setuptools integration?

I see there is an example setup.py in the examples directory, but it is not very much a standard example. For example I cannot tell if it is referring to any .clif source code files directly or indirectly.

@akekawit @rwgk

[akekawit]

It seems like the optimization I made in commit d4d7ce3 made the examples a bit difficult to follow. I can update this to be more explicit as to where the clif source files are coming from.

Right now, the workflow is to run cmake which would invoke pyclif binary to generate CLIF cc files and then setuptools would take those and create a python extension via CLIFExtension. Are you suggesting that we should get rid of cmake in the examples and integrate pyclif invocation into setuptools?

[rainwoodman]

The use case I am picturing here is deploying Clif wrapped C++ runtime to Google Cloud Dataflow.

My current reading is that Dataflow supports installing python dependencies as sdist packages (but I haven't find out what to do about the C++ dependencies).

Are you suggesting that we should get rid of cmake in the examples and integrate pyclif invocation into setuptools?

That would certainly be very nice, but not strictly required.
For my imagined use case, as long as the sdist package is pip install-able it will work.

Does pybind11 also require bootstrapping with cmake before setup.py? Cython doesn't require a bootstrap.

[rwgk]

@henryiii, Yu is asking: Does pybind11 ... require bootstrapping with cmake before setup.py?

[henryiii]

The SDist does not require CMake, though making the SDist from the GitHub source does (for pybind11 itself anyway, so that it can generate the CMake files to put into the SDist; dependent projects are not affected).

[rainwoodman]

Thanks. The current cmake + sdist approach is also aligned with pybind11 then.

[akekawit]

For my imagined use case, as long as the sdist package is pip install-able it will work.

I think this should already be doable with the setuptools integration that we have since we're just passing the cc files that CLIF generates to setuptools Extension so they should be included when we're building sdist package. I can add sdist flow to our examples github actions to make sure that this works too. You'd still need to somehow handle CLIF C++ dependencies (absl, protobuf, etc.) in Dataflow.

Does pybind11 also require bootstrapping with cmake before setup.py? Cython doesn't require a bootstrap.

I don't think so, but pybind11 is also header-only. CMake in the examples is only here to orchestrate pyclif invocation (and protoc). We could replace CMake with Bazel, GNU Make, or doing so directly. Invoking pyclif directly from setuptools might not work too well actually for sdist since users would need the clif-matcher binary, which means you'd need LLVM dependencies among other things in the target environment.