Broken installation of volue.mesh package via pip

[tnoczyns-volue]

Stubs generated from proto files are not included in the package when the wheel is built locally during pip installation.
The volue.mesh package installation is successful, but when trying to use it we are getting errors like:

Traceback (most recent call last):
  File "C:\my_fancy_script.py", line 11, in <module>
    from volue import mesh
  File "C:\Python311\Lib\site-packages\volue\mesh\__init__.py", line 5, in <module>
    from ._authentication import Authentication
  File "C:\Python311\Lib\site-packages\volue\mesh\_authentication.py", line 16, in <module>
    from volue.mesh.proto.auth.v1alpha import auth_pb2, auth_pb2_grpc
ImportError: cannot import name 'auth_pb2' from 'volue.mesh.proto.auth.v1alpha' (unknown location)

All Python SDK versions are affected.

[tnoczyns-volue]

TL;DR;

It seems the problem is due to the new version of poetry core (2.0.0 was released 4th on January). We use poetry core as a backend for building the package.

For now the temporary solution is to install build dependencies manually and install volue.mesh package with --no-build-isolation option:

python -m pip install poetry-core==1.9.1
python -m pip install grpcio-tools==1.66.1
python -m pip install --no-build-isolation git+https://github.com/Volue-Public/energy-mesh-python@v1.10.0

More context

Our package installation contains a build step to be executed on the client machine when installing the package, e.g. via pip. By default this is done by pip in an isolated build environment where all needed dependencies are installed. Those dependencies are defined in:

energy-mesh-python/pyproject.toml

Lines 76 to 81 in 0d2f524

	[build-system]
	requires = [
	"poetry-core>=1.9.0",
	"grpcio-tools>=1.37.0, <=1.66.1"
	]
	build-backend = "poetry.core.masonry.api"

This build step uses poetry core. The poetry core version does not have an upper bound limitation.
One of the things the build step does is to execute protoc to generate python stubs from proto files.
With poetry core 2.0.0 this step is executed but the generated python stubs are finally not in the python package.
I'm not sure why. Whether the protoc fails silently, does the generation in some different directory, etc.?

A temporary solution could be to not use isolated build environment, where the user decides what versions of the dependencies to use.

[tnoczyns-volue]

With poetry core 2.0.0 this step is executed but the generated python stubs are finally not in the python package.
I'm not sure why. Whether the protoc fails silently, does the generation in some different directory, etc.?

I did more digging and it turns out this is expected. I'm copying explanation from #527:

With poetry-core 2.0.0 there was a breaking change introduced:
https://python-poetry.org/blog/announcing-poetry-2.0.0#consistent-include-behavior

This affected the way we included generated python proto stubs. Namely by default if a VCS is used by default ignored files will be excluded. To avoid this we need to explicitly set needed files in include section. Include has higher priority than exclude rules.
For more info see: https://python-poetry.org/docs/pyproject#exclude-and-include

However, until now we didn't use format option. With poetry-core 2.0.0 if format is not specified the file is only included in sdist, previously it was sdist and wheel.

[tnoczyns-volue]

New Mesh Python SDK version with fix was released: v1.11.0

Added FAQ section with instruction what to do: https://volue-public.github.io/energy-mesh-python/faq.html#i-get-an-importerror-cannot-import-name-auth-pb2-from-volue-mesh-proto-auth-v1alpha

For Mesh server versions 2.15 and above upgrade to Mesh Python SDK version with the fix¹: v1.11.0
For older Mesh server versions (2.14 and below) install Mesh Python SDK package with specified dependencies versions:

# Generally we recommend to run this in virtual environment
python -m pip install poetry-core==1.9.1
python -m pip install grpcio-tools==1.66.1
python -m pip install --no-build-isolation git+https://github.com/Volue-Public/energy-mesh-python@v1.10.0

Footnotes

1. fix: set upper bound for required poetry-core version #527 ↩