Add support to protobuf5 for opentelemetry-proto and regenerate proto files

[emdneto]

Description

Follow-up of this thread #3958 and continue the work from this PR #3931

We are adding support to only one major version of protobuf. In that case, 5

Main changes

• Drop support to Protobuf 4 and 3
• Regenerate proto files using libprotoc 26.1
• Update tests to use protobuf==5.26.1, grpcio=1.66.2 and googleapis-common-protos==1.63.2
• Require grpcio >= 1.63.2 for exporter-otlp-grpc

Steps to generate

grpcio-tools==1.63.2

1. Update gen-requirements.txt
2. run ./scripts/proto_codegen.sh

Issues

1. Issues with pylint: solved by add --prefer-stubs yes parameter for lint in the exporters that use proto
2. Issues with docs: solved by using the workaround PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python environment variable.

[aabmass]

Thanks Emidio!

[aabmass]

@emdneto regarding the docs CI failure

TypeError: Descriptors cannot be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

Should we just do the workaround since it's only happening as a result of docs needing everyhting installed in one virtualenv?

[emdneto]

The 95 expected checks stuck in the CI are because changes were made to workflows, so it isn't a real issue.

[xrmx]

LGTM, as already noted we have a few tests-requirements-{0,1}.txt to remove though

[aabmass]

I am trying to check if users who are stuck on protobuf <5 will be able to continue getting API/SDK updates by using older exporter version. It looks like no because we have this dep specifier in both the proto-http and proto-grpc exporters:

"opentelemetry-sdk ~= 1.27.0",

which is equivalent

>= 1.27.0, == 1.27.*

i.e. no new minor versions of the SDK. Does this concern you @emdneto @xrmx? I don't think there is any specific reason for this, we should probably fix it.

[aabmass]

One other option. Since the newly generated protos didn't require any code changes in their usage sites, we could relax the opentelemetry-proto constraint to allow the previous release (with a test) until we have to break it. Something like opentelemetry-proto>=1.27.0, <= currentversion.

[emdneto]

i.e. no new minor versions of the SDK. Does this concern you @emdneto @xrmx? I don't think there is any specific reason for this, we should probably fix it.

I think we bump this in every release no? So I suppose that's why is 1.27.0 - to match the release script?. I'm also concerned about this:

"googleapis-common-protos ~= 1.52"

Since in this version they re-generated the pb2 files to support protobuf5

and also this

"grpcio >= 1.0.0, < 2.0.0"

Because we re-generated the files with grpcio-tools 1.63.2. If we have grpcio < 1.63.2 user will get the warning:

RuntimeWarning: The grpc package installed is at version 1.62.3, but the generated code in opentelemetry/proto/collector/logs/v1/logs_service_pb2_grpc.py depends on grpcio>=1.63.2. Please upgrade your grpc module to grpcio>=1.63.2 or downgrade your generated code using grpcio-tools<=1.63.0. This warning will become an error in 1.65.0, scheduled for release on June 25, 2024.

[lzchen]

As discussed in the 10/10/2024 SIG,

I am trying to check if users who are stuck on protobuf <5 will be able to continue getting API/SDK updates by using older exporter version

@aabmass after you had to drop, we do recognize that this is a valid use case and we have gotten issues before where users could not use new versions of protobuf but also wanted to use the latest versions of the sdk. We don't want users to be in this "stuck" position, especially if we are considering "fixing this bug" (loosening the version dependency on the sdk in the exporter) AND releasing protobuf 5 at the same release. I think for the sake of speed and to not disrupt the momentum of this PR, we can go ahead with this change and release at the same time, and then we can always retroactively patch the previous release version if we ever encounter users who are in need of using sdk >= 1.27 to loosen the restriction. wdyt?

[aabmass]

I think for the sake of speed and to not disrupt the momentum of this PR, we can go ahead with this change and release at the same time, and then we can always retroactively patch the previous release version if we ever encounter users who are in need of using sdk >= 1.27 to loosen the restriction. wdyt?

Yes that's what I was going to suggest too. SGTM!

[aabmass]

about this:

"googleapis-common-protos ~= 1.52"

Since in this version they re-generated the pb2 files to support protobuf5

and also this

"grpcio >= 1.0.0, < 2.0.0"

Because we re-generated the files with grpcio-tools 1.63.2. If we have grpcio < 1.63.2 user will get the warning:

I'm OK to set a stricter constraint in this PR, wdyt? Re googleapis-common-protos, ideally they would have set proper dependency constraints and the pip resolver could figure it out.

[emdneto]

I'm OK to set a stricter constraint in this PR, wdyt? Re googleapis-common-protos, ideally they would have set proper dependency constraints and the pip resolver could figure it out.

@aabmass Added constraint to grpc>=1.63.2 for otlp grpc exporter

[aabmass]

Thanks, LGTM!