check tests/integration/instrumentation flakyness

[JoanFM]

Describe the bug
This test tests/integration/instrumentation looks very flaky and do not let us work with 3.8

[girishc13]

The root cause is due to improper shutdown of the opentelmetry grpc client. Each test is creating all docker dependencies and then destroying at the end of the test. Since the client running in a different thread doesn't shutdown properly and leads to a segmentation fault. The gprc client has a very large back off 64s which is long compared to the length of the test execution.

[girishc13]

Ref:

• Make Exporters Shutdown spec-compliant open-telemetry/opentelemetry-python#1791
• https://github.com/open-telemetry/opentelemetry-python/blob/edf4d6cd07b5ea922180b52c9e737b7be0d59f9f/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/exporter.py#L381

[girishc13]

Waiting for review of open-telemetry/opentelemetry-python#3138.

[girishc13]

The instrumentation worker runs in a daemon thread/s in:

• https://github.com/open-telemetry/opentelemetry-python/blob/fa19e1f3753d622697bb4d69d394409c6127158e/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/export/__init__.py#L469
• https://github.com/open-telemetry/opentelemetry-python/blob/fa19e1f3753d622697bb4d69d394409c6127158e/opentelemetry-sdk/src/opentelemetry/sdk/trace/export/__init__.py#L195.

Is there a way to make sure that this thread terminates before closing the Flow context manager. The logs clearly show that the exporter is still working in the background due to the max 64s exponential back off retry logic.

[JoanFM]

Every Runtime is ensured to call the cancel and so on, we may be able to close it there

[JoanFM]

Ww should be able to call this shutdown https://github.com/open-telemetry/opentelemetry-python/blob/fa19e1f3753d622697bb4d69d394409c6127158e/opentelemetry-sdk/src/opentelemetry/sdk/trace/export/__init__.py#L412 somewhere?

[JoanFM]

And this one:

https://github.com/open-telemetry/opentelemetry-python/blob/fa19e1f3753d622697bb4d69d394409c6127158e/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/export/__init__.py#L510

[girishc13]

The runtime already does call the shutdown but the code doesn't break the exponential back off and retry loop. I have asked a question in my pr regarding the cancellation of the retry after shutdown is set.

[JoanFM]

Seems related to https://bugs.python.org/issue39853 and the same problem as astropy/astropy#10008

[JoanFM]

python/cpython#84034