Tornado, getting "description should only be set when status_code is set to StatusCode.ERROR" for all spans

[tmr-prestolabs]

Describe your environment
Python 3.8.2
Platform: NAME="CentOS Linux", VERSION="8 (Core)"
Dependencies:

opentelemetry-api==1.2.0
opentelemetry-exporter-jaeger==1.2.0
opentelemetry-exporter-jaeger-proto-grpc==1.2.0
opentelemetry-exporter-jaeger-thrift==1.2.0
opentelemetry-exporter-otlp==1.2.0
opentelemetry-exporter-otlp-proto-grpc==1.2.0
opentelemetry-instrumentation==0.21b0
opentelemetry-instrumentation-tornado==0.21b0
opentelemetry-proto==1.2.0
opentelemetry-sdk==1.2.0
opentelemetry-semantic-conventions==0.21b0
opentelemetry-util-http==0.21b0

Steps to reproduce

• Run the module, which contains the following code:

# this is within one module of my code
...
from opentelemetry.instrumentation import tornado as tornado_instrument
...
tornado_instrument.TornadoInstrumentor().instrument()
...
logging.info(f'Version of Tornado instrumentation: {tornado_instrument.__version__}')
...

• Make a request, which is handled successfully, to create new spans and traces.
• Check that request has been handled successfully (200 code)

What is the expected behavior?
Logs have not to output any problems.

What is the actual behavior?
Logs output the following:
Version of Tornado instrumentation: 0.21b0
status.py:57] description should only be set when status_code is set to StatusCode.ERROR
and it happens for every span.

Additional context
I am not using any additional automatic or manual instrumentation. I don't set the description in any part of my code.

I am not sure it it's the issue on my side (dependency mismanagement) or if the code is not working as expected. I would like to get some help with that.
Whenever I make a request, status.py:57] description should only be set when status_code is set to StatusCode.ERROR is logged, so the logs get filled very quickly with this useless (not expected) information.

[tmr-prestolabs]

It happens because in https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-tornado/src/opentelemetry/instrumentation/tornado/__init__.py#L279, reason is always non-empty string, and the description of Status is always set to reason. So there is always a warning that Description should only be set when status_code is StatusCode.ERROR.

[owais]

Looks like a bug. We should set it only in case there are errors. PRs welcome.

[tuxtek]

+1 Had to disable for now to avoid log spend from spiking. If not fixed sooner, will take a stab at it when I get home tonight.

[owais]

@codeboten @lzchen I've heard this complain about logs being flooded and causing a considerable increase in operating costs. Users can either disable tornado instrumentation which means they lose observability for their services or disable logging which turns off all logs globally for a service. Provided we have a fix for this soon from @tuxtek or someone else, can we issue a hotfix release just for tornado-instrumentation Monday or Tuesday? This is serious enough as it's costing people a lot of money right now to instrument their Tornado apps. It could be even worse if we don't patch it as some teams might not spot the issue early and be very surprised next month when they look at their logging bills :)

I think we should revisit how we log things. In the short run, we should add support for OTEL_LOG_LEVEL so users can global disable logging from Otel libs and instrumentations while keeping it for rest of the components. In the long run, we probably want to rate limit logging to prevent loggin same message repeatedly in a very short span of time.

[codeboten]

@owais is tornado the only instrumentation that's setting the description in the event of a non-error status? Would be good to ensure this isn't happening across other instrumentation as well.

I agree that the fix here could be released pretty quickly.

In the long term, the log level is the correct way to solve this from happening in the future.

[owais]

@codeboten Sounds good. I'll take a look at other instrumentation packages and update here.

@tuxtek LMK if you can't get to this soon enough. I can totally pick it up if needed.