Setup log level depending of the environment

[Tansito]

Summary

Fix #727 . This PR setup different levels of logging depending of the DEBUG mode.

Details and comments

• For DEBUG=1 the gateway will print all the logs available
• For DEBUG=0 the gateway will print the logs from "WARNING" and above (ref: here)

In this case instead to add a new environment variable to manage the logs independently I opted for a way where the log level depends on the DEBUG mode to avoid to add more environment variables.

[psschwei]

Tangential question on logging that seeing "don't run debug in prod" made me think of: if we did need to run debug in prod (say there's a major incident that we need to resolve), what would it take to enable debug logs? Would we have to restart the gateway (and if so, what are the consequences of doing that)?

[psschwei]

Is there ever a case where we'd want to run with log level other than DEBUG or WARNING? I'm going to assume not (you opened the issue, so you would know 😄 ), and if we did need to run in INFO it wouldn't be too difficult to update later on.

[akihikokuroda]

It may be easier to understand to expose all log levels (instead of DEBUG 0 or 1) but this may be good for now because it requires minimum changes.

[Tansito]

It may be easier to understand to expose all log levels (instead of DEBUG 0 or 1) but this may be good for now because it requires minimum changes.

Yep, it's one of the things that I was thinking meanwhile I was developing it. If all of you agree that it should have a dedicated environment I can change it. My assumption was that most of the time the configuration will be:

• For DEBUG=1 the gateway will print all the logs available
• For DEBUG=0 the gateway will print the logs from "WARNING" and above

And we could afford one environment to take care. If you think that we can have an use case where we are going to need more granularity I'm open to change it.

Tangential question on logging that seeing "don't run debug in prod" made me think of: if we did need to run debug in prod (say there's a major incident that we need to resolve), what would it take to enable debug logs? Would we have to restart the gateway (and if so, what are the consequences of doing that)?

Currently yeah, we need to restart the application. We have all the previous logs stored though. But I assume that it will happen with every environment that we need to change currently.

[Tansito]

Is there ever a case where we'd want to run with log level other than DEBUG or WARNING? I'm going to assume not (you opened the issue, so you would know 😄 ), and if we did need to run in INFO it wouldn't be too difficult to update later on.

Initially no, to be honest. Or I don't see an use case where we are going to need to see all the logs without activate the DEBUG mode. Other thing is what you commented above, it would be nice if we could activate or deactivate the DEBUG mode without restarting the application. But I think that's another conversation haha.

[psschwei]

Assuming restarting gateway / scheduler is totally fine and wouldn't impact running jobs?

[Tansito]

Assuming restarting gateway / scheduler is totally fine and wouldn't impact running jobs?

Thinking in loud right now. So let's take my words carefully but the problematic points would be:

• If RUNNING job wants to save something. If the gateway is down the job is not going to be able to store it
• New ray-cluster of the user takes some time and we can get a Job in the middle of the process
• We interrupt in the middle of a logic the queue

We could find bad situations but it could affect to one job or so. It should not have bigger implications. The jobs running should not be affected by the restart of the gateway or the scheduler if they are just running in the ray-cluster.