Skip to content

Commit

Permalink
[Core] Enable JSON Structured Logging Format in Ray (#50471)
Browse files Browse the repository at this point in the history 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

1. Added the `JSON` logging format option in Ray
2. Update the API doc as well as the ray logging doc according to the
change:
    - Rearranged the sections in existing ray logging document
- Copied the content of the structured logging document from Anyscale
document to ray logging document

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Mengjin Yan <mengjinyan3@gmail.com>
Co-authored-by: Dhyey Shah <dhyey2019@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
  • Loading branch information
@MengjinYan @dayshah
@edoakes @israbbani
3 people authored and israbbani committed Feb 25, 2025
1 parent 1c3441b commit 532cddd
Show file tree
Hide file tree
Showing 2 changed files with 230 additions and 87 deletions.
284 changes: 201 additions & 83 deletions doc/source/ray-observability/user-guides/configure-logging.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@ All Ray loggers are automatically configured in ``ray._private.ray_logging``. To
import logging

logger = logging.getLogger("ray")
logger # Modify the Ray logging config
logger.setLevel(logging.WARNING) # Modify the Ray logging config
```
Similarly, to modify the logging configuration for Ray libraries, specify the appropriate logger name:

Expand All @@ -242,12 +242,9 @@ ray_data_logger.setLevel(logging.WARNING)
# Here's how to add an aditional file handler for Ray Tune:
ray_tune_logger.addHandler(logging.FileHandler("extra_ray_tune_log.log"))
```
(structured-logging)=
## Structured logging
Implement structured logging to enable downstream users and applications to consume the logs efficiently.

### Application logs
A Ray app includes both driver and worker processes. For Python apps, use Python loggers to format and structure your logs.
### Using Ray logger for application logs
A Ray app includes both driver and worker processes. For Python apps, use Python loggers to format your logs.
As a result, you need to set up Python loggers for both driver and worker processes.

::::{tab-set}
Expand All @@ -265,7 +262,7 @@ Set up the Python logger for driver and worker processes separately:

![Set up python loggers](../images/setup-logger-application.png)

If you want to control the logger for particular actors or tasks, view [customizing logger for individual worker process](#customizing-worker-process-loggers)
If you want to control the logger for particular actors or tasks, view the following [customizing logger for individual worker process](#customizing-worker-process-loggers).

:::

Expand All @@ -275,38 +272,202 @@ If you are using any of the Ray libraries, follow the instructions provided in t

::::

### System logs
Most of Ray’s system or component logs are structured by default. <br />
### Customizing worker process loggers

Ray executes Tasks and Actors remotely in Ray's worker processes. To provide your own logging configuration for the worker processes, customize the worker loggers with the instructions below:
::::{tab-set}

:::{tab-item} Ray Core: individual worker process
Customize the logger configuration when you define the Tasks or Actors.
```python
import ray
import logging
# Initiate a driver.
ray.init()

@ray.remote
class Actor:
def __init__(self):
# Basic config automatically configures logs to
# stream to stdout and stderr.
# Set the severity to INFO so that info logs are printed to stdout.
logging.basicConfig(level=logging.INFO)

def log(self, msg):
logger = logging.getLogger(__name__)
logger.info(msg)

actor = Actor.remote()
ray.get(actor.log.remote("A log message for an actor."))

@ray.remote
def f(msg):
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.info(msg)

ray.get(f.remote("A log message for a task."))
```

Logging format for Python logs <br />
```bash
%(asctime)s\t%(levelname)s %(filename)s:%(lineno)s -- %(message)s
(Actor pid=179641) INFO:__main__:A log message for an actor.
(f pid=177572) INFO:__main__:A log message for a task.
```
:::

Example: <br />
:::{tab-item} Ray Core: all worker processes of a job

```{admonition} Caution
:class: caution
This is an experimental feature. The semantic of the API is subject to change.
It doesn't support [Ray Client](ray-client-ref) yet.
```
2023-06-01 09:15:34,601 INFO job_manager.py:408 -- Submitting job with RAY_ADDRESS = 10.0.24.73:6379

Use `worker_process_setup_hook` to apply the new logging configuration to all worker processes within a job.

```python
# driver.py
def logging_setup_func():
logger = logging.getLogger("ray")
logger.setLevel(logging.DEBUG)
warnings.simplefilter("always")

ray.init(runtime_env={"worker_process_setup_hook": logging_setup_func})

logging_setup_func()
```
:::

:::{tab-item} Ray libraries
If you use any of the Ray libraries, follow the instructions provided in the documentation for the library.
:::

Logging format for CPP logs <br />
::::

```bash
[year-month-day, time, pid, thread_id] (component) [file]:[line] [message]
(structured-logging)=
## Structured logging
Implement structured logging to enable downstream users and applications to consume the logs efficiently.

### Application logs

Ray enables users to configure the Python logging library to output logs in a structured format. This setup standardizes log entries, making them easier to handle.

#### Configure structured logging for Ray Core

```{admonition} Ray libraries
If you are using any of the Ray libraries, follow the instructions provided in the documentation for the library.
```

Example: <br />
The following methods are ways to configure Ray Core's structure logging format:

```bash
[2023-06-01 08:47:47,457 I 31009 225171] (gcs_server) gcs_node_manager.cc:42: Registering node info, node id = 8cc65840f0a332f4f2d59c9814416db9c36f04ac1a29ac816ad8ca1e, address = 127.0.0.1, node name = 127.0.0.1
##### Method 1: Configure structured logging with `ray.init`
```python
ray.init(
log_to_driver=False,
logging_config=ray.LoggingConfig(encoding="JSON", log_level="INFO")
)
```

:::{note}
Some system component logs are not structured as suggested above as of 2.5. The migration of system logs to structured logs is ongoing.
:::
You can configure the following parameters:

* `encoding`: The encoding format for the logs. The default is `TEXT` for plain text logs.
The other option is `JSON` for structured logs.
In both `TEXT` and `JSON` encoding formats, the logs include Ray-specific fields such as `job_id`, `worker_id`, `node_id`, `actor_id`, and `task_id`, if available.

* `log_level`: The log level for the driver process. The default is `INFO`.
Available log levels are defined in the [Python logging library](https://docs.python.org/3/library/logging.html#logging-levels).

When you set up `logging_config` in `ray.init`, it configures the root loggers for the driver process, Ray actors, and Ray tasks.

```{admonition} note
The `log_to_driver` parameter is set to `False` to disable logging to the driver
process as the redirected logs to the driver will include prefixes that made the logs
not JSON parsable.
```

##### Method 2: Configure structured logging with an environment variable

You can set the `RAY_LOGGING_CONFIG_ENCODING` environment variable to `TEXT` or `JSON` to set the encoding format for the logs.
Note that the environment variable needs to be set before `import ray`.

```python
import os
os.environ["RAY_LOGGING_CONFIG_ENCODING"] = "JSON"

import ray
import logging

### Add metadata to structured logs
If you need additional metadata to make logs more structured, fetch the metadata of Jobs, Tasks or Actors with Ray’s {py:obj}`ray.runtime_context.get_runtime_context` API.
ray.init(log_to_driver=False)
# Use the root logger to print log messages.
```
#### Example

The following example configures the `LoggingConfig` to output logs in a structured JSON format and sets the log level to `INFO`.
It then logs messages with the root loggers in the driver process, Ray tasks, and Ray actors.
The logs include Ray-specific fields such as `job_id`, `worker_id`, `node_id`, `actor_id`, and `task_id`.

```python
import ray
import logging

ray.init(
log_to_driver=False,
logging_config=ray.LoggingConfig(encoding="JSON", log_level="INFO", additional_log_standard_attrs=['name'])
)

def init_logger():
"""Get the root logger"""
return logging.getLogger()

logger = logging.getLogger()
logger.info("Driver process")

@ray.remote
def f():
logger = init_logger()
logger.info("A Ray task")

@ray.remote
class actor:
def print_message(self):
logger = init_logger()
logger.info("A Ray actor")

task_obj_ref = f.remote()
ray.get(task_obj_ref)

actor_instance = actor.remote()
ray.get(actor_instance.print_message.remote())

"""
{"asctime": "2024-07-15 19:06:06,469", "levelname": "INFO", "message": "Driver process", "filename": "test.py", "lineno": 12, "job_id": "03000000", "worker_id": "03000000ffffffffffffffffffffffffffffffffffffffffffffffff", "node_id": "824f9d7c6a82a0faf42b91f07b42667df0831034a713f04f28ba84b9"}
(f pid=4871) {"asctime": "2024-07-15 19:06:07,435", "levelname": "INFO", "message": "A Ray task", "filename": "test.py", "lineno": 17, "job_id": "03000000", "worker_id": "f8f84d811683e5d9e03744a4386b26a5cd6f6ca09fc5cdc8e1dbe5a3", "node_id": "824f9d7c6a82a0faf42b91f07b42667df0831034a713f04f28ba84b9", "task_id": "fa31b89f94899135ffffffffffffffffffffffff03000000"}
(actor pid=4939) {"asctime": "2024-07-15 19:06:08,700", "levelname": "INFO", "message": "A Ray actor", "filename": "test.py", "lineno": 23, "job_id": "03000000", "worker_id": "51d62f87e3867cdcad9aecd7b431068ea433b3104c8cc4ed1db6eef7", "node_id": "824f9d7c6a82a0faf42b91f07b42667df0831034a713f04f28ba84b9", "actor_id": "4a03b12afe5598a00eadcf9503000000", "task_id": "0ab01f2d6283d7194a03b12afe5598a00eadcf9503000000"}
"""
```

#### Add metadata to structured logs

Add extra fields to the log entries by using the `extra` parameter in the `logger.info` method.

```python
import ray
import logging

ray.init(
log_to_driver=False,
logging_config=ray.LoggingConfig(encoding="JSON", log_level="INFO")
)

logger = logging.getLogger()
logger.info("Driver process with extra fields", extra={"username": "anyscale"})

# The log entry includes the extra field "username" with the value "anyscale".

# {"asctime": "2024-07-17 21:57:50,891", "levelname": "INFO", "message": "Driver process with extra fields", "filename": "test.py", "lineno": 9, "username": "anyscale", "job_id": "04000000", "worker_id": "04000000ffffffffffffffffffffffffffffffffffffffffffffffff", "node_id": "76cdbaa32b3938587dcfa278201b8cef2d20377c80ec2e92430737ae"}
```

If needed, you can fetch the metadata of Jobs, Tasks, or Actors with Ray’s {py:obj}`ray.runtime_context.get_runtime_context` API.
::::{tab-set}

:::{tab-item} Ray Job
Expand Down Expand Up @@ -382,78 +543,35 @@ If you need node IP, use {py:obj}`ray.nodes` API to fetch all nodes and map the

::::

## Customizing worker process loggers

When using Ray, Tasks and Actors are executed remotely in Ray's worker processes. To provide your own logging configuration for the worker processes, customize the worker loggers with the instructions below:
::::{tab-set}

:::{tab-item} Ray Core: individual worker process
Customize the logger configuration when you define the Tasks or Actors.
```python
import ray
import logging
# Initiate a driver.
ray.init()

@ray.remote
class Actor:
def __init__(self):
# Basic config automatically configures logs to
# stream to stdout and stderr.
# Set the severity to INFO so that info logs are printed to stdout.
logging.basicConfig(level=logging.INFO)

def log(self, msg):
logger = logging.getLogger(__name__)
logger.info(msg)

actor = Actor.remote()
ray.get(actor.log.remote("A log message for an actor."))

@ray.remote
def f(msg):
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.info(msg)

ray.get(f.remote("A log message for a task."))
```
### System logs
Ray structures most system or component logs by default. <br />

Logging format for Python logs <br />
```bash
(Actor pid=179641) INFO:__main__:A log message for an actor.
(f pid=177572) INFO:__main__:A log message for a task.
%(asctime)s\t%(levelname)s %(filename)s:%(lineno)s -- %(message)s
```
:::

:::{tab-item} Ray Core: all worker processes of a job

```{admonition} Caution
:class: caution
This is an experimental feature. The semantic of the API is subject to change.
It doesn't support [Ray Client](ray-client-ref) yet.
Example: <br />
```
2023-06-01 09:15:34,601 INFO job_manager.py:408 -- Submitting job with RAY_ADDRESS = 10.0.24.73:6379
```

Use `worker_process_setup_hook` to apply the new logging configuration to all worker processes within a job.
Logging format for c++ logs <br />

```python
# driver.py
def logging_setup_func():
logger = logging.getLogger("ray")
logger.setLevel(logging.DEBUG)
warnings.simplefilter("always")
```bash
[year-month-day, time, pid, thread_id] (component) [file]:[line] [message]
```

ray.init(runtime_env={"worker_process_setup_hook": logging_setup_func})
Example: <br />

logging_setup_func()
```bash
[2023-06-01 08:47:47,457 I 31009 225171] (gcs_server) gcs_node_manager.cc:42: Registering node info, node id = 8cc65840f0a332f4f2d59c9814416db9c36f04ac1a29ac816ad8ca1e, address = 127.0.0.1, node name = 127.0.0.1
```
:::

:::{tab-item} Ray libraries
If you are using any of the Ray libraries, follow the instructions provided in the documentation for the library.
:::{note}
Some system component logs aren't structured as suggested preceding as of 2.5. The migration of system logs to structured logs is ongoing.
:::

::::

(log-rotation)=
## Log rotation

Expand Down
Loading

0 comments on commit 532cddd

Please sign in to comment.