From 7f931c4fe9248bff76e4890f8d83f4cf5f43de84 Mon Sep 17 00:00:00 2001 From: "abby.huang" <78209557+abby-cyber@users.noreply.github.com> Date: Fri, 5 Jan 2024 11:00:37 +0800 Subject: [PATCH] how to recycle running logs (#215) (#2415) * how to recycle running logs * Update 0.FAQ.md Update 0.FAQ.md * addons * Update docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md * Update docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md * Update docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md * Update docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md * comment fix --------- Co-authored-by: Chris Chen --- docs-2.0-en/20.appendix/0.FAQ.md | 7 +- .../2.log-management/logs.md | 158 +++++++++++++++++- docs-2.0-zh/20.appendix/0.FAQ.md | 7 +- .../2.log-management/logs.md | 158 +++++++++++++++++- 4 files changed, 314 insertions(+), 16 deletions(-) diff --git a/docs-2.0-en/20.appendix/0.FAQ.md b/docs-2.0-en/20.appendix/0.FAQ.md index f463025895b..50dbad6b60e 100644 --- a/docs-2.0-en/20.appendix/0.FAQ.md +++ b/docs-2.0-en/20.appendix/0.FAQ.md @@ -313,12 +313,7 @@ You can use [NebulaGraph Algorithm](../graph-computing/nebula-algorithm.md). ### "The runtime log files are too large. How to recycle the logs?" -By default, the runtime logs of NebulaGraph are stored in `/usr/local/nebula/logs/`. The INFO level log files are `nebula-graphd.INFO, nebula-storaged.INFO, nebula-metad.INFO`. If an alarm or error occurs, the suffixes are modified as `.WARNING` or `.ERROR`. - -NebulaGraph uses [glog](https://github.com/google/glog) to print logs. `glog` cannot recycle the outdated files. To rotate logs, you can: - -- Add the parameters `timestamp_in_logfile_name=true` (timestamp added to logfile) and `max_log_size=500` (log size limit in MB) to the configuration files of the three services, and then use crontab to delete logs periodically. For more information, see [`Glog should delete old log files automatically`](https://github.com/google/glog/issues/423). -- Use [logrotate](https://github.com/logrotate/logrotate) to manage log files. Before using logrotate, modify the configurations of corresponding services and set `timestamp_in_logfile_name` to `false`. +{{nebula.name}} uses [glog](https://github.com/google/glog) for log printing, which does not support log recycling. You can manage runtime logs by using cron jobs or the log management tool [logrotate](https://man7.org/linux/man-pages/man8/logrotate.8.html). For operational details, see [Log recycling](../5.configurations-and-logs/2.log-management/logs.md). ### "How to check the NebulaGraph version?" diff --git a/docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md b/docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md index 721ecf88cbf..848e58c5357 100644 --- a/docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md +++ b/docs-2.0-en/5.configurations-and-logs/2.log-management/logs.md @@ -75,6 +75,160 @@ If the log level is changed while NebulaGraph is running, it will be restored to RocksDB runtime logs are usually used to debug RocksDB parameters and stored in `/usr/local/nebula/data/storage/nebula/$id/data/LOG`. `$id` is the ID of the example. -## Log recycle +## Log recycling -How to recycle logs when the runtime log file is too large, see [FAQs](../../20.appendix/0.FAQ.md). +Glog does not inherently support log recycling. To implement this feature, you can either use [cron jobs](https://man7.org/linux/man-pages/man1/crontab.1.html) in Linux to regularly remove old log files or use the log management tool, [logrotate](https://github.com/logrotate/logrotate), to rotate logs for regular archiving and deletion. + +### Log recycling using cron jobs + +This section provides an example of how to use cron jobs to regularly delete old log files from the Graph service's runtime logs. + +1. In the Graph service configuration file, apply the following settings and restart the service: + + ```bash + timestamp_in_logfile_name = true + max_log_size = 500 + ``` + + - By setting `timestamp_in_logfile_name` to `true`, the log file name includes a timestamp, allowing regular deletion of old log files. + - The `max_log_size` parameter sets the maximum size of a single log file in MB, such as `500`. Once this size is exceeded, a new log file is automatically created. The default value is `1800`. + +2. Use the following command to open the cron job editor. + + ```bash + crontab -e + ``` + +3. Add a cron job command to the editor to regularly delete old log files. + + ```bash + * * * * * find -name "" -mtime +7 -delete + ``` + + !!! caution + + The `find` command in the above command should be executed by the root user or a user with sudo privileges. + + - `* * * * *`: This cron job time field signifies that the task is executed every minute. For other settings, see [Cron Expression](https://crontab.cronhub.io/). + - ``: The path of the service runtime log file, such as `/usr/local/nebula/logs`. + - ``: The log file name, such as `nebula-graphd.*`. + - `-mtime +7`: This deletes log files that are older than 7 days. Alternatively, use `-mmin +n` to delete log files older than `n` minutes. For details, see the find command. + - `-delete`: This deletes log files that meet the conditions. + + For example, to automatically delete the Graph service runtime log files older than 7 days at 3 o'clock every morning, use: + + ```bash + 0 3 * * * find /usr/local/nebula/logs -name nebula-graphd.* -mtime +7 -delete + ``` + +4. Save the cron job and exit the editor. + + +### Log recycling using logrotate + +Logrotate is a tool that can rotate specified log files for archiving and recycling. + +!!! note + + You must be the root user or a user with sudo privileges to install or run logrotate. + +This section provides an example of how to use logrotate to manage the Graph service's `INFO` level log file (`/usr/local/nebula/logs/nebula-graphd.INFO.impl`). + +1. In the Graph service configuration file, set `timestamp_in_logfile_name` to `false` so that the logrotate tool can recognize the log file name. Then, restart the service. + + ```bash + timestamp_in_logfile_name = false + ``` + +2. Install logrotate. + + - For Debian/Ubuntu: + + ```bash + sudo apt-get install logrotate + ``` + + - For CentOS/RHEL: + + ```bash + sudo yum install logrotate + ``` + +3. Create a logrotate configuration file, add log rotation rules, and save the configuration file. + + In the `/etc/logrotate.d` directory, create a new logrotate configuration file `nebula-graphd.INFO`. + + ```bash + sudo vim /etc/logrotate.d/nebula-graphd.INFO + ``` + + Then, add the following content: + + ```bash + # The absolute path of the log file needs to be configured + # And the file name cannot be a symbolic link file, such as `nebula-graph.INFO` + /usr/local/nebula/logs/nebula-graphd.INFO.impl { + daily + rotate 2 + copytruncate + nocompress + missingok + notifempty + create 644 root root + dateext + dateformat .%Y-%m-%d-%s + maxsize 1k + } + ``` + + | Parameter | Description | + | --------------- | ------------------------------------------------------------ | + | `daily` | Rotate the log daily. Other available time units include `hourly`, `daily`, `weekly`, `monthly`, and `yearly`. | + | `rotate 2` | Keep the most recent 2 log files before deleting the older one. | + | `copytruncate` | Copy the current log file and then truncate it, ensuring no disruption to the logging process. | + | `nocompress` | Do not compress the old log files. | + | `missingok` | Do not report errors if the log file is missing. | + | `notifempty` | Do not rotate the log file if it's empty. | + | `create 644 root root` | Create a new log file with the specified permissions and ownership. | + | `dateext` | Add a date extension to the log file name.
The default is the current date in the format `-%Y%m%d`.
You can extend this using the `dateformat` option. | + | `dateformat .%Y-%m-%d-%s` | This must follow immediately after `dateext` and defines the file name after log rotation.
Before V3.9.0, only `%Y`, `%m`, `%d`, and `%s` parameters were supported.
Starting from V3.9.0, the `%H` parameter is also supported.| + | `maxsize 1k` | Rotate the log when it exceeds 1 kilobyte (`1024` bytes) in size or when the specified time unit (e.g., `daily`) passes.
You can use size units like `k` and `M`, with the default unit being bytes. | + + Modify the parameters in the configuration file according to actual needs. For more information about parameter configuration, see [logrotate](https://github.com/logrotate/logrotate). + +4. Test the logrotate configuration. + + To verify whether the logrotate configuration is correct, use the following command for testing. + + ```bash + sudo logrotate --debug /etc/logrotate.d/nebula-graphd.INFO + ``` + +5. Execute logrotate. + + Although `logrotate` is typically executed automatically by cron jobs, you can manually execute the following command to perform log rotation immediately. + + ```bash + sudo logrotate -fv /etc/logrotate.d/nebula-graphd.INFO + ``` + + `-fv`: `f` stands for forced execution, `v` stands for verbose output. + +6. Verify the log rotation results. + + After log rotation, new log files are found in the `/usr/local/nebula/logs` directory, such as `nebula-graphd.INFO.impl.2024-01-04-1704338204`. The original log content is cleared, but the file is retained for new log entries. When the number of log files exceeds the value set by `rotate`, the oldest log file is deleted. + + For example, `rotate `2` means keeping the 2 most recently generated log files. When the number of log files exceeds 2, the oldest log file is deleted. + + ```bash + [test@test logs]$ ll + -rw-r--r-- 1 root root 0 Jan 4 11:18 nebula-graphd.INFO.impl + -rw-r--r-- 1 root root 6894 Jan 4 11:16 nebula-graphd.INFO.impl.2024-01-04-1704338204 # This file is deleted when a new log file is generated + -rw-r--r-- 1 root root 222 Jan 4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338287 + [test@test logs]$ ll + -rw-r--r-- 1 root root 0 Jan 4 11:18 nebula-graphd.INFO.impl + -rw-r--r-- 1 root root 222 Jan 4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338287 + -rw-r--r-- 1 root root 222 Jan 4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338339 # The new log file is generated + ``` + +If you need to rotate multiple log files, create multiple configuration files in the `/etc/logrotate.d` directory, with each configuration file corresponding to a log file. For example, to rotate the `INFO` level log file and the `WARNING` level log file of the Meta service, create two configuration files `nebula-metad.INFO` and `nebula-metad.WARNING`, and add log rotation rules in them respectively. \ No newline at end of file diff --git a/docs-2.0-zh/20.appendix/0.FAQ.md b/docs-2.0-zh/20.appendix/0.FAQ.md index 4110b6eadca..a868090ecd0 100644 --- a/docs-2.0-zh/20.appendix/0.FAQ.md +++ b/docs-2.0-zh/20.appendix/0.FAQ.md @@ -331,12 +331,7 @@ nebula > MATCH (s)<-[e]-() WHERE id(s) == "given" RETURN count(e); #入度 ### 运行日志文件过大时如何回收日志？ -{{nebula.name}} 的运行日志默认在 `/usr/local/nebula/logs/` 下，正常 INFO 级别日志文件为 `nebula-graphd.INFO, nebula-storaged.INFO, nebula-metad.INFO`，报警和错误级别后缀为 `.WARNING` 和 `.ERROR`。 - -{{nebula.name}} 使用 [glog](https://github.com/google/glog) 打印日志。glog 没有日志回收的功能，用户可以： - -- 在三种服务的配置文件内添加参数`timestamp_in_logfile_name=true`（日志文件添加时间戳）和`max_log_size=500`（日志大小限制，单位 MB），然后使用 crontab 设置定期任务回收日志文件。详情请参见 [Glog should delete old log files automatically](https://github.com/google/glog/issues/423)。 -- 使用 [logrotate](https://github.com/logrotate/logrotate) 实现[日志轮询](https://discuss.nebula-graph.com.cn/t/topic/7803)。使用 logrotate 管理日志前需修改相应 {{nebula.name}} 服务的配置，将`timestamp_in_logfile_name`参数的值改成`false`。 +{{nebula.name}}使用 [glog](https://github.com/google/glog) 打印日志，glog 没有日志回收的功能。用户可以通过定时任务或日志管理工具 [logrotate](https://man7.org/linux/man-pages/man8/logrotate.8.html) 来管理运行日志。操作详情，参见[回收日志](../5.configurations-and-logs/2.log-management/logs.md)。 ### 如何查看 {{nebula.name}} 版本 diff --git a/docs-2.0-zh/5.configurations-and-logs/2.log-management/logs.md b/docs-2.0-zh/5.configurations-and-logs/2.log-management/logs.md index 3720fe58ac5..dc04035bd9e 100644 --- a/docs-2.0-zh/5.configurations-and-logs/2.log-management/logs.md +++ b/docs-2.0-zh/5.configurations-and-logs/2.log-management/logs.md @@ -69,7 +69,7 @@ $ curl -X PUT -H "Content-Type: application/json" -d '{"minloglevel":0,"v":3}' " ``` -如果在 {{nebula.name}} 运行时修改了运行日志级别，重启服务后会恢复为配置文件中设置的级别，如果需要永久修改，请修改[配置文件](../1.configurations/1.configurations.md)。 +如果在{{nebula.name}}运行时修改了运行日志级别，重启服务后会恢复为配置文件中设置的级别，如果需要永久修改，请修改[配置文件](../1.configurations/1.configurations.md)并重启服务。 ## RocksDB 运行日志 @@ -77,4 +77,158 @@ RocksDB 的运行日志通常在 `/usr/local/nebula/data/storage/nebula/$id/data ## 回收日志 -运行日志文件过大时如何回收日志，请参见[常见问题](../../20.appendix/0.FAQ.md)。 +glog 本身不支持回收日志，如果需要回收日志，可以使用 Linux 系统中[定时任务（Cron Jobs）](https://man7.org/linux/man-pages/man1/crontab.1.html)来定期删除旧的日志文件。或者，使用日志管理工具 [logrotate](https://github.com/logrotate/logrotate) 来轮转日志以定期归档和删除日志。 + +### 使用定时任务回收日志 + +本文以回收 Graph 服务的运行日志为例，说明如何使用定时任务来定期删除旧的日志文件。操作步骤如下： + +1. 在 [Graph 服务配置文件](../1.configurations/3.graph-config.md)中，进行如下配置，然后重启服务。 + + ```bash + timestamp_in_logfile_name = true + max_log_size = 500 + ``` + + - 设置`timestamp_in_logfile_name`为`true`，这样日志文件名中会包含时间戳，以定期删除旧的日志文件。 + - 添加`max_log_size`参数，设置单个日志文件的最大大小，例如`500`。超过这个大小后，会自动创建新的日志文件，单位 MB，默认值为`1800`。 + +2. 在 Linux 系统中，使用如下命令编辑定时任务： + + ```bash + crontab -e + ``` + +3. 在定时任务中添加命令，以定期删除旧的日志文件。 + + ```bash + * * * * * find -name "" -mtime +7 -delete + ``` + + !!! caution + + 以上命令中的`find`命令需要使用 root 用户或者具有 sudo 权限的用户来执行。 + + - `* * * * *`：定时任务的时间字段，五个星号表示这个任务每分钟都会执行。其他设置，参见[Cron Expression](https://crontab.cronhub.io/)。 + - ``：服务运行日志文件的路径，例如`/usr/local/nebula/logs`。 + - ``：日志文件名，例如`nebula-graphd.*`。 + - `-mtime +7`：表示删除更新时间超过 7 天的日志文件。也可以使用`-mmin +n`，表示删除更新时间超过 n 分钟的日志文件。详情参见 [find 命令](https://man7.org/linux/man-pages/man1/find.1.html)。 + - `-delete`：表示删除满足条件的日志文件。 + + 例如，每天凌晨 3 点自动删除更新时间超过 7 天的 Graph 服务运行日志文件的命令： + + ```bash + 0 3 * * * find /usr/local/nebula/logs -name nebula-graphd.* -mtime +7 -delete + ``` + +4. 保存定时任务。 + + +### 使用 logrotate 回收日志 + +用户可以使用 logrotate 工具对指定的日志文件进行轮转，以达到归档和回收日志的目的。 + +!!! note + + 需要使用 root 用户或者具有 sudo 权限的用户来安装 logrotate 或者运行 logrotate。 + +本文以回收 Graph 服务`INFO`级别的日志文件（`/usr/local/nebula/logs/nebula-graphd.INFO.impl`）为例说明如何使用 logrotate 工具。操作步骤如下： + +1. 在 [Graph 服务配置文件](../1.configurations/3.graph-config.md)中，将`timestamp_in_logfile_name`设置为`false`，以便 logrotate 工具可以识别日志文件名。然后重启服务。 + + ```bash + timestamp_in_logfile_name = false + ``` + +2. 安装 logrotate。 + + - Debian/Ubuntu： + + ```bash + sudo apt-get install logrotate + ``` + + - CentOS/RHEL： + + ```bash + sudo yum install logrotate + ``` + +3. 创建 logrotate 配置文件，添加日志轮转规则，然后保存配置文件。 + + 在`/etc/logrotate.d`目录下，创建一个新的 logrotate 配置文件`nebula-graphd.INFO` + + ```bash + sudo vim /etc/logrotate.d/nebula-graphd.INFO + ``` + + 添加以下内容： + + ```bash + # 需配置日志文件的绝对路径 + # 并且文件名不能为软链接文件，如不能为`nebula-graph.INFO` + /usr/local/nebula/logs/nebula-graphd.INFO.impl { + daily + rotate 2 + copytruncate + nocompress + missingok + notifempty + create 644 root root + dateext + dateformat .%Y-%m-%d-%s + maxsize 1k + } + ``` + + |参数|说明| + |:--|:--| + |`daily`| 每天轮转日志。可用的时间单位有：`hourly`、`daily`、`weekly`、`monthly`、`yearly`。| + |`rotate 2`| 在删除前日志文件前，其被轮转的次数。即保留最近生成的 2 个日志文件。| + |`copytruncate`| 将当前日志文件复制一份，然后清空当前日志文件。| + |`nocompress`| 不压缩旧的日志文件。| + |`missingok`| 如果日志文件丢失，不报告错误。| + |`notifempty`| 如果日志文件为空，不进行轮转。| + |`create 644 root root`| 创建新的日志文件，并设置适当的权限和所有者。| + |`dateext`| 在日志文件名中添加日期后缀。
默认是当前日期。默认是`-%Y%m%d`的后缀。可用`dateformat`选项扩展配置。| + |`dateformat .%Y-%m-%d-%s`| 必须配合`dateext`使用，紧跟在下一行出现，定义文件切割后的文件名。
在V3.9.0 之前，只支持`%Y`、`%m`、`%d`、`%s`参数。在 V3.9.0 及之后，支持 %H 参数。| + |`maxsize 1k`| 当日志文件大小超过`1`千字节（`1024`字节）或者超过设定的周期（如`daily`）时，进行日志轮转。可用的大小单位有：`k`、`M`，默认单位为字节。| + + 用户可以根据实际需求修改配置文件中的参数。更多关于参数的配置及解释，参见 [logrotate](https://man7.org/linux/man-pages/man8/logrotate.8.html)。 + +4. 测试 logrotate 配置。 + + 为了验证 logrotate 的配置是否正确，可以使用以下命令来进行测试： + + ```bash + sudo logrotate --debug /etc/logrotate.d/nebula-graphd.INFO + ``` + +5. 运行 logrotate。 + + 尽管`logrotate`通常由定时作业自动执行，但也可以手动执行以下命令，以立即进行日志轮转： + + ```bash + sudo logrotate -fv /etc/logrotate.d/nebula-graphd.INFO + ``` + + `-fv`：`f`表示强制执行，`v`表示打印详细信息。 + +6. 查看日志轮转结果。 + + 日志轮转后，会在`/usr/local/nebula/logs`目录下看到新的日志文件，例如`nebula-graphd.INFO.impl.2024-01-04-1704338204`。原始日志内容会被清空，但文件会被保留，新日志继续写入。当日志文件数量超过`rotate`设置的值时，会删除最旧的日志文件。 + + 例如，`rotate 2`表示保留最近生成的 2 个日志文件，当日志文件数量超过 2 个时，会删除最旧的日志文件。 + + ```bash + [test@test logs]$ ll + -rw-r--r-- 1 root root 0 Jan 4 11:18 nebula-graphd.INFO.impl + -rw-r--r-- 1 root root 6894 Jan 4 11:16 nebula-graphd.INFO.impl.2024-01-04-1704338204 # 当新的日志文件生成时，此文件被删除 + -rw-r--r-- 1 root root 222 Jan 4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338287 + [test@test logs]$ ll + -rw-r--r-- 1 root root 0 Jan 4 11:18 nebula-graphd.INFO.impl + -rw-r--r-- 1 root root 222 Jan 4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338287 + -rw-r--r-- 1 root root 222 Jan 4 11:18 nebula-graphd.INFO.impl.2024-01-04-1704338339 # 新生成的日志文件 + ``` + +如果用户需要对多个日志文件进行轮转，可以在`/etc/logrotate.d`目录下创建多个配置文件，每个配置文件对应一个日志文件。例如，用户需要对 Meta 服务的`INFO`级别日志文件和`WARNING`级别日志文件进行轮转，可以创建两个配置文件`nebula-metad.INFO`和`nebula-metad.WARNING`，并在其中分别添加日志轮转规则。 \ No newline at end of file