From 7e070f2eff5308e10351d653845391c5251a066e Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Wed, 14 Aug 2024 16:28:38 +0800 Subject: [PATCH] Update the log redaction documentation (#18487) --- log-redaction.md | 40 +++++++++++++++++++++++++------- pd-configuration-file.md | 3 ++- tiflash/tiflash-configuration.md | 17 ++++++++------ tikv-configuration-file.md | 6 ++++- 4 files changed, 48 insertions(+), 18 deletions(-) diff --git a/log-redaction.md b/log-redaction.md index 04c8ff7965de9..61108e93af67a 100644 --- a/log-redaction.md +++ b/log-redaction.md @@ -9,14 +9,12 @@ When TiDB provides detailed log information, it might print sensitive data (for ## Log redaction in TiDB side -To enable log redaction in the TiDB side, set the value of [`global.tidb_redact_log`](/system-variables.md#tidb_redact_log) to `1`. This configuration value defaults to `0`, which means that log redaction is disabled. +To enable log redaction in the TiDB side, set the value of [`global.tidb_redact_log`](/system-variables.md#tidb_redact_log) to `ON` or `MARKER`. This configuration value defaults to `OFF`, which means that log redaction is disabled. You can use the `set` syntax to set the global variable `tidb_redact_log`: -{{< copyable "sql" >}} - ```sql -set @@global.tidb_redact_log=1; +set @@global.tidb_redact_log = ON; ``` After the setting, all logs generated in new sessions are redacted: @@ -32,19 +30,43 @@ ERROR 1062 (23000): Duplicate entry '1' for key 't.a' The error log for the `INSERT` statement above is printed as follows: ``` -[2020/10/20 11:45:49.539 +08:00] [INFO] [conn.go:800] ["command dispatched failed"] [conn=5] [connInfo="id:5, addr:127.0.0.1:57222 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="insert into t values ( ? ) , ( ? )"] [txn_mode=OPTIMISTIC] [err="[kv:1062]Duplicate entry '?' for key 't.a'"] +[2024/07/02 11:35:32.686 +08:00] [INFO] [conn.go:1146] ["command dispatched failed"] [conn=1482686470] [session_alias=] [connInfo="id:1482686470, addr:127.0.0.1:52258 status:10, collation:utf8mb4_0900_ai_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="insert into `t` values ( ... )"] [txn_mode=PESSIMISTIC] [timestamp=450859193514065921] [err="[kv:1062]Duplicate entry '?' for key 't.a'"] +``` + +From the preceding error log, you can see that when the value of `tidb_redact_log` is set to `ON`, sensitive information is replaced by the `?` mark in the TiDB log to avoid data security risks. + +In addition, TiDB provides the `MARKER` option. When the value of `tidb_redact_log` is set to `MARKER`, TiDB marks sensitive information in the log with `‹›` instead of replacing it directly, so you can customize the redaction rules. + +```sql +set @@global.tidb_redact_log = MARKER; +``` + +After the preceding configuration, the sensitive information is marked rather than replaced in all logs generated by new sessions: + +```sql +create table t (a int, unique key (a)); +Query OK, 0 rows affected (0.07 sec) + +insert into t values (1),(1); +ERROR 1062 (23000): Duplicate entry '‹1›' for key 't.a' +``` + +The error log is as follows: + +``` +[2024/07/02 11:35:01.426 +08:00] [INFO] [conn.go:1146] ["command dispatched failed"] [conn=1482686470] [session_alias=] [connInfo="id:1482686470, addr:127.0.0.1:52258 status:10, collation:utf8mb4_0900_ai_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="insert into `t` values ( ‹1› ) , ( ‹1› )"] [txn_mode=PESSIMISTIC] [timestamp=450859185309483010] [err="[kv:1062]Duplicate entry '‹1›' for key 't.a'"] ``` -From the error log above, you can see that all sensitive information is shielded using `?` after `tidb_redact_log` is enabled. In this way, data security risks are avoided. +As you can see from the preceding error log, after you set `tidb_redact_log` to `MARKER`, TiDB marks sensitive information using `‹ ›` in the log. You can customize redaction rules to handle sensitive information in the log as needed. ## Log redaction in TiKV side -To enable log redaction in the TiKV side, set the value of [`security.redact-info-log`](/tikv-configuration-file.md#redact-info-log-new-in-v408) to `true`. This configuration value defaults to `false`, which means that log redaction is disabled. +To enable log redaction in the TiKV side, set the value of [`security.redact-info-log`](/tikv-configuration-file.md#redact-info-log-new-in-v408) to `true` or `"marker"`. This configuration value defaults to `false`, which means that log redaction is disabled. ## Log redaction in PD side -To enable log redaction in the PD side, set the value of [`security.redact-info-log`](/pd-configuration-file.md#redact-info-log-new-in-v50) to `true`. This configuration value defaults to `false`, which means that log redaction is disabled. +To enable log redaction in the PD side, set the value of [`security.redact-info-log`](/pd-configuration-file.md#redact-info-log-new-in-v50) to `true` or `"marker"`. This configuration value defaults to `false`, which means that log redaction is disabled. ## Log redaction in TiFlash side -To enable log redaction in the TiFlash side, set both the [`security.redact_info_log`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) value in tiflash-server and the [`security.redact-info-log`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) value in tiflash-learner to `true`. Both configuration values default to `false`, which means that log redaction is disabled. +To enable log redaction in the TiFlash side, set both the [`security.redact_info_log`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) value in tiflash-server and the [`security.redact-info-log`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) value in tiflash-learner to `true` or `"marker"`. Both configuration values default to `false`, which means that log redaction is disabled. diff --git a/pd-configuration-file.md b/pd-configuration-file.md index f727ce33ba9b5..eebe1e5420430 100644 --- a/pd-configuration-file.md +++ b/pd-configuration-file.md @@ -198,8 +198,9 @@ Configuration items related to security ### `redact-info-log` New in v5.0 + Controls whether to enable log redaction in the PD log -+ When you set the configuration value to `true`, user data is redacted in the PD log. ++ Optional value: `false`, `true`, `"marker"` + Default value: `false` ++ For details on how to use it, see [Log redaction in PD side](/log-redaction.md#log-redaction-in-pd-side). ## `log` diff --git a/tiflash/tiflash-configuration.md b/tiflash/tiflash-configuration.md index e8c383b6e70d3..78f036d1d2d63 100644 --- a/tiflash/tiflash-configuration.md +++ b/tiflash/tiflash-configuration.md @@ -256,10 +256,11 @@ delta_index_cache_size = 0 ## Security settings take effect starting from v4.0.5. [security] - ## New in v5.0. This configuration item enables or disables log redaction. Value options: true, false, marker. The marker option is introduced in v8.2.0. - ## The default value is false, which means that log redaction is disabled. - ## If the configuration item is set to true, all user data in the log is replaced by `?`. - ## If the configuration item is set to marker, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed. + ## New in v5.0. This configuration item enables or disables log redaction. Value options: `true`, `false`, `"on"`, `"off"`, and `"marker"`. The `"on"`, `"off"`, and `"marker"` options are introduced in v8.2.0. + ## If the configuration item is set to `false` or `"off"`, log redaction is disabled. + ## If the configuration item is set to `true` or `"on"`, all user data in the log is replaced by `?`. + ## If the configuration item is set to `"marker"`, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed. + ## The default value is `false`. ## Note that you also need to set security.redact-info-log for tiflash-learner's logging in tiflash-learner.toml. # redact_info_log = false @@ -307,9 +308,11 @@ The parameters in `tiflash-learner.toml` are basically the same as those in TiKV snap-handle-pool-size = 2 [security] - ## New in v5.0. This configuration item enables or disables log redaction. Value options: true, false. - ## The default value is false, which means that log redaction is disabled. - ## If the configuration item is set to true, all user data in the log is replaced by `?`. + ## New in v5.0. This configuration item enables or disables log redaction. Value options: `true`, `false`, `"on"`, `"off"`, and `"marker"`. The `"on"`, `"off"`, and `"marker"` options are introduced in v8.3.0. + ## If the configuration item is set to `false` or `"off"`, log redaction is disabled. + ## If the configuration item is set to `true` or `"on"`, all user data in the log is replaced by `?`. + ## If the configuration item is set to `"marker"`, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed. + ## The default value is `false`. redact-info-log = false [security.encryption] diff --git a/tikv-configuration-file.md b/tikv-configuration-file.md index 8c96e8528b37c..f07fdd1ca6c08 100644 --- a/tikv-configuration-file.md +++ b/tikv-configuration-file.md @@ -2041,8 +2041,12 @@ Configuration items related to security. ### `redact-info-log` New in v4.0.8 -+ This configuration item enables or disables log redaction. If the configuration value is set to `true`, all user data in the log will be replaced by `?`. ++ This configuration item enables or disables log redaction. Value options: `true`, `false`, `"on"`, `"off"`, and `"marker"`. The `"on"`, `"off"`, and `"marker"` options are introduced in v8.3.0. ++ If the configuration item is set to `false` or `"off"`, log redaction is disabled. ++ If the configuration item is set to `true` or `"on"`, all user data in the log is replaced by `?`. ++ If the configuration item is set to `"marker"`, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed. + Default value: `false` ++ For details on how to use it, see [Log redaction in TiKV side](/log-redaction.md#log-redaction-in-tikv-side). ## security.encryption