diff --git a/configure-memory-usage.md b/configure-memory-usage.md index 65c45fbd89d70..9de3d6e469a29 100644 --- a/configure-memory-usage.md +++ b/configure-memory-usage.md @@ -228,4 +228,24 @@ The following example uses a memory-consuming SQL statement to demonstrate the d | └─TableFullScan_16 | 256.00 | 256 | cop[tikv] | table:t1 | tikv_task:{time:71.4µs, loops:256} | keep order:false, stats:pseudo | N/A | N/A | +---------------------------------+-------------+----------+-----------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------+-----------+----------+ 9 rows in set (1 min 37.428 sec) - ``` \ No newline at end of file + ``` + +## Others + +### Mitigate OOM issues by configuring `GOMEMLIMIT` + +GO 1.19 introduces an environment variable [`GOMEMLIMIT`](https://pkg.go.dev/runtime@go1.19#hdr-Environment_Variables) to set the memory limit that triggers GC. + +For v6.1.3 <= TiDB < v6.5.0, you can mitigate a typical category of OOM issues by manually setting `GOMEMLIMIT`. The typical category of OOM issues is: before OOM occurs, the estimated memory in use on Grafana occupies only half of the entire memory (TiDB-Runtime > Memory Usage > estimate-inuse), as shown in the following figure: + +![normal OOM case example](/media/configure-memory-usage-oom-example.png) + +To verify the performance of `GOMEMLIMIT`, a test is performed to compare the specific memory usage with and without `GOMEMLIMIT` configuration. + +- In TiDB v6.1.2, the TiDB server encounters OOM (system memory: about 48 GiB) after the simulated workload runs for several minutes: + + ![v6.1.2 workload oom](/media/configure-memory-usage-612-oom.png) + +- In TiDB v6.1.3, `GOMEMLIMIT` is set to 40000 MiB. It is found that the simulated workload runs stably for a long time, OOM does not occur in the TiDB server, and the maximum memory usage of the process is stable at around 40.8 GiB: + + ![v6.1.3 workload no oom with GOMEMLIMIT](/media/configure-memory-usage-613-no-oom.png) diff --git a/media/configure-memory-usage-612-oom.png b/media/configure-memory-usage-612-oom.png new file mode 100644 index 0000000000000..107d2ed613c2b Binary files /dev/null and b/media/configure-memory-usage-612-oom.png differ diff --git a/media/configure-memory-usage-613-no-oom.png b/media/configure-memory-usage-613-no-oom.png new file mode 100644 index 0000000000000..0e8f15504900b Binary files /dev/null and b/media/configure-memory-usage-613-no-oom.png differ diff --git a/media/configure-memory-usage-oom-example.png b/media/configure-memory-usage-oom-example.png new file mode 100644 index 0000000000000..3300ee502a4fc Binary files /dev/null and b/media/configure-memory-usage-oom-example.png differ