Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

query cluster_slow_query table cause TiDB server oom #33893

Closed
adunhappy opened this issue Apr 12, 2022 · 0 comments · Fixed by #33953 or #38755
Closed

query cluster_slow_query table cause TiDB server oom #33893

adunhappy opened this issue Apr 12, 2022 · 0 comments · Fixed by #33953 or #38755
Assignees
Labels
affects-4.0 This bug affects 4.0.x versions. affects-5.0 This bug affects 5.0.x versions. affects-5.1 This bug affects 5.1.x versions. affects-5.2 This bug affects 5.2.x versions. affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.0 affects-6.1 severity/major sig/diagnosis SIG: Diagnosis sig/execution SIG execution type/bug The issue is confirmed as a bug.
Milestone

Comments

@adunhappy
Copy link

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1.进入dashboard慢查询界面;
2.选择业务高峰期的时间段,自定义时间范围略长一些,点击查询按钮;
3.到grafana中查看tidb-Overview命名空间下tidb组件的Memory Usage面板,发现内存直线上升后降低到0;
4.到以上第3步问题节点的服务器上,执行命令dmesg -T|grep tidb-server,能看到OOM的日志:
[Tue Apr 12 19:06:56 2022] tidb-server invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[Tue Apr 12 19:06:56 2022] tidb-server cpuset=/ mems_allowed=0
[Tue Apr 12 19:06:56 2022] CPU: 0 PID: 10434 Comm: tidb-server Kdump: loaded Not tainted 3.10.0-1127.el7.x86_64 #1
[Tue Apr 12 19:06:56 2022] [10429] 1001 10429 290468 12488 101 0 0 tidb-server
[Tue Apr 12 19:06:56 2022] [17785] 1001 17785 2521970 1790373 4153 0 0 tidb-server
[Tue Apr 12 19:06:56 2022] Out of memory: Kill process 17785 (tidb-server) score 896 or sacrifice child
[Tue Apr 12 19:06:56 2022] Killed process 17785 (tidb-server), UID 1001, total-vm:10087880kB, anon-rss:7161492kB, file-rss:0kB, shmem-rss:0kB
[Tue Apr 12 19:08:55 2022] [10429] 1001 10429 290468 13355 101 0 0 tidb-server
[Tue Apr 12 19:08:55 2022] [17875] 1001 17875 2336392 1782316 4066 0 0 tidb-server
[Tue Apr 12 19:08:55 2022] Out of memory: Kill process 17875 (tidb-server) score 892 or sacrifice child
[Tue Apr 12 19:08:55 2022] Killed process 17875 (tidb-server), UID 1001, total-vm:9345568kB, anon-rss:7129264kB, file-rss:0kB, shmem-rss:0kB

2. What did you expect to see? (Required)

查询慢日志不会发生OOM

3. What did you see instead (Required)

查询慢日志发生OOM

4. What is your TiDB version? (Required)

Release Version: v5.1.2
Edition: Community
Git Commit Hash: dc07933
Git Branch: heads/refs/tags/v5.1.2
UTC Build Time: 2021-09-27 02:52:12
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

@adunhappy adunhappy added the type/bug The issue is confirmed as a bug. label Apr 12, 2022
@seiya-annie seiya-annie added the sig/execution SIG execution label Apr 12, 2022
@crazycs520 crazycs520 added the sig/diagnosis SIG: Diagnosis label Apr 12, 2022
@ti-chi-bot ti-chi-bot added may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.0 labels Apr 13, 2022
@crazycs520 crazycs520 self-assigned this Apr 13, 2022
@crazycs520 crazycs520 added affects-5.2 This bug affects 5.2.x versions. affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.0 and removed may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.0 may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. labels Apr 13, 2022
@crazycs520 crazycs520 changed the title v5.1.2集群通过dashboard或SQL查询慢SQL引发TiDB_Server组件OOM query cluster_slow_query table cause TiDB server oom Apr 14, 2022
@crazycs520 crazycs520 added affects-4.0 This bug affects 4.0.x versions. affects-5.0 This bug affects 5.0.x versions. affects-5.1 This bug affects 5.1.x versions. and removed may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. labels Apr 14, 2022
@tiancaiamao tiancaiamao added this to the v5.4.1 milestone Apr 22, 2022
@crazycs520 crazycs520 reopened this Oct 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-4.0 This bug affects 4.0.x versions. affects-5.0 This bug affects 5.0.x versions. affects-5.1 This bug affects 5.1.x versions. affects-5.2 This bug affects 5.2.x versions. affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.0 affects-6.1 severity/major sig/diagnosis SIG: Diagnosis sig/execution SIG execution type/bug The issue is confirmed as a bug.
Projects
None yet
7 participants