tidb oom when insert 4m big value

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tidb：8core 32GB
insert 4m big value with 256 Concurrency
code：
https://github.com/pingcap/endless/blob/master/pkg/workload/append.go

2. What did you expect to see? (Required)

tidb shoud not oom

3. What did you see instead (Required)

After running for a while, one tidb oom

w := workload.AppendWorkload{
DB: db,
Concurrency: 256,
Tables: 256,
PadLength: 4000000,
}
Try(w.Prepare())
w.Run(gctx)

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v6.2.0-alpha
Edition: Community
Git Commit Hash: 28a8ffc
Git Branch: heads/refs/tags/v6.2.0-alpha
UTC Build Time: 2022-07-23 11:07:09
GoVersion: go1.18.2
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: unistore

[Lily2025]

/type bug
/severity major
/assign XuHuaiyu

[XuHuaiyu]

reproducible script:
main.txt

heap profile:
heap_tidb_172.16.4.7_4040_1658737560.txt

[XuHuaiyu]

1. The dashboard shows that inuse space is about 7GB~13GB.

2. We can only see ~5.6GB inuse memory space from the profile

3. We can see from the alloc_space that GetTextToLog allocates too many small objects, which are GCed thus we can not see from the profile inuse space.

[tiancaiamao]

After analyzing the heap profile, I find the memory usage is expected.

Maybe you can try the non-prepared SQL, it might reduce the memory by more than a half.

[tiancaiamao]

Or a tiny optimization could be, after handleStmtExecute, we clean up the params for the prepared statment. @qw4990

[qw4990]

Or a tiny optimization could be, after handleStmtExecute, we clean up the params for the prepared statment. @qw4990

OK, I'll take a look at this in Sprint10. It should be an enhancement instead of a bug.

[tiancaiamao]

After #33651, the reproduce script fail to run.

[tiancaiamao]

I test this again with tiup nightly...
Since the nightly use general plan cache, the performance should be better
(nightly, not master because master disable it #46504)

I have some new findings.
The actually inuse memory is 6G (<7G) stably, and the RSS is 27G ...
'reserved by Go' takes too much memory

And after I run "set @@global.tidb_enable_gogc_tuner = 0", the memory usage decrease a lot!
From 29G -> 15G, as you see from the below picture.

And the 'reserved-by-go' nearly gone.

So this OOM issue is by the gctunner feature, and we can workaround it, close this issue.