executor: trace the memory usage of Projection executors

[qw4990]

What problem does this PR solve?

Trace the memory usage of Projection executors and display the result in Explain Analyze.

Check List

Tests

• Unit test

[qw4990]

/rebuild

[codecov]

Codecov Report

Merging #13914 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #13914   +/-   ##
===========================================
  Coverage   80.5325%   80.5325%           
===========================================
  Files           483        483           
  Lines        123288     123288           
===========================================
  Hits          99287      99287           
  Misses        16245      16245           
  Partials       7756       7756

[qw4990]

/run-all-tests

[qw4990]

/run-integration-common-test

[ichn-hu]

Would you please add a test that shows after the execution the memTracker will be 0? Just like

tidb/distsql/distsql_test.go

Line 126 in 0a22034

c.Assert(response.memTracker.BytesConsumed(), Equals, int64(0))

Rest LGTM.

[qw4990]

Would you please add a test that shows after the execution the memTracker will be 0? Just like

tidb/distsql/distsql_test.go

Line 126 in 0a22034

c.Assert(response.memTracker.BytesConsumed(), Equals, int64(0))

Rest LGTM.

updated, PTAL~

[qw4990]

Would you please add a test that shows after the execution the memTracker will be 0? Just like

tidb/distsql/distsql_test.go

Line 126 in 0a22034

c.Assert(response.memTracker.BytesConsumed(), Equals, int64(0))

Rest LGTM.

updated, PTAL~

After testing, I find it's hard to guarantee it is zero since when the main goroutine exit, other worker-goroutines may be still running and chunks hold by them are hard to trace then.

[SunRunAway]

Would you please add a test that shows after the execution the memTracker will be 0? Just like

tidb/distsql/distsql_test.go

Line 126 in 0a22034

c.Assert(response.memTracker.BytesConsumed(), Equals, int64(0))

Rest LGTM.

updated, PTAL~

After testing, I find it's hard to guarantee it is zero since when the main goroutine exit, other worker-goroutines may be still running and chunks hold by them are hard to trace then.

Could you explain it with more detail?
It's better to keep zero goroutines running after executor.Close is called.

[qw4990]

Would you please add a test that shows after the execution the memTracker will be 0? Just like

tidb/distsql/distsql_test.go

Line 126 in 0a22034

c.Assert(response.memTracker.BytesConsumed(), Equals, int64(0))

Rest LGTM.

updated, PTAL~

After testing, I find it's hard to guarantee it is zero since when the main goroutine exit, other worker-goroutines may be still running and chunks hold by them are hard to trace then.

Could you explain it with more detail?
It's better to keep zero goroutines running after executor.Close is called.

Projection uses some channels to transfer chunks between its internal goroutines.
When closing it, we should close these channels and drain them to trace all chunks in them, but there may be some other goroutines are still reading or writing these channels because they haven't received the exit signal yet, which causes DATA RACE on these channels (the main goroutine want to close them while others are reading/writing them).

We can use WaitGroup to sync the exit signal and let the main goroutine close these channels until all worker-goroutines exit. But is it worth that?

[qw4990]

/run-all-tests

[qw4990]

All comments are addressed, PTAL @SunRunAway @ichn-hu

[ichn-hu]

LGTM

[SunRunAway]

LGTM

[SunRunAway]

/merge

[sre-bot]

/run-all-tests

[sre-bot]

@qw4990 merge failed.

[qw4990]

/run-all-tests