Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tidb crash when querying information_schema: fatal error: concurrent map writes #33385

Closed
WolfeeTJ opened this issue Mar 24, 2022 · 6 comments
Closed

tidb crash when querying information_schema: fatal error: concurrent map writes #33385

WolfeeTJ opened this issue Mar 24, 2022 · 6 comments
Assignees
@tangenta
Labels
affects-5.2 This bug affects 5.2.x versions. affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 may-affects-4.0 This bug maybe affects 4.0.x versions. severity/major type/bug The issue is confirmed as a bug.

Comments

@WolfeeTJ
Copy link

WolfeeTJ commented Mar 24, 2022 •
edited
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

you could easily reproduce and see tidb crashes when run the below query many times in a loop:

select src.column_name,src.column_type from (select column_name,column_type from INFORMATION_SCHEMA.columns where table_schema = 'mysql' and table_name = 'user' ) src left join (select column_name,column_type from INFORMATION_SCHEMA.columns where table_schema = 'mysql' and table_name = 'user') tar on src.column_name = tar.column_name where tar.column_name is null;

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

Release Version: v5.4.0 Edition: Community Git Commit Hash: 55f3b24 Git Branch: heads/refs/tags/v5.4.0 UTC Build Time: 2022-01-25 08:39:26 GoVersion: go1.16.4 Race Enabled: false TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306 Check Table Before Drop: false

fatal error: concurrent map writes

goroutine 84783511 [running]:
runtime.throw(0x3fb0f58, 0x15)

log file uploaded:
tidb_stderr.202203240819.log.zip
@WolfeeTJ WolfeeTJ added the type/bug The issue is confirmed as a bug. label Mar 24, 2022
@WolfeeTJ WolfeeTJ changed the title tidb: fatal error: concurrent map writes tidb crash when querying information_schema: fatal error: concurrent map writes Mar 24, 2022
@seiya-annie
Copy link

@WolfeeTJ could you please add reproduce steps for this issue?

@ti-chi-bot ti-chi-bot added may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.0 labels Mar 24, 2022
@WolfeeTJ
Copy link
Author

@WolfeeTJ could you please add reproduce steps for this issue?

please kindly try this:
run the below sql in loops, and you could easily see tidb crashes.

select
src.column_name
, src.column_type
from (select column_name, column_type
from INFORMATION_SCHEMA.columns
where table_schema = 'mysql' and table_name = 'user') src
left join (select column_name, column_type
from INFORMATION_SCHEMA.columns
where table_schema = 'mysql' and table_name = 'user') tar
on src.column_name = tar.column_name
where tar.column_name is null ;

@WolfeeTJ
Copy link
Author

updated the original post. not sure why github didn't show my previsous edit but only in the draft. but nevermind.

@wshwsh12
Copy link
Contributor

wshwsh12 commented Mar 28, 2022 •
edited
Loading

I can reproduce this issue.
We need create any view first, and then run the sql above...

error stack:

fatal error: concurrent map writes

goroutine 595 [running]:
runtime.throw({0x3860407?, 0x5b05c28?})
        /home/wshwsh12/go/go1.18/src/runtime/panic.go:992 +0x71 fp=0xc0109cf630 sp=0xc0109cf600 pc=0x129d811
runtime.mapassign_fast64ptr(0x3df17f8?, 0xc00fe0cfc0?, 0xc000ed6900)
        /home/wshwsh12/go/go1.18/src/runtime/map_fast64.go:192 +0x2dc fp=0xc0109cf668 sp=0xc0109cf630 pc=0x127971c
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildDataSource(0xc010828680, {0x3d9e680, 0xc0107e84b0}, 0xc0106b9d40, 0xc01090b080)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:4265 +0x2b45 fp=0xc0109cffe0 sp=0xc0109cf668 pc=0x26aa0c5
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildResultSetNode(0xc010828680, {0x3d9e680, 0xc0107e84b0}, {0x3daff10?, 0xc01090b030?})
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:348 +0x196 fp=0xc0109d0278 sp=0xc0109cffe0 pc=0x2683e96
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildJoin(0x0?, {0x3d9e680?, 0xc0107e84b0?}, 0x0?)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:669 +0x71a fp=0xc0109d0730 sp=0xc0109d0278 pc=0x268961a
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildResultSetNode(0x0?, {0x3d9e680?, 0xc0107e84b0?}, {0x3daf360?, 0xc0107dafc0?})
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:335 +0x2d1 fp=0xc0109d09c8 sp=0xc0109d0730 pc=0x2683fd1
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildTableRefs(0xc010828680?, {0x3d9e680?, 0xc0107e84b0?}, 0x0?)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:329 +0x85 fp=0xc0109d0a40 sp=0xc0109d09c8 pc=0x2683be5
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildSelect(0xc010828680, {0x3d9e680, 0xc0107e84b0}, 0xc01024db00)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:3580 +0x55b fp=0xc0109d0ec0 sp=0xc0109d0a40 pc=0x26a3a5b
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildIntersect(0x0?, {0x3d9e680, 0xc0107e84b0}, {0xc0109d1230, 0x1, 0x127196d?})
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:1561 +0xac fp=0xc0109d1020 sp=0xc0109d0ec0 pc=0x269272c
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildSetOpr(0xc010828680, {0x3d9e680, 0xc0107e84b0}, 0xc01090b110)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:1475 +0x2cb fp=0xc0109d1270 sp=0xc0109d1020 pc=0x269130b
github.com/pingcap/tidb/planner/core.(*PlanBuilder).Build(0xc010828680, {0x3d9e680, 0xc0107e84b0}, {0x3dab8c8?, 0xc01090b110?})
        /home/wshwsh12/project/tidb/planner/core/planbuilder.go:701 +0x91c fp=0xc0109d12d8 sp=0xc0109d1270 pc=0x26f43dc
github.com/pingcap/tidb/planner/core.(*PlanBuilder).BuildDataSourceFromView(0xc010828680, {0x3d9e680, 0xc0107e84b0}, {{0xc000e018c8?, 0xc0009c9420?}, {0xc000e018cc?, 0xc0009c9420?}}, 0xc000ed6a80)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:4545 +0x2c5 fp=0xc0109d1550 sp=0xc0109d12d8 pc=0x26ad825
github.com/pingcap/tidb/executor.tryFillViewColumnType({0x3d9e680, 0xc0107e84b0}, {0x3df17f8, 0xc00fe0cfc0}, {0x3dbd590, 0xc0009c9420}, {{0xc000e018c8, 0x4}, {0xc000e018cc, 0x4}}, ...)
        /home/wshwsh12/project/tidb/executor/show.go:1902 +0x235 fp=0xc0109d1628 sp=0xc0109d1550 pc=0x2ed4675
github.com/pingcap/tidb/executor.(*hugeMemTableRetriever).dataForColumnsInTable(0xc0107fd570, {0x3d9e680, 0xc0107e84b0}, {0x3df17f8?, 0xc00fe0cfc0?}, 0xc0009c8770, 0xc000ed6a80, 0x4?, 0xc0107ec230)
        /home/wshwsh12/project/tidb/executor/infoschema_reader.go:680 +0x108 fp=0xc0109d1a28 sp=0xc0109d1628 pc=0x2e39de8
github.com/pingcap/tidb/executor.(*hugeMemTableRetriever).setDataForColumns(0xc0107fd570, {0x3d9e680, 0xc0107e84b0}, {0x3df17f8, 0xc00fe0cfc0?}, 0x12f41b7?)
        /home/wshwsh12/project/tidb/executor/infoschema_reader.go:669 +0x1bd fp=0xc0109d1af0 sp=0xc0109d1a28 pc=0x2e39b3d
github.com/pingcap/tidb/executor.(*hugeMemTableRetriever).retrieve(0xc0107fd570, {0x3d9e680?, 0xc0107e84b0?}, {0x3df17f8?, 0xc00fe0cfc0?})
        /home/wshwsh12/project/tidb/executor/infoschema_reader.go:2547 +0x1bf fp=0xc0109d1bb8 sp=0xc0109d1af0 pc=0x2e4c33f
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next(0xc0107dad80, {0x3d9e680, 0xc0107e84b0}, 0xc0107c05a0)
        /home/wshwsh12/project/tidb/executor/memtable_reader.go:118 +0x29b fp=0xc0109d1d20 sp=0xc0109d1bb8 pc=0x2e9133b
github.com/pingcap/tidb/executor.Next({0x3d9e680, 0xc0107e84b0}, {0x3da0c40, 0xc0107dad80}, 0xc0107c05a0)
        /home/wshwsh12/project/tidb/executor/executor.go:302 +0x496 fp=0xc0109d1e60 sp=0xc0109d1d20 pc=0x2e04016
github.com/pingcap/tidb/executor.(*HashJoinExec).fetchProbeSideChunks(0xc0107f0d80, {0x3d9e680, 0xc0107e84b0})
        /home/wshwsh12/project/tidb/executor/join.go:214 +0x1d6 fp=0xc0109d1f28 sp=0xc0109d1e60 pc=0x2e75816
github.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndProbeHashTable.func1()
        /home/wshwsh12/project/tidb/executor/join.go:338 +0x8f fp=0xc0109d1f90 sp=0xc0109d1f28 pc=0x2e76acf
github.com/pingcap/tidb/util.WithRecovery(0x25c8d26?, 0xc000166500?)
        /home/wshwsh12/project/tidb/util/misc.go:100 +0x53 fp=0xc0109d1fc0 sp=0xc0109d1f90 pc=0x1ee2533
github.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndProbeHashTable.func3()
        /home/wshwsh12/project/tidb/executor/join.go:336 +0x2a fp=0xc0109d1fe0 sp=0xc0109d1fc0 pc=0x2e76a0a
runtime.goexit()
        /home/wshwsh12/go/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0109d1fe8 sp=0xc0109d1fe0 pc=0x12d1761
created by github.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndProbeHashTable
        /home/wshwsh12/project/tidb/executor/join.go:336 +0x14f

Related stack:

goroutine 565 [runnable]:
github.com/pingcap/tidb/planner/core.appendVisitInfo(...)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:6341
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildDataSource(0xc0109c8000, {0x3d9e680, 0xc0107e84b0}, 0xc0108304e0, 0xc0106d6a60)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:3998 +0x4ce
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildResultSetNode(0xc0109c8000, {0x3d9e680, 0xc0107e84b0}, {0x3daff10?, 0xc0106d6a10?})
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:348 +0x196
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildJoin(0x0?, {0x3d9e680?, 0xc0107e84b0?}, 0x0?)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:669 +0x71a
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildResultSetNode(0x0?, {0x3d9e680?, 0xc0107e84b0?}, {0x3daf360?, 0xc01082bcb0?})
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:335 +0x2d1
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildTableRefs(0xc0109c8000?, {0x3d9e680?, 0xc0107e84b0?}, 0x3d9e680?)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:329 +0x85
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildSelect(0xc0109c8000, {0x3d9e680, 0xc0107e84b0}, 0xc0106ce360)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:3580 +0x55b
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildIntersect(0x0?, {0x3d9e680, 0xc0107e84b0}, {0xc0109d5040, 0x1, 0x127196d?})
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:1561 +0xac
github.com/pingcap/tidb/planner/core.(*PlanBuilder).buildSetOpr(0xc0109c8000, {0x3d9e680, 0xc0107e84b0}, 0xc0106d6a80)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:1475 +0x2cb
github.com/pingcap/tidb/planner/core.(*PlanBuilder).Build(0xc0109c8000, {0x3d9e680, 0xc0107e84b0}, {0x3dab8c8?, 0xc0106d6a80?})
        /home/wshwsh12/project/tidb/planner/core/planbuilder.go:701 +0x91c
github.com/pingcap/tidb/planner/core.(*PlanBuilder).BuildDataSourceFromView(0xc0109c8000, {0x3d9e680, 0xc0107e84b0}, {{0xc000e018c8?, 0xc0009c9420?}, {0xc000e018cc?, 0xc0009c9420?}}, 0xc000ed6a80)
        /home/wshwsh12/project/tidb/planner/core/logical_plan_builder.go:4545 +0x2c5
github.com/pingcap/tidb/executor.tryFillViewColumnType({0x3d9e680, 0xc0107e84b0}, {0x3df17f8, 0xc00fe0cfc0}, {0x3dbd590, 0xc0009c9420}, {{0xc000e018c8, 0x4}, {0xc000e018cc, 0x4}}, ...)
        /home/wshwsh12/project/tidb/executor/show.go:1902 +0x235
github.com/pingcap/tidb/executor.(*hugeMemTableRetriever).dataForColumnsInTable(0xc0107fd5e0, {0x3d9e680, 0xc0107e84b0}, {0x3df17f8?, 0xc00fe0cfc0?}, 0xc0009c8770, 0xc000ed6a80, 0x4?, 0xc0107ec380)
        /home/wshwsh12/project/tidb/executor/infoschema_reader.go:680 +0x108
github.com/pingcap/tidb/executor.(*hugeMemTableRetriever).setDataForColumns(0xc0107fd5e0, {0x3d9e680, 0xc0107e84b0}, {0x3df17f8, 0xc00fe0cfc0?}, 0xc0105ad430?)
        /home/wshwsh12/project/tidb/executor/infoschema_reader.go:669 +0x1bd
github.com/pingcap/tidb/executor.(*hugeMemTableRetriever).retrieve(0xc0107fd5e0, {0x3d9e680?, 0xc0107e84b0?}, {0x3df17f8?, 0xc00fe0cfc0?})
        /home/wshwsh12/project/tidb/executor/infoschema_reader.go:2547 +0x1bf
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next(0xc0107dae10, {0x3d9e680, 0xc0107e84b0}, 0xc0105b3db0)
        /home/wshwsh12/project/tidb/executor/memtable_reader.go:118 +0x29b
github.com/pingcap/tidb/executor.Next({0x3d9e680, 0xc0107e84b0}, {0x3da0c40, 0xc0107dae10}, 0xc0105b3db0)
        /home/wshwsh12/project/tidb/executor/executor.go:302 +0x496
github.com/pingcap/tidb/executor.(*SelectionExec).Next(0xc0106b9ba0, {0x3d9e680, 0xc0107e84b0}, 0xc0106d2000)
        /home/wshwsh12/project/tidb/executor/executor.go:1394 +0xef
github.com/pingcap/tidb/executor.Next({0x3d9e680, 0xc0107e84b0}, {0x3da10c0, 0xc0106b9ba0}, 0xc0106d2000)
        /home/wshwsh12/project/tidb/executor/executor.go:302 +0x496
github.com/pingcap/tidb/executor.(*HashJoinExec).fetchBuildSideRows(0xc0107f0d80, {0x3d9e680, 0xc0107e84b0}, 0xc0106ca0c0, 0xc000ee7620)
        /home/wshwsh12/project/tidb/executor/join.go:282 +0x1a5
github.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndBuildHashTable.func2()
        /home/wshwsh12/project/tidb/executor/join.go:731 +0xab
github.com/pingcap/tidb/util.WithRecovery(0x3d9e680?, 0xc0105807e0?)
        /home/wshwsh12/project/tidb/util/misc.go:100 +0x53
created by github.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndBuildHashTable
        /home/wshwsh12/project/tidb/executor/join.go:728 +0x22d

@Defined2014
Copy link
Contributor

Same as #32459

@jebter jebter added affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 labels Apr 2, 2022
@ti-chi-bot ti-chi-bot removed may-affects-6.0 may-affects-5.4 This bug maybe affects 5.4.x versions. labels Apr 2, 2022
@tangenta
Copy link
Contributor

Closed by #33946.

@tangenta tangenta self-assigned this Apr 26, 2022
@tangenta tangenta added affects-5.2 This bug affects 5.2.x versions. affects-5.3 This bug affects 5.3.x versions. and removed may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. labels Feb 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tangenta tangenta

Labels
affects-5.2 This bug affects 5.2.x versions. affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 may-affects-4.0 This bug maybe affects 4.0.x versions. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jebter @wshwsh12 @Defined2014 @WolfeeTJ @tangenta @seiya-annie @ti-chi-bot