1.5.9 panic: concurrent map read and map write

[tesibelda]

Running kapacitor 1.5.9 in a linux x64 box. We were doing periodic sideload reload. Some instances of kapacitor have restarted at different times with a panic error:

fatal error: concurrent map read and map write
goroutine 7331 [running]:
runtime.throw(0x23ddd87, 0x21)
/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00673da10 sp=0xc00673d9e0 pc=0x439512
runtime.mapaccess2_faststr(0x201af20, 0xc0260176b0, 0xc0381fdd10, 0x42, 0xc004e351d0, 0xc00450c768)
/usr/local/go/src/runtime/map_faststr.go:116 +0x4a5 fp=0xc00673da80 sp=0xc00673da10 pc=0x416065
github.com/influxdata/kapacitor/services/sideload.(*fileSource).Lookup(0xc0006ffc70, 0xc00354e680, 0x2, 0x2, 0xc002d25000, 0x13, 0x0, 0x0)
/root/go/src/github.com/influxdata/kapacitor/services/sideload/service.go:241 +0xfc fp=0xc00673dae8 sp=0xc00673da80 pc=0x1c5141c
github.com/influxdata/kapacitor.(*SideloadNode).doSideload(0xc0012271e0, 0x7f0b241ab5d8, 0xc02cecb7a0)
/root/go/src/github.com/influxdata/kapacitor/sideload.go:131 +0x71d fp=0xc00673dd28 sp=0xc00673dae8 pc=0x1ca019d
github.com/influxdata/kapacitor.(*SideloadNode).Point(0xc0012271e0, 0x26dbf20, 0xc02cecb7a0, 0xc00673ddb8, 0x1, 0x40c0ff, 0xc00a56a000)
/root/go/src/github.com/influxdata/kapacitor/sideload.go:184 +0x7e fp=0xc00673dd60 sp=0xc00673dd28 pc=0x1ca0ade
github.com/influxdata/kapacitor/edge.(*timedForwardReceiver).Point(0xc0029c1b00, 0x26dbf20, 0xc02cecb5f0, 0x2234c80, 0x22c3640, 0xc02cecb501, 0x26dbf20)
/root/go/src/github.com/influxdata/kapacitor/edge/timed.go:62 +0x62 fp=0xc00673ddc8 sp=0xc00673dd60 pc=0xce9962

Looks like a race condition while reloading sideload data and at the same time processing metrics that use this data. Thanks.

[tesibelda]

These instances were previously in version 1.5.3 with the same configuration and we never saw this panic. We have minimized the reload calls while this gets fixed. Any ideas?

[docmerlin]

Thanks, yah, looks like the race condition has existed for a while, but the newer version of Go makes it panic instead of race.