fatal error: concurrent map writes on kuma 1.2.3

[jgrund]

Summary

I am seeing the following error fairly frequently when many dataplanes report in for the first time:

fatal error: concurrent map writes

Including the dump from when this occurs:

https://gist.githubusercontent.com/jgrund/18413f8209173316b375b91753e401c7/raw/042635c17995736feb7d2088379e4e5bce32abd7/kuma-1.2.3-dump

Additional Details & Logs

• Version: kuma 1.2.3
• Configuration: Universal
• Platform and Operating System: Centos 7.9

[jpeach]

Aug 14 17:20:36 node1 kuma-cp[17624]: fatal error: concurrent map writes
Aug 14 17:20:36 node1 kuma-cp[17624]: goroutine 87718 [running]:
Aug 14 17:20:36 node1 kuma-cp[17624]: runtime.throw(0x27813f1, 0x15)
Aug 14 17:20:36 node1 kuma-cp[17624]: /usr/lib/golang/src/runtime/panic.go:1117 +0x72 fp=0xc003f09420 sp=0xc003f093f0 pc=0x4384f2
Aug 14 17:20:36 node1 kuma-cp[17624]: runtime.mapassign(0x2296de0, 0xc000cb02a0, 0xc003f09508, 0x0)
Aug 14 17:20:36 node1 kuma-cp[17624]: /usr/lib/golang/src/runtime/map.go:585 +0x5aa fp=0xc003f094a0 sp=0xc003f09420 pc=0x41006a
Aug 14 17:20:36 node1 kuma-cp[17624]: github.com/kumahq/kuma/pkg/xds/server/callbacks.(*DataplaneLifecycle).register(0xc0009fe1c0, 0x2a, 0xc001f0f200, 0x7, 0xc001f0f208, 0x32, 0x0, 0x0, 0xc001c74460, 0x91, ...)
Aug 14 17:20:36 node1 kuma-cp[17624]: /__w/pkg-kuma/pkg-kuma/rpm/BUILD/kuma-1.2.3/pkg/xds/server/callbacks/dataplane_lifecycle.go:53 +0x2f0 fp=0xc003f09538 sp=0xc003f094a0 pc=0x1fe10d0
Aug 14 17:20:36 node1 kuma-cp[17624]: github.com/kumahq/kuma/pkg/xds/server/callbacks.(*DataplaneLifecycle).OnProxyConnected(0xc0009fe1c0, 0x2a, 0xc001f0f200, 0x7, 0xc001f0f208, 0x32, 0x33db278, 0xc000ff96b0, 0x0, 0x0, ...)
Aug 14 17:20:36 node1 kuma-cp[17624]: /__w/pkg-kuma/pkg-kuma/rpm/BUILD/kuma-1.2.3/pkg/xds/server/callbacks/dataplane_lifecycle.go:39 +0xa5 fp=0xc003f095e8 sp=0xc003f09538 pc=0x1fe0cc5
Aug 14 17:20:36 node1 kuma-cp[17624]: github.com/kumahq/kuma/pkg/xds/server/callbacks.(*xdsCallbacks).OnStreamRequest(0xc0009fe200, 0x2a, 0x33f0b58, 0xc0006297c0, 0x0, 0x0)
Aug 14 17:20:36 node1 kuma-cp[17624]: /__w/pkg-kuma/pkg-kuma/rpm/BUILD/kuma-1.2.3/pkg/xds/server/callbacks/dataplane_callbacks.go:118 +0x475 fp=0xc003f097c8 sp=0xc003f095e8 pc=0x1fe0775
Aug 14 17:20:36 node1 kuma-cp[17624]: github.com/kumahq/kuma/pkg/util/xds/v3.(*adapterCallbacks).OnStreamRequest(0xc000688900, 0x2a, 0xc0031e7f00, 0x0, 0x0)