Use RWLock in prometheus metrics

[MikeGoldsmith]

Which problem is this PR solving?

A mutex is used in the prometheus metrics implementation and in #324 was used the lock when using the different metrics types. This PR switches the mutex type to a RWLock to allow concurrent retrieving of metrics from the map.

Register is the only place where a write lock is required.

Short description of the changes

• Update mutex to RWLock
• use RLock/RUnlock when using the different metric types

[robbkidd]

During our team sync today, we talked of adding tests around concurrency and race conditions for this change.

[robbkidd]

The new TestRaciness test shows the race condition when run against v1.5.0, the version prior to Lock or RWLock being added to the other PromMetrics functions.

go test -tags race --race --timeout 60s -v ./...

=== RUN   TestMultipleRegistrations
--- PASS: TestMultipleRegistrations (0.00s)
=== RUN   TestRaciness
==================
WARNING: DATA RACE
Write at 0x00c0002d3e30 by goroutine 19:
  runtime.mapassign_faststr()
      /Users/robbkidd/.asdf/installs/golang/1.16.6/go/src/runtime/map_faststr.go:202 +0x0
  github.com/honeycombio/refinery/metrics.(*PromMetrics).Register()
      /Users/robbkidd/work/sampling/refinery/metrics/prometheus.go:78 +0x1cc
  github.com/honeycombio/refinery/metrics.TestRaciness.func1()
      /Users/robbkidd/work/sampling/refinery/metrics/prometheus_test.go:46 +0xc6

Previous read at 0x00c0002d3e30 by goroutine 20:
  runtime.mapaccess2_faststr()
      /Users/robbkidd/.asdf/installs/golang/1.16.6/go/src/runtime/map_faststr.go:107 +0x0
  github.com/honeycombio/refinery/metrics.(*PromMetrics).Increment()
      /Users/robbkidd/work/sampling/refinery/metrics/prometheus.go:82 +0x76
  github.com/honeycombio/refinery/metrics.TestRaciness.func2()
      /Users/robbkidd/work/sampling/refinery/metrics/prometheus_test.go:50 +0x4d

Goroutine 19 (running) created at:
  github.com/honeycombio/refinery/metrics.TestRaciness()
      /Users/robbkidd/work/sampling/refinery/metrics/prometheus_test.go:44 +0x23e
  testing.tRunner()
      /Users/robbkidd/.asdf/installs/golang/1.16.6/go/src/testing/testing.go:1193 +0x202

Goroutine 20 (finished) created at:
  github.com/honeycombio/refinery/metrics.TestRaciness()
      /Users/robbkidd/work/sampling/refinery/metrics/prometheus_test.go:49 +0x26a
  testing.tRunner()
      /Users/robbkidd/.asdf/installs/golang/1.16.6/go/src/testing/testing.go:1193 +0x202
==================
    testing.go:1092: race detected during execution of test
--- FAIL: TestRaciness (0.01s)
=== CONT  
    testing.go:1092: race detected during execution of test
FAIL
FAIL    github.com/honeycombio/refinery/metrics 0.234s

I contributed the test I requested.

[robbkidd]

Oh. Heh. A previous iteration of this used p.Count("race", j) which needed i to be passed in to avoid triggering the wrong race condition. Doesn't need to be any more. 😶