Closing a Kafka writer hangs

[danxmoran]

• Our Kafka writers run a metric-collecting goroutine in the background.
• The goroutine ranges over a time.Ticker's channel, and calls Done() on a sync.WaitGroup after the range completes.
• Close() on the writer calls Stop() on the ticker, then Wait() on the wait group
• Calling Stop() on a ticker doesn't actually close its channel, so:
  • The range over the channel never completes
  • Done() is never called on the wait group
  • Wait() on the wait group never completes

[danxmoran]

Note: this can also cause requests to update Kafka configs to hang, because certain config updates call Close() on every existing writer.