Automatically restart brokers with offline log directory

[bderrly]

I have run into a scenario two times now where a broker marks a log directory offline. We have four PVCs in JBOD configuration for each of six brokers. We have a small number of topics all with replication of 3 and one with only a single replica. For our current use case with Kafka if we ever encounter an offline log directory we have an operator restart the broker (as that seems to be the only way to recover from such a condition). Given that this is our default mitigation action it would be nice to have it automated—especially in the middle of the night.

I am sure there are a multitude of edge cases I am not aware of to demonstrate why this is a terrible idea. For our use case however it seems like an obvious quality of life improvement for both the cluster and the human operator. It could be nice feature for people that understand the risks and know that it is acceptable for their use.

[scholzj]

@tombentley @mimaison @showuon Any thoughts on this?