exposing the tidb cluster states/changes as k8s events to users

[weekface]

Many users report that they cannot know what the tidb-operator is doing when failover occurs, which confuses them.

We need to expose the internal state of the cluster to users with the help of Events or ContainerConditions, or anything else, for example:

$ kubectl describe tc <tc-name>
...
Events:
  Type     Reason            Age                 From            Message
  ----     ------            ----                ----            -------
  Warning  StoreOffline  26s (x25 over 45s)  tidb-controler  store-1 is offline 2 minitues
  Warning  StoreDown     17s (x25 over 41s)  tidb-controler  store-1 is down 30 minutes
  Warning  Failover      17s (x25 over 41s)  tidb-controler  store-1 failover occurs now
  ...

• tidb scheduler (Done) scheduler: refine scheduler error message #1373
• pd failover (WIP) Emit k8s events when pd failover #1465 failover: emit events when pd failover #1466
• tikv failover
• tidb failover
• emit an event when pd member status changed report an event when pd status changed #1495
• setting the events BurstSize/QPS to proper values setting the events BurstSize/QPS to proper values #1494
• Emit an event when a failed(Failover) member recovers Emit an event when a failed(Failover) member recovers #1798

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 15 days