Reconciliation can get blocked when Statefulset Controller is stuck trying to create a Pod

[d-guo]

Bug Report

This is very similar to the problem reported in #2266, except it is surfaced in a way that the fix for that issue doesn't work.

The way it surfaced is I modified the spec with a toleration that is not allowed. This propagated to the statefulset which then resulted in the error pasted below. I then fixed the spec by removing this toleration, but eck is not propagating the change to the statefulset anymore.

Events:
  Type     Reason        Age                     From                    Message
  ----     ------        ----                    ----                    -------
  Warning  FailedCreate  6m18s (x32 over 3h59m)  statefulset-controller  create Pod es-test-2 in StatefulSet es-test failed error: admission webhook "{my-auth-service}" denied the request: {my-error-message — related to a toleration i set in the spec that is not allowed}

The problem here is that the "dry-run" solution in #3020 (which works in most cases) fails here because the failure does not occur in the pod but rather in the statefulset controller.

Is this an issue the community is interested in? I think a fix here could be to implement the timeout described in #2266 (comment). This would still be nice to have on top of the existing "dry-run" validation.

[d-guo]

@barkbay @sebgl in case you guys are interested here :)