fix: provide possibility to have a soft error mode

[szuecs]

fix: provide possibility to have a soft error mode

fixes #4067

closes #4147

[szuecs]

@mloiseleur wdyt?
It has also the nice added value of dropping all the different kind of old error packages, that are not useful anymore.

[mloiseleur]

If I understand correctly, this SoftError mode allow providers to decide if an error throwed should be fatal or just displayed.
Considering it's working on DNS, it makes sense to me ; instead of failing fast for all cases, it may allow to fail-fast only when it makes sense.

@gregsidelinger @jbilliau-rcd Wdyt of this approach ?

[jbilliau-rcd]

@mloiseleur If it reverts to the functionality of 0.13.4, I'm all for it. We have 150+ clusters in our environment with hundreds of devs deploying to them. We have errors in external-dns all the time from bad deploys, bad URL's, duplicate records, typical "huge enterprise with devs who are still learning" type stuff. Having a controller crash whenever there is any problem with any record....I don't understand why anybody would ever want that. Currently, on 0.13.4, the bad records error out but any new ones that are correct create just fine, so the bad records don't blow everything up.

So yes, a RESOUNDING yes to this, lol

[szuecs]

@mloiseleur If it reverts to the functionality of 0.13.4, I'm all for it. We have 150+ clusters in our environment with hundreds of devs deploying to them. We have errors in external-dns all the time from bad deploys, bad URL's, duplicate records, typical "huge enterprise with devs who are still learning" type stuff. Having a controller crash whenever there is any problem with any record....I don't understand why anybody would ever want that. Currently, on 0.13.4, the bad records error out but any new ones that are correct create just fine, so the bad records don't blow everything up.

We run more than 200 clusters with that version and every dev can deploy on their own. We have 0 restarts of external-dns. It just depends on how you structure things such that devs can or can not make errors easily.

If you would check more issues in this repository that have similar "start", you will see that many things are hidden miss configurations and that's actually good if cluster owners start to understand what kind of errors they have in their clusters.

[jbilliau-rcd]

You run 200 clusters with 0.13.6+ and it never restarts? Well, I guess you've achieved the holy grail of flawlessly created records and perfectly working DNS 100% of the time, I would think that would be the exception, not the rule. This controller has been noticeably more "finicky" ever since the change with the TXT records. Our controllers will sporadically try to recreate TXT records that already exist (it created them, go figure) and then, with the 0.13.6 version, crash because it can't.

We aren't doing anything crazy or fancy, so I can't see how you've never come across these issues but who knows.

Regardless, I stand by my opinion; if ONE app has a misconfiguration problem, it shouldn't affect the ENTIRE cluster. Isn't that completely antithetical to the whole point of Kubernetes?

[szuecs]

@jbilliau-rcd feel free to fork and maintain your own.
Sorry, but you seem to be out of reasonable commenting here.
Feel also free to do something else, but I won't reply to your comments anymore, have a great day and re-think about how you reach out to people trying to make things better.

[jbilliau-rcd]

@szuecs Apologies if I came across that way, that wasn't my intent. I was just honestly curious as to how you never have DNS creation issues with that many clusters or restarts, I wonder what's different about our configs. I could've worded that better, again, apologies.

[szuecs]

@jbilliau-rcd thanks that you clarified it! Makes my day much better :)

[mloiseleur]

/lgtm

[szuecs]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: szuecs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [szuecs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jbilliau-rcd]

@szuecs is this change being released anytime soon? How do we turn it on, the "soft error mode" so it doesn't crash on any errors? Thanks again for the effort here.

[zbcon]

@szuecs is this change being released anytime soon? How do we turn it on, the "soft error mode" so it doesn't crash on any errors? Thanks again for the effort here.

Also interested in how we can turn this soft error mode on, thanks 👍

[SimonKienzler]

@jbilliau-rcd @zbcon I just wanted to follow up on your questions, as we've encountered similar problems and found the SoftError mode PR #4166 some time ago.

It really depends on the provider you're using, and if it already implements SoftErrors where applicable. E.g. #4166 came with AWS support, I added it for the webhook provider in #4319, there is also #4229 for the OCI provider and #4199 for Azure. I think that's it, so far. So, "turning it on" basically means contributing it if it's not yet supported in the provider that you use. OCI, Azure and AWS providers support has been released in v0.14.1. Hope that helps.

[jbilliau-rcd]

Sounds good, thank you @SimonKienzler . I'll give 0.14.1 a go and hopefully we can now upgrade from our current version we've been pinned at due to this issue, 0.13.4