fix(process worker): fix the crashing of pod #1119
Conversation
dhiren-singh-007
requested review from
ntruchsess and
Phil91
and removed request for
ntruchsess
…calls. terminate processworker only when running out of memory. update framework version
ntruchsess
force-pushed
the
bugifx/1114-fix-process-worker-crash
branch
from
c9ebf6f to
b62273b
Compare
|
|
@dhiren-singh-007 @Phil91 : I think it makes sense to force a restart of the pod when it runs out of memory - everything else should be logged as critical. I also changed the way SocketException and IOException are handled within the remote api calls. Those wouln't throw this kind of systemexception but ServiceException flagged as RecoverOptions.NETWORK (being mapped to RecoverOptions.INFRASTRUCTURE which is most external system calls treat as recoverable skipping the process-step to be retried later). Nevertheless we might want to evaluate whether this is the right strategy for all external system calls - e.g. when calling a url that is provided by the customer it might be better to set the respective process-step to ERROR while retrying the same call with the next process-worker run would be suitable for calls to 'known as good' urls where it's reasonable to assume the network-related issue is temporary |
* terminate processworker only when running out of memory. * handle SocketException and IOException explicitly in external system calls. * update framework version * update process worker test --------- Co-authored-by: Norbert Truchsess <norbert.truchsess@t-online.de>
* terminate processworker only when running out of memory. * handle SocketException and IOException explicitly in external system calls. * update framework version * update process worker test --------- Co-authored-by: Norbert Truchsess <norbert.truchsess@t-online.de>
Description
Handled the system exception and added the Critical/Fatal log type so that we can monitor the logs for criticality.
Why
It was causing issues for other processes as well .
Eg if a OSP customer sets the wrong callback address or if it is down then it returns the referenced class of SystemException and then it crashed the pod , which stops all other processes.
Issue
#1114
Checklist
Please delete options that are not relevant.