Signal shutdown / failed startup

[cpnielsen]

Hi 👋

I am working an a kopf operator, and want to load some configuration before the operator starts. Looking at the documentation, I assume that kopf.on.startup is the correct place to do this.

However, in my case there are situations where I do not want to continue loading - ie. some configuration is missing, malformed, etc. or the startup cannot complete for other reasons (no connectivity to a database, etc.).

I tried throwing an exception from the startup function, but that just causes the method to be retried indefinitely.

My questions are:

1. How do I shutdown / halt startup from the startup function?
2. In general, if I want to signal shutdown (from some other activity function or lister), how do I do that?

Thank you!

[nolar]

Hello. Technically, there is no way to stop the operator from inside. But there is a way to stop it from outside, which you can extend to the insides.

You need to abandon the kop run CLI and make your own CLI with an embedded operator, combined with the in-memory container (memo):

import asyncio
import threading
import kopf


@kopf.on.create('kopfexamples')
def create_kex(memo, name, **_):
    if name == 'stop-me':
        memo.my_stop_flag.set()


def main():
    kopf.configure(verbose=True, log_prefix=True)  # purely for logging

    # either threading.Event(), or asyncio.Event(), or asyncio/concurrent Future().
    memo = kopf.Memo(my_stop_flag=threading.Event())
    asyncio.run(kopf.operator(memo=memo, stop_flag=memo.my_stop_flag, ...))


if __name__ == '__main__':
    main()

[2021-11-23 21:02:42,774] kopf.objects         [DEBUG   ] [default/stop-me] Handler 'create_kex' is invoked.
[2021-11-23 21:02:42,775] kopf.objects         [INFO    ] [default/stop-me] Handler 'create_kex' succeeded.
[2021-11-23 21:02:42,776] kopf.objects         [INFO    ] [default/stop-me] Creation is processed: 1 succeeded; 0 failed.
[2021-11-23 21:02:42,777] kopf.objects         [DEBUG   ] [default/stop-me] Patching with: {'metadata': {'annotations': {'kopf.zalando.org/last-handled-configuration': '{"spec":{"duration":"1m","field":"value","items":["item1","item2"]},"metadata":{"labels":{"somelabel":"somevalue"},"annotations":{"someannotation":"somevalue"}}}\n'}}}
[2021-11-23 21:02:42,779] kopf._core.reactor.r [INFO    ] Stop-flag is set to True. Operator is stopping.
[2021-11-23 21:02:42,780] kopf._core.reactor.r [DEBUG   ] Namespace observer is cancelled.
[2021-11-23 21:02:42,780] kopf._core.reactor.r [DEBUG   ] Credentials retriever is cancelled.

Don't worry, there is not much which kopf run does above that. It is as simple as this: https://github.com/nolar/kopf/blob/1.35.3/kopf/cli.py#L94-L116 — which you can do yourself. All CLI options should be replaced with relevant arguments: namespaces=, clusterwide=, peering_name=, standalone=, etc.

PS: Oops. The stop_flag/ready_flag turn out to be undocumented. That is a mistake. They should be documented. They are the public interface of kopf.operator()/kopf.run(), so feel free to use them safely. Contributions to the documentation are welcome!

PPS: Just for some extra clarity: kopf.run(...) is almost the same as asyncio.run(kopf.operator(...)) — unless you manage the asyncio event loops yourself (in which case you know the difference).

[cpnielsen]

@nolar Is it necessary to wrap the stop flag in a kopf.Memo, or can I pass in a asyncio.Event directly? Given that I am running other asyncio tasks in the same loop, there should not be any concurrency issues.

[nolar]

You can pass it any way that works. A global state or contextvars are also fine.

running other asyncio tasks in the same loop

Keep in mind: when Kopf is stopping, it cancels and waits for all tasks that have appeared after its startup — assuming that they are Kopf's or the handlers' tasks and belong to the operator. If there is something else running in the event loop, it can be unexpectedly stopped by this logic. It is generally advised to run Kopf in a dedicated event loop.

[cpnielsen]

In my case, there should only ever be some upfront thing happening, then everything will be more or less handed over to kopf, so having it all in a sync event loop makes it simpler.