Feature request: per-process shutdown command / trap support

[thenonameguy]

For my use-case most of the commands I'm managing via process-compose eventually yield a process into the background, and exit cleanly with 0, with a no retry policy.
Other processes use the process_completed dependency feature to realize that the given process (let's say a postgres init script) has concluded and now the postgres server is ready in the background.
So far so good 🎉

Now, the only problem with these daemon-launching scripts are that if the process-compose process is killed, the underlying daemons are not turned off.
This leaves the environment in an inconsistent state (unlike calling docker-compose down/ Ctrl-C'ing the up command).

Proposal:

• add processes.$PROCESSNAME.shutdown_command: string to the schema
• handle SIGTERM in process-compose itself, exec.Command these commands in parallel, wait for the command to terminate.

stretch goal/kinda unrelated but-nice-to-have:

• revisit the current SIGKILL way of killing the underlying process (instead use SIGTERM), so the command has a chance to run any kind of bash trap like functionality

[F1bonacc1]

I think that the proposal is clear:

1. On process termination, if processes.$PROCESSNAME.shutdown_command is defined, execute the command
2. Wait for processes.$PROCESSNAME.shutdown_timeout amount of seconds for the command to return
3. In case of timeout - sigkill
4. If processes.$PROCESSNAME.shutdown_timeout is not defined use 10sec?

Does it sound reasonable?

[thenonameguy]

I think that the proposal is clear:

1. On process termination, if processes.$PROCESSNAME.shutdown_command is defined, execute the command

2. Wait for processes.$PROCESSNAME.shutdown_timeout amount of seconds for the command to return

3. In case of timeout - sigkill

4. If processes.$PROCESSNAME.shutdown_timeout is not defined use 10sec?

Does it sound reasonable?

Perfect, good catch with not waiting indefinitely! 10s seems plenty, but not too much.

[F1bonacc1]

@thenonameguy I've implemented the first stage of the solution in release 0.11.0 in which a manual termination request (F9 in the TUI) will use the shutdown procedure.
Once we confirm that it behaves as expected I will use the same termination method to terminate all the running processes on process-compose exit.

[F1bonacc1]

@thenonameguy v0.12.0 should resolve the issue fully.
Please confirm that it works on your end as expected and I will close the issue.

[thenonameguy]

Thanks @F1bonacc1 for the very fast implementation!
I checked today, but it seems the last PR introduced an extra check that turns the feature void:
https://github.com/F1bonacc1/process-compose/pull/5/files#r907846499

Once this is addressed via either idempotency or extra configuration IMO the feature can be considered done. 🎉

[F1bonacc1]

@thenonameguy you are right!
I think my main mistake was to treat daemons as processes... :(
Daemons should have a different execution state machine.

I've been thinking about something along those lines:

1. Introduce new configuration flag processes.$PROCESSNAME.is_daemon
2. In case a process is daemon it will be considered running until stopped
3. Stop will run on daemons in any state and ignore the return code of the processes.$PROCESSNAME.shutdown_command
4. I will also add liveness and readiness probes to make sure the daemon is actually running.

On the positive side, a lot of infra that I added can be reused :)
What do you think?

[thenonameguy]

That sounds perfect! Communicating to the supervisor that it's a daemon/service vs. process makes sense to be able to implement this kind of logic.

I guess for 2. in this case you could translate the Completed state to Launched or something to differentiate between the start script running vs. it having exited => daemon launched.

You are awfully getting close to a minimal systemd service definition language here with this change though.
https://www.freedesktop.org/software/systemd/man/systemd.service.html see example 5 for instance for this exact problem
For 4. maybe consider using a similar structure as kubernetes uses?

The end-goal might be for this project that you can almost trivially switch from systemd service files/kubernetes-based pod definitions to locally running processes with minimal service definition change / wide-enough feature set to be usable.

Looking good so far! 🎉

[F1bonacc1]

@thenonameguy if you still didn't give up on this project, the changes that we discussed are in v0.15.0:

• Proper background processes support
• Health checks (liveness and readiness probes)

It took longer than expected... Lot's of edge cases to iron out.

Please confirm that it works on your end as expected and I will close the issue.

[thenonameguy]

@thenonameguy if you still didn't give up on this project, the changes that we discussed are in v0.15.0:

• Proper background processes support

• Health checks (liveness and readiness probes)

It took longer than expected... Lot's of edge cases to iron out.

Please confirm that it works on your end as expected and I will close the issue.

I absolutely haven't, have been using the daemon support happily since the release. Really glad about the health checks, will integrate it tommorrow in my project 👏

[F1bonacc1]

@thenonameguy is the issue resolved from your point of view?

[thenonameguy]

Yes, it works flawlessly, great job! It's a great resource for our project 🎉
Haven't tried health checks yet, but read the docs and love the feature parity with Kubernetes. Lot of intermediate steps skipped between naive implementation and what's actually needed by users like me.
Let's close this, as it's as done as it gets from my part.