can we run prolog/epilog/housekeeping in the flux systemd instance? #6619
Comments
|
Just did an experiment with the following change diff --git a/src/modules/job-manager/housekeeping.c b/src/modules/job-manager/housekeeping.c
index 82b376fc6..04faf76da 100644
--- a/src/modules/job-manager/housekeeping.c
+++ b/src/modules/job-manager/housekeeping.c
@@ -191,7 +191,7 @@ static struct allocation *allocation_create (struct housekeeping *hk,
allocation_timeout,
a))
|| !(a->bulk_exec = bulk_exec_create (&bulk_ops,
- "rexec",
+ "sdexec",
id,
"housekeeping",
a))
@@ -698,6 +698,8 @@ static void housekeeping_kill_cb (flux_t *h,
while (a) {
if (a->id == jobid || jobid == FLUX_JOBID_ANY) {
if (a->bulk_exec) {
+ if (signum == SIGKILL)
+ signum = SIGUSR1;
f = bulk_exec_kill (a->bulk_exec, ids, signum);
if (flux_future_then (f, -1, kill_continuation, hk) < 0)
flux_future_destroy (f);
@@ -817,6 +819,11 @@ static int housekeeping_parse_config (const flux_conf_t *conf,
flux_cmd_destroy (cmd);
return errprintf (error, "error duplicating IMP path");
}
+ if (flux_cmd_setopt (cmd, "SDEXEC_PROP_KillMode", "process") < 0
+ || flux_cmd_setopt (cmd, "SDEXEC_PROP_SendSIGKILL", "off") < 0) {
+ flux_cmd_destroy (cmd);
+ return errprintf (error, "error setting sdexec options");
+ }
}
done:
flux_cmd_destroy (hk->cmd);
and a config change [run.housekeeping]
allowed-users = [ "flux" ]
allowed-environment = [ "FLUX_*" ]
# path = "/usr/lib/aarch64-linux-gnu/flux/cmd/flux-run-housekeeping"
path = "/etc/flux/system/housekeeping"Somewhat shockingly, that worked pretty well! The one thing that didn't work was the SIGUSR1 hack. I haven't dug into it yet but I guess SIGUSR1 isn't translated to SIGKILL in the IMP for the Also, since the unit file was what drained a node on failure, that would have to be added to housekeeping. |
|
Hmm, it looks like signals (including the SIGUSR1 translation) are handled the same for both Puzzling. |
|
sdexec may be waiting for stdout which is held open by the lingering processes and that is why the job manager is not notified that the unit has ended until all the processes are gone. That is OK I guess. The question is why isn't the IMP delivering SIGKILL to those other processes and then why is it exiting. It must be shooting itself for some reason. |
|
Yeah the IMP seems to have SIGKILLed itself. This change to the IMP fixes it. diff --git a/src/imp/cgroup.c b/src/imp/cgroup.c
index 9375ec2..14e1a99 100644
--- a/src/imp/cgroup.c
+++ b/src/imp/cgroup.c
@@ -197,7 +197,8 @@ struct cgroup_info *cgroup_info_create (void)
/* Note: GNU basename(3) never modifies its argument. (_GNU_SOURCE
* is defined in config.h.)
*/
- if (strncmp (basename (cgroup->path), "imp-shell", 9) == 0)
+ if (strncmp (basename (cgroup->path), "imp-shell", 9) == 0
+ || strncmp (basename (cgroup->path), "housekeeping", 12) == 0)
cgroup->use_cgroup_kill = true;
return cgroup; |
This was referenced Feb 19, 2025
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem: in #6616, a connection to the system dbus is established and two systemd instances are monitored because we start jobs in the flux one and prolog/epilog/housekeeping in the system one.
Is there any reason we couldn't run them all in the flux systemd instance and avoid the extra dbus connection?
The text was updated successfully, but these errors were encountered: