Nexus: Add --user $USER to squeue command

[mcbennet]

Proposed changes

Adding flag --me to squeue command in machines.py. This prevents users from being forced to provide their usernames to avoid strange errors when other users format their job names in weird ways. This also resolves #2380

What type(s) of changes does this code introduce?

• Other (please describe): Add stability

Does this introduce a breaking change?

• No. All nxs-test tests pass

What systems has this change been tested on?

Laptop. Fedora Linux

Checklist

• Yes. This PR is up to date with current the current state of 'develop'
• N/A. Code added or changed in the PR has been clang-formatted
• N/A. This PR adds tests to cover any new code, or to catch a bug that is being fixed
• N/A. Documentation has been added (if appropriate)

[jtkrogel]

What Slurm versions support the --me flag?

[mcbennet]

I found this running list of updates (https://github.com/SchedMD/slurm/blob/master/NEWSupdates). It does not mention --me being added to squeue all the way back to v2.1.0 -- so I assume it was supported all the back to this point(?).

CADES uses v20.02.4 and supports the --me flag
Cori uses v20.11.08 and supports the --me flag

Perhaps someone at Sandia could let us know if squeue --me is supported on those machines? @camelto2

[prckent]

It looks to have been added to the longoptions of slurm in SchedMD/slurm@8483780

[mcbennet]

Thanks Paul, but I am correct that the commit there relates to "auto completion" functionality? And it looks to be commited in v21, however, Cori and CADES support the flag and use v20

[mcbennet]

It was added to the man page 5 months ago it appears

Line 1362 of
https://github.com/SchedMD/slurm/blame/master/doc/man/man1/squeue.1

[ye-luo]

Can you just use os.getenv('USER') and then -u $USER? This should work in all job management I have ever seen.

[mcbennet]

Correction to my previous comment, it appears to show up in man page for the first time on Feb 2020

https://github.com/SchedMD/slurm/blob/cc6a1be5ee2cf0e1ac5a7c14a6df97023701e8e8/doc/man/man1/squeue.1

@ye-luo This might be okay, but will let Jaron weigh in

[prckent]

Wearing my sometimes sysadmin hat, updating slurm is one of the last things i would want to do on a running system. So we can't rely on this --me feature being available or slurm being highly up to date => this can only be a nice to have and not overly important feature. I think Jaron is the decider here. Catching any errors would be wise.

[mcbennet]

Just throwing out one other alternative that is worth chewing on. We could consider formatting squeue's output such that it only gives what nexus needs (jobid and status only?). Something like

For instance,

squeue -o "%.18i %.2t" 

Or something similar. The parsing done by Nexus will need a minor change as well

[jtkrogel]

I like the simplicity offered here, but I have to agree with Paul that the chance of breaking based on the local environment is too high.

For the feature to be introduced, I think the current behavior with user should be preserved and a new flag used instead to invoke querying with --me (perhaps invoked as user_me=True with a default of False).

[mcbennet]

The underlying "issue" is not resolved unless we change the default behavior in some way. What users can run into (on Cori for instance) is some strange error like job state jackm is unrecognized -- this is due to squeue not providing a consistent format across output lines then Nexus is not able to parse correctly. IMHO, the solution offered by @ye-luo or the solution to reformat the squeue output via -o are clean ways to resolve this and do not require recent slurm versions

[jtkrogel]

Perhaps squeue --user $USER can be the default (test it at a few sites please). There needs to be a way to turn this behavior off, perhaps by passing user=False to settings.

[mcbennet]

The following command has been run on multiple machines at multiple sites

execute('squeue --user=$USER')

The tested machines and their results are:

• OLCF/Andes (SUCCESS)
• NERSC/Cori (SUCCESS)
• Sandia machines (SUCCESS)
• ORNL/CADES (SUCCESS)

The SLURM machines that Nexus supports:

Edison (RETIRED)
Cori
Chama
Uno
Eclipse
Attaway
Skybridge
Solo
Stampede2
CadesSlurm
Rhea
Andes

[jtkrogel]

Good. I think we can proceed with a change to using --user=$USER.

Perhaps the logic should be:

if isinstance(self.user,bool) and self.user==False:
    extra = ''
elif self.user is not None:
    extra = ' -u {}'.format(self.user)
else:
    extra = ' --user=$USER'
#end if

[mcbennet]

Thank you @jtkrogel . pushed

[jtkrogel]

LGTM. Tests pass?

[mcbennet]

Yes. 100% of nxs-test tests pass

[jtkrogel]

Great

[ye-luo]

Test this please