Submit commands on another host via ssh

[matthu017]

Made Adapters::Helper.ssh_wrap to return commands wrapped in ssh

[ericfranz]

I mentioned one example inline, but for each adapter we want to keep both host and submit host as they are different values.

• host is the host of the batch server we are submitting the job to
• submit_host is the host where we are executing the job submission command

For example, I could specify the submit_host as being ruby.osc.edu and the host as being owens.osc.edu. Then I would be ssh-ing to ruby.osc.edu and executing qsub with the batch server specified as owens.osc.edu to submit a batch job running on owens.

[johrstrom]

With the realization in #201 we may have to rethink how the job_environment works for this feature.

Instead of just passing the command, we may have to template a small script that can inline the correct environment variables (like the linux host adapter does). I don't believe we want to get involved with SendEnv and AcceptEnv just because that would require admins to go through some pain, then realize there's missing sshd configuration.

Let's look at how this would work for PBSPro.
job_environment is OK because it's a part of the command.

ood_core/lib/ood_core/job/adapters/pbspro.rb

Lines 262 to 263 in 9eaf3cf

	envvars = script.job_environment.to_h
	args.concat ["-v", envvars.map{|k,v| "#{k}=#{v}"}.join(",")] unless envvars.empty?

But these two environment variables need to be passed to the destination. Anything that is set in the environment to the call or Open3 functions needs to be passed to the submit_host.

ood_core/lib/ood_core/job/adapters/pbspro.rb

Lines 164 to 165 in 9eaf3cf

	env["PBS_DEFAULT"] = host.to_s if host
	env["PBS_EXEC"] = pbs_exec.to_s if pbs_exec

Maybe instead of templating a script (because that would require all the adapters to toggle the cmd or stdin - the script), we just inline the variables in the command.

Something to this affect:
ssh johrstrom@owens.osc.edu 'export FOO=BAR; echo $FOO'

The ssh_wrap interface probably expands to include environment you want to pass, and you prepend exports to it.

So submitting to owens with PBSPro adapter ends up resulting in a command like this:
ssh johrstrom@owens.osc.edu 'export PBS_DEFAULT=owens; qsub myscript.sh'

(Note that the quotes are important becuase the shell will see the ; as the terminating command which it isn't. At least not on the source side).

[johrstrom]

Also, I thought adapters like Torque had some restriction because they use shared libraries (FFI) that may not be on the destination hosts?

[ericfranz]

Torque does not use FFI for qsub

[johrstrom]

Is that the behavior we want, where some things are executing on the submit_host and others on the webhost?

[ericfranz]

#170 Addresses that. This PR is the first half, enabling only qsub by specifying submit_host: "some_other_host". Another feature would be adding use_submit_host_for_all_commands: true to enable the rest of the commands executed on the submit host. In Torque and SGE's cases this second step is problematic because of their use of FFI. We could drop FFI then.

[ericfranz]

This does the same thing:

args.push(env.map {|k,v| "export #{key}=#{value};" } + [cmd] + cmd_args).join(' '))

though env.map {|k,v| "export #{key}=#{value};" } should be a separate method

[matthu017]

Should a new method still be made for this with the current commit? There are quite a few params in this method already, but everything in this command deals with ssh wrapping.

[ericfranz]

cmd_args.unshift(cmd) modifies the argument cmd_args which should not happen. I know you just changed a bunch of + to concats, but in this case + may improve readability.

return 'ssh', args + [cmd] + cmd_args