per cluster dataroot for interactive apps

[johrstrom]

There have been several discourse topics (links needed) where a site has filesystems that are distinct to a given cluster. Specifically, their HOME directories are separate for each cluster.

Historically we've told folks they need a different ondemand instance per cluster because stuff like dataroot evaluate to $HOME/ondemand/data. The staged_root for a given job (the job's working directory, the webserver templates files and puts them here before submitting the job) is a subdirectory under this dataroot.

So clearly there's a need for some sites to have these directories on a per cluster basis. If the site could come up with a sshfs scheme to mount the other filesystems, then the webserver could access 2 or more file systems.

# here's an example of 'dataroot' locations on a per cluster basis for the oakley and owens clusters.

$HOME/ondemand/data/oakley
$HOME/ondemand/data/owens

What the $HOME filesystem is on the webserver is anyones guess. Maybe it could be a local filesystem that mounts the others?

TODO

• batch connect apps to use a per cluster dataroot #1409
• change location of context.json #1521
• can't switch between portals when one has per cluster dataroots set #1457

[johrstrom]

This initializer may patch a system for batch connect apps to be able to submit to schedulers with different file systems. Place the initializer at /etc/ood/config/apps/dashboard/initializers/bc_cluster_dataroot.rb.

# extend the BatchConnect::Session so that it submits jobs on a 'per-cluster' basis.

module BatchConnect
  class Session

    class << self
      def dataroot(token = "", cluster: nil)
        OodAppkit.dataroot.join('batch_connect').join(cluster.to_s).join(token.to_s)
      end
    end

    def save(app:, context:, format: nil)
      self.id         = SecureRandom.uuid
      self.token      = app.token
      self.title      = app.title
      self.view       = app.session_view
      self.created_at = Time.now.to_i
      self.cluster_id = context.try(:cluster).to_s

      submit_script = app.submit_opts(context, fmt: format, staged_root: staged_root) # could raise an exception

      self.cluster_id = submit_script.fetch(:cluster, cluster_id).to_s
      raise(ClusterNotFound, I18n.t('dashboard.batch_connect_missing_cluster')) unless cluster_id.present?

      stage(app.root.join("template"), context: context) && submit(submit_script)
    rescue => e   # rescue from all standard exceptions (app never crashes)
      errors.add(:save, e.message)
      Rails.logger.error("ERROR: #{e.class} - #{e.message}")
      false
    end

    def staged_root
      self.class.dataroot(token, cluster: cluster_id).join("output", id)
    end
  end
end

This changes the dataroot such that clustername is a part of the path where the app is staged and ran from.
Instead of this as the root directory for this app ran on any cluster:

~/ondemand/data/sys/dashboard/batch_connect/sys/<APPNAME>

It turns into this with the cluster in the directory structure.

~/ondemand/data/sys/dashboard/batch_connect/<CLUSTER>/sys/<APPNAME>

Now to get this to work with multiple file-systems, sites would have to mount ~/ondemand/data/sys/dashboard/batch_connect/<CLUSTER> on the web node to the same location on the compute node.

[johrstrom]

I'm going to open this back up until we have the job composer functionality as well.

[johrstrom]

Actually, we're not back porting this to the job composer.