Support cloud storage file transfers

[robinkar]

This PR adds support for moving, copying, deleting and renaming files and directories on cloud storages using Rclone, fixes #2129.

This adds from_fs and to_fs optional parameters to the transfers API, which are used to define from and to which Rclone remote files are transferred. The default is fs, which is treated as the local (posix) filesystem so that it is fully backwards compatible. The feature is enabled by setting OOD_FILES_APP_REMOTE_FILES to true.

Overwriting files is avoided so that it behaves in the same way as normal transfers on the local filesystem. That also simplifies the Rclone commands that are needed significantly. Allowlist paths are also checked for the local filesystem, and remotes of type local and alias are disabled when the allowlist is enabled (same behaviour as in #2119).

Short demo (includes #2140 for easier demonstration):

┆Issue is synchronized with this Asana task by Unito

[robinkar]

Synced and resolved conflicts with master.

[johrstrom]

I wonder if we want to keep the same pattern of src and dest instead of to and from. They're the same semantically, I just wonder if we want to standardize the language everywhere.

[robinkar]

Using src and dest sounds better to me. Current naming of from and to is from clip_board.js which uses that naming pattern. Is this something we want to standardize across both the backend and frontend, or would simply renaming the request parameters be sufficient?

[johrstrom]

I think at least changing new code is our best bet. We can keep the js as is and defer it's refactor.

So, we can't change what we're getting, but we can change what we name the internal variable sort of thing.

Suggested change

	from_fs = body_params.fetch(:from_fs, "fs")
	to_fs = body_params.fetch(:to_fs, "fs")
	src_fs = body_params.fetch(:from_fs, "fs")
	dest_fs = body_params.fetch(:to_fs, "fs")

[robinkar]

Do we want to change to use src_fs and dest_fs as the names of the params in the request at the same time too as this PR introduces that? Easier to get that right from the beginning than change it later.
I.e. change only the body here:

ondemand/apps/dashboard/app/javascript/packs/files/file_ops.js

Lines 473 to 481 in bc73353

	return fetch(transfersPath, {
	method: 'post',
	body: JSON.stringify({
	command: action,
	files: files,
	from_fs: from_fs,
	to_fs: to_fs,
	}),
	headers: { 'X-CSRF-Token': csrf_token }

[johrstrom]

Easier to get that right from the beginning than change it later.

I tend to think the opposite with changes these large and #2200 possibly conflicting. But if you have the time, you can go ahead and change it - I don't feel too strongly about now or later and #2200 can rebase or merge.

[johrstrom]

We can defer this - but the string "fs" is around enough that we likely need to make it a constant and refer to it instead of hoping we type fs correctly all over the place.

[robinkar]

That would be a good idea. "fs" definitely is used in quite a few different places.
What would be the correct place for that constant? RcloneUtil?

[johrstrom]

That seems as good as any.

[robinkar]

Added the constant RcloneUtil::LOCAL_FS_NAME. Only changed the places where this PR adds "fs".

[robinkar]

Strange, tests ran perfectly fine locally and worked fine in browser too, but somehow it's broken now. Will fix.

[robinkar]

Should be good now, tests passed locally since the tests using RemoteFile ran first when I ran them locally, and same situation in browser.
https://github.com/OSC/ondemand/compare/bc733533fa94f53440ca97a6d74d8f51b6508e28..7f8652e21a65b205835b1af107fe0acd213574e5

[robinkar]

Resynced with master (to get #2220) and added with_indifferent_access.

[Oglopf]

Looks good, thank you so much!