Ensure destination directory exists + handle index mount directories (#…

…167)

This change un-reverts PR #161 with a couple twists: account for index
mount directories and use mpirun to perform the actions.

Logic was added to prepare the destination directory by performing a
`mkdir -p` on the user supplied destination. This logic ensures that the
correct pathing is performed by determining the filetype (dir vs file)
on both the source and destination paths. In addition to the supplied
destination path, if the source filesystem is determined to have index
mount directories (i.e. xfs/gfs2), those directories are created on the
destination path to ensure that all data is retained.

- All necessary filesystem interaction is ran through mpirun so that the
  proper mpi worker nodes are used to prepare the destination. This
  required for lustre2lustre data movement.
- These file system interactions (i.e. `stat`, `mkdir`) are performed
  through `setpriv` to ensure that all interactions are done as the
  supplied UID/GID (from the workflow).
- The hostfile createion logic now standsalone since these new mpirun
  calls need the hostfile before the final data movement command is
  built
- Added in some local system calls when using unit tests since mpirun is
  not available in textenv

Signed-off-by: Blake Devcich <blake.devcich@hpe.com>

go.mod

@@ -4,7 +4,7 @@ go 1.19
 
 require (
 	github.com/NearNodeFlash/lustre-fs-operator v0.0.1-0.20240220213720-51597bca637d
-	github.com/NearNodeFlash/nnf-sos v0.0.1-0.20240221214302-e7989177289a
+	github.com/NearNodeFlash/nnf-sos v0.0.1-0.20240403124551-ca6c6a22ce36
 	github.com/onsi/ginkgo/v2 v2.11.0
 	github.com/onsi/gomega v1.27.10
 	github.com/prometheus/client_golang v1.16.0
@@ -27,7 +27,7 @@ require (
 	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/fsnotify/fsnotify v1.6.0 // indirect
-	github.com/go-logr/logr v1.2.4 // indirect
+	github.com/go-logr/logr v1.2.4
 	github.com/go-logr/zapr v1.2.4 // indirect
 	github.com/go-openapi/jsonpointer v0.19.6 // indirect
 	github.com/go-openapi/jsonreference v0.20.2 // indirect
@@ -73,7 +73,7 @@ require (
 )
 
 require (
-	github.com/DataWorkflowServices/dws v0.0.1-0.20240221183421-1a123a9274b6
+	github.com/DataWorkflowServices/dws v0.0.1-0.20240223212516-e29a8a3306e1
 	go.openly.dev/pointy v1.3.0
 )
 

go.sum

@@ -1,11 +1,11 @@
-github.com/DataWorkflowServices/dws v0.0.1-0.20240221183421-1a123a9274b6 h1:LYKIIoawsuo+1ByvQaIpgl8vZc2KrE0q7AE7t0YumrI=
-github.com/DataWorkflowServices/dws v0.0.1-0.20240221183421-1a123a9274b6/go.mod h1:vSTBLWbsFjMYxx+sjMDyZpMXLY9m5Bp73cjnmAL30WU=
+github.com/DataWorkflowServices/dws v0.0.1-0.20240223212516-e29a8a3306e1 h1:BSVqA7mYIzb867DGukYIPjL2lsagvlrZ6xHBwC2VzsU=
+github.com/DataWorkflowServices/dws v0.0.1-0.20240223212516-e29a8a3306e1/go.mod h1:vSTBLWbsFjMYxx+sjMDyZpMXLY9m5Bp73cjnmAL30WU=
 github.com/NearNodeFlash/lustre-fs-operator v0.0.1-0.20240220213720-51597bca637d h1:AP1TgQlneYZT/AxkYFyvJp1j86+7MTYOoo3I1Zw3L2E=
 github.com/NearNodeFlash/lustre-fs-operator v0.0.1-0.20240220213720-51597bca637d/go.mod h1:qBcz9p8sXm1qhDf8WUmhxTlD1NCMEjoAD7NoHbQvMiI=
 github.com/NearNodeFlash/nnf-ec v0.0.0-20231010162453-a8168bb6a52f h1:aWtSSQLLk9mUZj94mowirQeVw9saf80gVe10X0rZe8o=
 github.com/NearNodeFlash/nnf-ec v0.0.0-20231010162453-a8168bb6a52f/go.mod h1:oxdwMqfttOF9dabJhqrWlirCnMk8/8eyLMwl+hducjk=
-github.com/NearNodeFlash/nnf-sos v0.0.1-0.20240221214302-e7989177289a h1:m3lPHiWObITk+zp3GAuOeawkaMr+U4aKlHcblYnsP58=
-github.com/NearNodeFlash/nnf-sos v0.0.1-0.20240221214302-e7989177289a/go.mod h1:4kJuGEwS46EYIt24NmNweaMryYp8M6RabNc8Nd+GIBE=
+github.com/NearNodeFlash/nnf-sos v0.0.1-0.20240403124551-ca6c6a22ce36 h1:c6MtP2ef94a9JfBj3H5E/xSdqXThuIXSMr1PeLqY5jE=
+github.com/NearNodeFlash/nnf-sos v0.0.1-0.20240403124551-ca6c6a22ce36/go.mod h1:T48LZQlkkQbIg5m3uUOevNjnpqP2Tl92cIwYt0/CQTc=
 github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
 github.com/benbjohnson/clock v1.3.0 h1:ip6w0uFQkncKQ979AypyG0ER7mqUSBdKLOgAle/AT8A=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=