-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data Movement: different behavior when destination directories do not exist #130
Comments
I think this becomes a problem of understanding $ tree
.
|-- dest
`-- src
|-- node-0
| `-- data.out
`-- node-1
`-- data.out
4 directories, 2 files
$ dcp src/node-0 dest/my-job
...
$ tree
.
|-- dest
| `-- my-job
| `-- data.out
`-- src
|-- node-0
| `-- data.out
`-- node-1
`-- data.out
5 directories, 3 files
$ dcp src/node-1 dest/my-job
...
$ tree
.
|-- dest
| `-- my-job
| |-- data.out
| `-- node-1
| `-- data.out
`-- src
|-- node-0
| `-- data.out
`-- node-1
`-- data.out
6 directories, 4 files |
This begs the question: should the destination directories be required to preexist? If you were to go one level deeper, dcp src/node-0 dest/my-job/rank0
[2024-02-16T20:34:50] [0] [/deps/mpifileutils/src/common/mfu_param_path.c:582] ERROR: Destination parent directory is not writable `/home/mpiuser/dest/my-job' (errno=2 No such file or directory)
[2024-02-16T20:34:50] [0] [/deps/mpifileutils/src/dcp/dcp.c:479] ERROR: Invalid src/dest paths provided. Exiting run: MFU_ERR(-1001) If this was a really long running job with a |
Might be related: hpc/mpifileutils#416 |
As discussed in the Flux meeting today, we need to investigate doing two things:
Creating the directory to ensure the destination exists is required behavior to work around how We are in agreement that we need a way to head off user mistakes early on otherwise the data movement could fail during the CopyOut. That mkdir could fail if the user does not have permission to create the directory in the location given. Things could also change between 1 and 2 where the user application changes directory structure (for example) - not exactly sure what we can do there. We can't guard against everything the user application is capable of. Additional idea: lost+found After further discussion on our end, we've decided to explore implementing lost+found alongside step No. 2. In the event that step 2 or the data movement encounters any issues, lost+found will be activated to salvage the data. Given the potential for various events to occur between steps 1 and 2, we believe that performing validation upfront is unnecessary. With the implementation of lost+found, users will have the option to retrieve their data if they encounter any issues. |
mkdir function has been added along with ensuring index mount directories are created on the destination when copying out from gfs2/xfs filesystems. |
Closing this and opened #151 for the lost+found part. |
When doing something like the following where the
my-job/
directory does not exist at/lus/global/user/
(the destination for thecopy_out
directive):The resulting copy out operation can look like this:
We get 4 total files, but one of them is at the root level of
my-job/
and the index mount directory (i.e. rabbit-node-1-0) did not get copied over. In this case, no harm, no foul (but confusing) since all thedata.out
files are present. But this can also result in something like this:Not good.
To break this down, the job/workflow will end up creating 4 NnfDatamovements in the
DataOut
state since it is ran on 4 compute nodes. That is, 4 different data movement operations to move each compute nodes' data from the rabbit to the global lustre filesystem. Those 4 data movement will run (almost) in parallel. They would look something like this for 2 computes per rabbit:One (or more) of these operations will win. That first time through, the
my-job
directory does not exist. So that firstdcp
operation is essentially a directory copy:Since
my-job
doesn't exist it, the contents of the source are copied directly to themy-job
directory. This is the reason for the lonedata.out
at the root level above.Afterwards, each subsequent
dcp
operation will perform the same request, but that directory now exists. Which results in the index mount directory being copied over.The text was updated successfully, but these errors were encountered: