do not copy lustre striping info with dsync

[thiell]

Hi,

dsync cannot always be used from Lustre 2.12 to Lustre 2.10, as it tries to copy lustre striping params. Is there a way to not copy striping info with dsync? IIRC, with dcp, it only tries to copy lustre striping info when -p is provided. We have seen a weird behavior after a user used dsync with PFL'ed files: the result was that the first stripe on the destination filesystem is only using OST 0 (so this OST has become full of course), while the source files are also PFL'ed but their first stripe is spread across all OSTs (normal behavior).

We would like to simply rely on the default striping configuration of the target filesystem. Each filesystem is different. This is related to #49 but more critical I think.

Resulting bogus files:

$ lfs getstripe /oak/stanford/groups/***/***/sherlock_scratch_archive_Feb2020_2/ml/QM9_Stanford_data/transfer_learning_E_ccsdt_avtz_nolinearcorrection_minibatches_batchsize48/ | grep l_ost_idx | head
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x1eb0396:0x0] }
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x1e9e323:0x0] }
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x1eb4f03:0x0] }
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x1eadcd3:0x0] }
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x1eaa3ab:0x0] }
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x1eb3b08:0x0] }
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x1eb9a49:0x0] }
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x1ea5c4b:0x0] }
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x1eaf35e:0x0] }
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x1ea5ad8:0x0] }

[adammoody]

Ugh, thanks for tracking this down and reporting it. I think this happens because dsync copies over the extended attributes verbatim from the source files. There is no command line option to disable copying those in dsync, though that would not be hard to add.

As a work around, you could disable copying of extended attributes by modifying this function to just immediately return 0 and avoid copying any attributes:

mpifileutils/src/common/mfu_flist_copy.c

Line 160 in c90292b

static int mfu_copy_xattrs(

Or we have a CMake option to disable support for extended attributes at configure time. My CMake syntax may be incorrect here:

cmake ... -DENABLE_XATTRS=0

Can you try either of those and let us know if it helps?

Long term, we need to come up with a proper fix here so that we can optionally copy extended attributes, but still exclude copying of the Lustre-related extended attributes. Shorter term, we could likely add an option to skip those on dsync.

[adilger]

This may be a bug in the Lustre xattr code for PFL file layouts - OST offset defaults to 0 when coying a PFL via xattrs.

[akesandgren]

I worked around it by filtering out lov.lustre xattrs completely in mfu_copy_xattrs. But my reason for doing so was that I wanted the new file systems PFL layout to take effect and not whatever was on the old file system.

That's likely not the correct thing to do...

[adilger]

I think this needs to be a runtime option, since there are definitely users that want to copy the layout. For "lfs_migrate" there is a "--restripe" option that uses the default layout of the target instead of copying the source layout xattr.

[akesandgren]

Yes, definitely, I just needed a quick fix since I hit the https://jira.whamcloud.com/browse/LU-13062 problem when testing.

What I've seen just now is that the fiemap call fails on anything that uses > 1 stripe, even without a PFL layout.

Is that expected too?

[adilger]

Yes, FIEMAP does not use the standard interface for files > 1 stripe because it would return a different extent for each 1MB (or other stripe_size) of the file, which would grow large rather quickly.

For efficiently copying files with holes, we are planning to use SEEK_HOLE and SEEK_DATA, which is being implemented in https://jira.whamcloud.com/browse/LU-10810 but has not landed yet. This interface doesn't really care about the underlying file layout.

[ofaaland]

Hi @adilger @akesandgren @thiell please see #503. I believe that PR allows users sufficient control over whether xattrs are copied (and whether striping is preserved), but it would be great if you could confirm that. And if you're in the mood for a full review, that would be great, too. Thank you!