dsync/dcmp: support symlinks targets changes

[rezib]

Update dsync and dcmp to detect a symlinks targets changes. In this case, the symlinks are reported to be different by dcmp and are updated by dsync.

Note this works for dsync both with or without c, --contents.

New function mfu_compare_symlinks() is introduced in library API to factorize the logic for all commands.

Method

In order to reproduce the bug and validate the patch, I wrote this little script symlink.sh:

#!/bin/sh

# Prepare test environment
DATA_DIR=../data

# Remove previous data if existing
if [ -d ${DATA_DIR}/from ]; then
	rm -r ${DATA_DIR}/from
fi
if [ -d ${DATA_DIR}/to ]; then
	rm -r ${DATA_DIR}/to
fi

echo "→ Initialiazing testing data"
mkdir ${DATA_DIR}/from
echo foo > ${DATA_DIR}/from/file1
echo bar > ${DATA_DIR}/from/file2
ln -s file1 ${DATA_DIR}/from/link1
ln -s file2 ${DATA_DIR}/from/link2

echo
echo "→ Launching first dsync"
mpirun -N 2 ${HOME}/bin/dsync ${DATA_DIR}/from ${DATA_DIR}/to

# Change one symlink target
rm ${DATA_DIR}/from/link2
ln -s file1 ${DATA_DIR}/from/link2

echo
echo "→ Launching first dcmp"
mpirun -N 2 ${HOME}/bin/dcmp ${DATA_DIR}/from ${DATA_DIR}/to

echo
echo "→ Launching second dsync"
mpirun -N 2 ${HOME}/bin/dsync ${DATA_DIR}/from ${DATA_DIR}/to

echo
echo "→ Launching second dcmp"
mpirun -N 2 ${HOME}/bin/dcmp ${DATA_DIR}/from ${DATA_DIR}/to

Before

Script output:

$ sh symlink.sh 
→ Initialiazing testing data

→ Launching first dsync
[2024-12-20T08:36:36] Walking source path
[2024-12-20T08:36:36] Walking /home/remi/git/data/from
[2024-12-20T08:36:36] Walked 5 items in 0.001 secs (4545.934 items/sec) ...
[2024-12-20T08:36:36] Walked 5 items in 0.001 seconds (4353.793 items/sec)
[2024-12-20T08:36:36] Walking destination path
[2024-12-20T08:36:36] Walking /home/remi/git/data/to
[2024-12-20T08:36:36] [0] [/home/remi/git/mpifileutils/src/common/mfu_flist_walk.c:516] ERROR: Failed to stat: '/home/remi/git/data/to' (errno=2 No such file or directory)
[2024-12-20T08:36:36] Walked 0 items in 0.001 secs (0.000 items/sec) ...
[2024-12-20T08:36:36] Walked 0 items in 0.001 seconds (0.000 items/sec)
[2024-12-20T08:36:36] Started   : Dec-20-2024, 08:36:36
[2024-12-20T08:36:36] Completed : Dec-20-2024, 08:36:36
[2024-12-20T08:36:36] Seconds   : 0.000
[2024-12-20T08:36:36] Items     : 0
[2024-12-20T08:36:36] Item Rate : 0 items in 0.000295 seconds (0.000000 items/sec)
[2024-12-20T08:36:36] Copying items to destination
[2024-12-20T08:36:36] Copying to /home/remi/git/data/to
[2024-12-20T08:36:36] Items: 5
[2024-12-20T08:36:36]   Directories: 1
[2024-12-20T08:36:36]   Files: 2
[2024-12-20T08:36:36]   Links: 2
[2024-12-20T08:36:36] Data: 8.000 B (4.000 B per file)
[2024-12-20T08:36:36] Creating 1 directories
[2024-12-20T08:36:36] Creating 4 files.
[2024-12-20T08:36:36] Copying data.
[2024-12-20T08:36:36] Copy data: 8.000 B (8 bytes)
[2024-12-20T08:36:36] Copy rate: 693.124 B/s (8 bytes in 0.012 seconds)
[2024-12-20T08:36:36] Syncing data to disk.
[2024-12-20T08:36:36] Sync completed in 0.011 seconds.
[2024-12-20T08:36:36] Setting ownership, permissions, and timestamps.
[2024-12-20T08:36:36] Updated 5 items in 0.000 seconds (25281.382 items/sec)
[2024-12-20T08:36:36] Syncing directory updates to disk.
[2024-12-20T08:36:36] Sync completed in 0.002 seconds.
[2024-12-20T08:36:36] Started: Dec-20-2024,08:36:36
[2024-12-20T08:36:36] Completed: Dec-20-2024,08:36:36
[2024-12-20T08:36:36] Seconds: 0.024
[2024-12-20T08:36:36] Items: 5
[2024-12-20T08:36:36]   Directories: 1
[2024-12-20T08:36:36]   Files: 2
[2024-12-20T08:36:36]   Links: 2
[2024-12-20T08:36:36] Data: 8.000 B (8 bytes)
[2024-12-20T08:36:36] Rate: 326.574 B/s (008 bytes in 0.024 seconds)
[2024-12-20T08:36:36] Updating timestamps on newly copied files
[2024-12-20T08:36:36] Completed updating timestamps
[2024-12-20T08:36:36] Completed sync

→ Launching first dcmp
[2024-12-20T08:36:37] Walking source path
[2024-12-20T08:36:37] Walking /home/remi/git/data/from
[2024-12-20T08:36:37] Walked 5 items in 0.001 secs (4468.399 items/sec) ...
[2024-12-20T08:36:37] Walked 5 items in 0.001 seconds (4269.239 items/sec)
[2024-12-20T08:36:37] Walking destination path
[2024-12-20T08:36:37] Walking /home/remi/git/data/to
[2024-12-20T08:36:37] Walked 5 items in 0.001 secs (7051.838 items/sec) ...
[2024-12-20T08:36:37] Walked 5 items in 0.001 seconds (6870.501 items/sec)
[2024-12-20T08:36:37] Comparing items
[2024-12-20T08:36:37] Comparing file contents
[2024-12-20T08:36:37] Started   : Dec-20-2024, 08:36:37
[2024-12-20T08:36:37] Completed : Dec-20-2024, 08:36:37
[2024-12-20T08:36:37] Seconds   : 0.010
[2024-12-20T08:36:37] Items     : 5
[2024-12-20T08:36:37] Item Rate : 5 items in 0.010120 seconds (494.092677 items/sec)
[2024-12-20T08:36:37] Bytes read: 16.000 B (16 bytes)
[2024-12-20T08:36:37] Byte Rate : 1.544 KiB/s (016 bytes in 0.010 seconds)
Number of items that exist in both directories: 5 (Src: 5 Dest: 5)
Number of items that exist only in one directory: 0 (Src: 0 Dest: 0)
Number of items that exist in both directories and have the same type: 5 (Src: 5 Dest: 5)
Number of items that exist in both directories and have different types: 0 (Src: 0 Dest: 0)
Number of items that exist in both directories and have the same content: 5 (Src: 5 Dest: 5)
Number of items that exist in both directories and have different contents: 0 (Src: 0 Dest: 0)

→ Launching second dsync
[2024-12-20T08:36:38] Walking source path
[2024-12-20T08:36:38] Walking /home/remi/git/data/from
[2024-12-20T08:36:38] Walked 5 items in 0.001 secs (6070.945 items/sec) ...
[2024-12-20T08:36:38] Walked 5 items in 0.001 seconds (5802.053 items/sec)
[2024-12-20T08:36:38] Walking destination path
[2024-12-20T08:36:38] Walking /home/remi/git/data/to
[2024-12-20T08:36:38] Walked 5 items in 0.001 secs (9796.104 items/sec) ...
[2024-12-20T08:36:38] Walked 5 items in 0.001 seconds (9540.583 items/sec)
[2024-12-20T08:36:38] Comparing file sizes and modification times of 2 items
[2024-12-20T08:36:38] Started   : Dec-20-2024, 08:36:38
[2024-12-20T08:36:38] Completed : Dec-20-2024, 08:36:38
[2024-12-20T08:36:38] Seconds   : 0.000
[2024-12-20T08:36:38] Items     : 2
[2024-12-20T08:36:38] Item Rate : 2 items in 0.000189 seconds (10564.682267 items/sec)
[2024-12-20T08:36:38] Updating timestamps on newly copied files
[2024-12-20T08:36:38] Completed updating timestamps
[2024-12-20T08:36:38] Completed sync

→ Launching second dcmp
[2024-12-20T08:36:39] Walking source path
[2024-12-20T08:36:39] Walking /home/remi/git/data/from
[2024-12-20T08:36:39] Walked 5 items in 0.002 secs (2566.423 items/sec) ...
[2024-12-20T08:36:39] Walked 5 items in 0.002 seconds (2456.619 items/sec)
[2024-12-20T08:36:39] Walking destination path
[2024-12-20T08:36:39] Walking /home/remi/git/data/to
[2024-12-20T08:36:39] Walked 5 items in 0.001 secs (3797.583 items/sec) ...
[2024-12-20T08:36:39] Walked 5 items in 0.001 seconds (3672.312 items/sec)
[2024-12-20T08:36:39] Comparing items
[2024-12-20T08:36:39] Comparing file contents
[2024-12-20T08:36:39] Started   : Dec-20-2024, 08:36:39
[2024-12-20T08:36:39] Completed : Dec-20-2024, 08:36:39
[2024-12-20T08:36:39] Seconds   : 0.001
[2024-12-20T08:36:39] Items     : 5
[2024-12-20T08:36:39] Item Rate : 5 items in 0.000552 seconds (9062.173762 items/sec)
[2024-12-20T08:36:39] Bytes read: 16.000 B (16 bytes)
[2024-12-20T08:36:39] Byte Rate : 28.319 KiB/s (016 bytes in 0.001 seconds)
Number of items that exist in both directories: 5 (Src: 5 Dest: 5)
Number of items that exist only in one directory: 0 (Src: 0 Dest: 0)
Number of items that exist in both directories and have the same type: 5 (Src: 5 Dest: 5)
Number of items that exist in both directories and have different types: 0 (Src: 0 Dest: 0)
Number of items that exist in both directories and have the same content: 5 (Src: 5 Dest: 5)
Number of items that exist in both directories and have different contents: 0 (Src: 0 Dest: 0)

The content in files tree:

$ ls ../data/from/ -lh
total 8,0K
-rw-rw-r-- 1 remi remi 4 déc.  20 08:36 file1
-rw-rw-r-- 1 remi remi 4 déc.  20 08:36 file2
lrwxrwxrwx 1 remi remi 5 déc.  20 08:36 link1 -> file1
lrwxrwxrwx 1 remi remi 5 déc.  20 08:36 link2 -> file1
$ ls ../data/to/ -lh
total 8,0K
-rw-rw-r-- 1 remi remi 4 déc.  20 08:36 file1
-rw-rw-r-- 1 remi remi 4 déc.  20 08:36 file2
lrwxrwxrwx 1 remi remi 5 déc.  20 08:36 link1 -> file1
lrwxrwxrwx 1 remi remi 5 déc.  20 08:36 link2 -> file2

You can see that link2 still points to file2 despite the second dsync run.

After

Script output:

$ sh symlink.sh 
→ Initialiazing testing data

→ Launching first dsync
[2024-12-20T08:34:33] Walking source path
[2024-12-20T08:34:33] Walking /home/remi/git/data/from
[2024-12-20T08:34:33] Walked 5 items in 0.003 secs (1832.340 items/sec) ...
[2024-12-20T08:34:33] Walked 5 items in 0.003 seconds (1760.092 items/sec)
[2024-12-20T08:34:33] Walking destination path
[2024-12-20T08:34:33] Walking /home/remi/git/data/to
[2024-12-20T08:34:33] [0] [/home/remi/git/mpifileutils/src/common/mfu_flist_walk.c:516] ERROR: Failed to stat: '/home/remi/git/data/to' (errno=2 No such file or directory)
[2024-12-20T08:34:33] Walked 0 items in 0.002 secs (0.000 items/sec) ...
[2024-12-20T08:34:33] Walked 0 items in 0.002 seconds (0.000 items/sec)
[2024-12-20T08:34:33] Started   : Dec-20-2024, 08:34:33
[2024-12-20T08:34:33] Completed : Dec-20-2024, 08:34:33
[2024-12-20T08:34:33] Seconds   : 0.001
[2024-12-20T08:34:33] Items     : 0
[2024-12-20T08:34:33] Item Rate : 0 items in 0.000680 seconds (0.000000 items/sec)
[2024-12-20T08:34:33] Copying items to destination
[2024-12-20T08:34:33] Copying to /home/remi/git/data/to
[2024-12-20T08:34:33] Items: 5
[2024-12-20T08:34:33]   Directories: 1
[2024-12-20T08:34:33]   Files: 2
[2024-12-20T08:34:33]   Links: 2
[2024-12-20T08:34:33] Data: 8.000 B (4.000 B per file)
[2024-12-20T08:34:33] Creating 1 directories
[2024-12-20T08:34:33] Creating 4 files.
[2024-12-20T08:34:33] Copying data.
[2024-12-20T08:34:33] Copy data: 8.000 B (8 bytes)
[2024-12-20T08:34:33] Copy rate: 631.445 B/s (8 bytes in 0.013 seconds)
[2024-12-20T08:34:33] Syncing data to disk.
[2024-12-20T08:34:33] Sync completed in 0.016 seconds.
[2024-12-20T08:34:33] Setting ownership, permissions, and timestamps.
[2024-12-20T08:34:33] Updated 5 items in 0.000 seconds (25938.453 items/sec)
[2024-12-20T08:34:33] Syncing directory updates to disk.
[2024-12-20T08:34:33] Sync completed in 0.001 seconds.
[2024-12-20T08:34:33] Started: Dec-20-2024,08:34:33
[2024-12-20T08:34:33] Completed: Dec-20-2024,08:34:33
[2024-12-20T08:34:33] Seconds: 0.032
[2024-12-20T08:34:33] Items: 5
[2024-12-20T08:34:33]   Directories: 1
[2024-12-20T08:34:33]   Files: 2
[2024-12-20T08:34:33]   Links: 2
[2024-12-20T08:34:33] Data: 8.000 B (8 bytes)
[2024-12-20T08:34:33] Rate: 253.288 B/s (008 bytes in 0.032 seconds)
[2024-12-20T08:34:33] Updating timestamps on newly copied files
[2024-12-20T08:34:33] Completed updating timestamps
[2024-12-20T08:34:33] Completed sync

→ Launching first dcmp
[2024-12-20T08:34:34] Walking source path
[2024-12-20T08:34:34] Walking /home/remi/git/data/from
[2024-12-20T08:34:34] Walked 5 items in 0.001 secs (5848.090 items/sec) ...
[2024-12-20T08:34:34] Walked 5 items in 0.001 seconds (5429.672 items/sec)
[2024-12-20T08:34:34] Walking destination path
[2024-12-20T08:34:34] Walking /home/remi/git/data/to
[2024-12-20T08:34:34] Walked 5 items in 0.001 secs (9674.211 items/sec) ...
[2024-12-20T08:34:34] Walked 5 items in 0.001 seconds (9396.977 items/sec)
[2024-12-20T08:34:34] Comparing items
[2024-12-20T08:34:34] Comparing file contents
[2024-12-20T08:34:34] Started   : Dec-20-2024, 08:34:34
[2024-12-20T08:34:34] Completed : Dec-20-2024, 08:34:34
[2024-12-20T08:34:34] Seconds   : 0.010
[2024-12-20T08:34:34] Items     : 5
[2024-12-20T08:34:34] Item Rate : 5 items in 0.010105 seconds (494.798970 items/sec)
[2024-12-20T08:34:34] Bytes read: 16.000 B (16 bytes)
[2024-12-20T08:34:34] Byte Rate : 1.546 KiB/s (016 bytes in 0.010 seconds)
Number of items that exist in both directories: 5 (Src: 5 Dest: 5)
Number of items that exist only in one directory: 0 (Src: 0 Dest: 0)
Number of items that exist in both directories and have the same type: 5 (Src: 5 Dest: 5)
Number of items that exist in both directories and have different types: 0 (Src: 0 Dest: 0)
Number of items that exist in both directories and have the same content: 4 (Src: 4 Dest: 4)
Number of items that exist in both directories and have different contents: 1 (Src: 1 Dest: 1)

→ Launching second dsync
[2024-12-20T08:34:35] Walking source path
[2024-12-20T08:34:35] Walking /home/remi/git/data/from
[2024-12-20T08:34:35] Walked 5 items in 0.001 secs (5686.063 items/sec) ...
[2024-12-20T08:34:35] Walked 5 items in 0.001 seconds (5300.288 items/sec)
[2024-12-20T08:34:35] Walking destination path
[2024-12-20T08:34:35] Walking /home/remi/git/data/to
[2024-12-20T08:34:35] Walked 5 items in 0.001 secs (9298.987 items/sec) ...
[2024-12-20T08:34:35] Walked 5 items in 0.001 seconds (9030.993 items/sec)
[2024-12-20T08:34:35] Comparing file sizes and modification times of 4 items
[2024-12-20T08:34:35] Started   : Dec-20-2024, 08:34:35
[2024-12-20T08:34:35] Completed : Dec-20-2024, 08:34:35
[2024-12-20T08:34:35] Seconds   : 0.000
[2024-12-20T08:34:35] Items     : 4
[2024-12-20T08:34:35] Item Rate : 4 items in 0.000298 seconds (13433.908529 items/sec)
[2024-12-20T08:34:35] Deleting items from destination
[2024-12-20T08:34:35] Removing 1 items
[2024-12-20T08:34:35] Removed 1 items in 0.000 seconds (3086.277 items/sec)
[2024-12-20T08:34:35] Copying items to destination
[2024-12-20T08:34:35] Copying to /home/remi/git/data/to
[2024-12-20T08:34:35] Items: 1
[2024-12-20T08:34:35]   Directories: 0
[2024-12-20T08:34:35]   Files: 0
[2024-12-20T08:34:35]   Links: 1
[2024-12-20T08:34:35] Data: 0.000 B (0.000 B per file)
[2024-12-20T08:34:35] Creating 1 files.
[2024-12-20T08:34:35] Copying data.
[2024-12-20T08:34:35] Copy data: 0.000 B (0 bytes)
[2024-12-20T08:34:35] Copy rate: 0.000 B/s (0 bytes in 0.000 seconds)
[2024-12-20T08:34:35] Syncing data to disk.
[2024-12-20T08:34:35] Sync completed in 0.012 seconds.
[2024-12-20T08:34:35] Setting ownership, permissions, and timestamps.
[2024-12-20T08:34:35] Updated 1 items in 0.000 seconds (6755.935 items/sec)
[2024-12-20T08:34:35] Syncing directory updates to disk.
[2024-12-20T08:34:35] Sync completed in 0.003 seconds.
[2024-12-20T08:34:35] Started: Dec-20-2024,08:34:35
[2024-12-20T08:34:35] Completed: Dec-20-2024,08:34:35
[2024-12-20T08:34:35] Seconds: 0.016
[2024-12-20T08:34:35] Items: 1
[2024-12-20T08:34:35]   Directories: 0
[2024-12-20T08:34:35]   Files: 0
[2024-12-20T08:34:35]   Links: 1
[2024-12-20T08:34:35] Data: 0.000 B (0 bytes)
[2024-12-20T08:34:35] Rate: 0.000 B/s (000 bytes in 0.016 seconds)
[2024-12-20T08:34:35] Updating timestamps on newly copied files
[2024-12-20T08:34:35] Completed updating timestamps
[2024-12-20T08:34:35] Completed sync

→ Launching second dcmp
[2024-12-20T08:34:36] Walking source path
[2024-12-20T08:34:36] Walking /home/remi/git/data/from
[2024-12-20T08:34:36] Walked 5 items in 0.002 secs (2460.492 items/sec) ...
[2024-12-20T08:34:36] Walked 5 items in 0.002 seconds (2283.881 items/sec)
[2024-12-20T08:34:36] Walking destination path
[2024-12-20T08:34:36] Walking /home/remi/git/data/to
[2024-12-20T08:34:36] Walked 5 items in 0.001 secs (3496.357 items/sec) ...
[2024-12-20T08:34:36] Walked 5 items in 0.001 seconds (3455.165 items/sec)
[2024-12-20T08:34:36] Comparing items
[2024-12-20T08:34:36] Comparing file contents
[2024-12-20T08:34:36] Started   : Dec-20-2024, 08:34:36
[2024-12-20T08:34:36] Completed : Dec-20-2024, 08:34:36
[2024-12-20T08:34:36] Seconds   : 0.000
[2024-12-20T08:34:36] Items     : 5
[2024-12-20T08:34:36] Item Rate : 5 items in 0.000320 seconds (15621.143530 items/sec)
[2024-12-20T08:34:36] Bytes read: 16.000 B (16 bytes)
[2024-12-20T08:34:36] Byte Rate : 48.816 KiB/s (016 bytes in 0.000 seconds)
Number of items that exist in both directories: 5 (Src: 5 Dest: 5)
Number of items that exist only in one directory: 0 (Src: 0 Dest: 0)
Number of items that exist in both directories and have the same type: 5 (Src: 5 Dest: 5)
Number of items that exist in both directories and have different types: 0 (Src: 0 Dest: 0)
Number of items that exist in both directories and have the same content: 5 (Src: 5 Dest: 5)
Number of items that exist in both directories and have different contents: 0 (Src: 0 Dest: 0)

The content in files tree:

$ ls ../data/from/ -lh
total 8,0K
-rw-rw-r-- 1 remi remi 4 déc.  20 08:34 file1
-rw-rw-r-- 1 remi remi 4 déc.  20 08:34 file2
lrwxrwxrwx 1 remi remi 5 déc.  20 08:34 link1 -> file1
lrwxrwxrwx 1 remi remi 5 déc.  20 08:34 link2 -> file1
$ ls ../data/to/ -lh
total 8,0K
-rw-rw-r-- 1 remi remi 4 déc.  20 08:34 file1
-rw-rw-r-- 1 remi remi 4 déc.  20 08:34 file2
lrwxrwxrwx 1 remi remi 5 déc.  20 08:34 link1 -> file1
lrwxrwxrwx 1 remi remi 5 déc.  20 08:34 link2 -> file1

You can see that link2 now points to file1 after the second dsync run.

Note

I would like to emphasize that this work is sponsored by @cea-hpc.

fix #412

[rezib]

Is there anything I can do to help you merge this PR?

[carbonneau1]

Salut Rémi,
vous devrez soumettre à nouveau votre commit signé.

Eric

[rezib]

Salut Eric,

Oops, I missed that requirement, I just force-pushed the commit signed with my GPG key.

[ofaaland]

Thanks for this contribution!

I have a couple of questions about the code.

Have you done any testing to determine the performance impact of this change? I agree with the change generally, but if the impact is significant we should warn users in the release notes.

[rezib]

Hello Eric, Olaf,

Thank you for this review!

I pushed an update with some changes to address your concerns. Let me know if there is anything wrong left!

Have you done any testing to determine the performance impact of this change? I agree with the change generally, but if the impact is significant we should warn users in the release notes.

TBH, no I didn't because personally I don't have access to an infrastructure where I can run significant performance tests…

[ofaaland]

LGTM. Thanks!

[ofaaland]

Have you done any testing to determine the performance impact of this change? I agree with the change generally, but if the impact is significant we should warn users in the release notes.

TBH, no I didn't because personally I don't have access to an infrastructure where I can run significant performance tests…

Eric, please do some brief before/after performance testing. Thanks!

[carbonneau1]

I am reviewing the code. If everything is fine, I will run performance test by tomorrow and merge your fix in.

[carbonneau1]

Looks good. I will tests for performance and if everything looks ok I will merge in your feature.

[carbonneau1]

I tested both feature and performance. Everything looked good. It's in.