Users are reporting incomplete downloads

[d3v-null]

This may be a scratch issue, but some users are reporting that giant-squid download will return succesfully, but upon inspecting the output files, some have null access times.

In the design of download jobs, I recommend checking that the contents of the archive are extracted succesfully using sha1sum -c birli_manifest.sha1sum. This will ensure that all file access times are updated, and those files don't get purged.

[d3v-null]

it may be extremely over-engineered, but so far this method of downloading observations has survived thousands of observations with no file corruption issues.

https://github.com/MWATelescope/MWAEoR-Pipeline/blob/065c38e10922d2f46ed6ace6cb24f4e8f980c56a/main.nf#L815

[gsleap]

Looks like this is nothing to do with /scratch. mtime AND atime are both 0 even in my local filesystem.

Looks like this issue is why mtime no longer gets propagated from the files within the tar. It looks like this is intended behaviour from the tar crate :-( . Looking to see what can be done....