[copier] minor clean up to copier #13820
Conversation
danking
commented
Oct 16, 2023
danking
commented
- File rate is more interesting for small files.
- The source_report controls the progress bar. By updating it eagerly while we are listing a directory, the progress bar is more accurate sooner. We currently wait until we get a semaphore for a particular file to update the progress bar.
1. File rate is more interesting for small files. 2. The source_report controls the progress bar. By updating it eagerly while we are listing a directory, the progress bar is more accurate sooner. We currently wait until we get a semaphore for a particular file to update the progress bar.
| async for srcentry in srcentries] | ||
| copy_thunks = [] | ||
| async for srcentry in srcentries: | ||
| bytes_to_copy += await (await srcentry.status()).size() # this is almost never a syscall/net-request (afaik: only local symlinks) |
There was a problem hiding this comment.
I don't understand the end of this comment. What do you mean this does a symlink?
There was a problem hiding this comment.
Getting the status and the size of an entry of a directory listing, AFAICT, only requires a system call or network request in one special case:
- You have a LocalFS
- The file in question is a symlink.
Linux doesn't eagerly retrieve the size of the target of a symlink (I think because it could be an arbitrary number of symlinks away).
There was a problem hiding this comment.
That's why status and size are async. If it wasn't for the local fs, they could be sync.
Maybe a better comment is:
In cloud FSes, status and size never make a network request. In local FS, they can make system calls on symlinks. This line will be fairly expensive if copying a tree with a lot of symlinks.
