Implement shutils equivalent (clutils) that has the same API but accepts AnyPath

[ClementPinard]

Hi, thanks for your awesome library ! Have been searching for a way to use both local and cloud path seamlessy and here it is :D

One particular problem I have with AnyPath is that since local paths are redirected to Pathlib, some function are not available, especially copy and copytree. the reason is that the function to copy are located in the module shutil and not associated to the a class method.

As a consequence we can easily copy one cloud folder to another cloud platform, but we cannot easily upload a whole folder from your local filesystem. path.copy is not available and shutil.copy obviously only works on strings and Path objects.

The two ways I see to solve the problem is either to have a LocalPath class (that basically clones PathLib except for copy methods that are compatible to a cloud destination), or to propose a custom shutil module (from cloudpathlib import AnyPath, shutil) that would give a standalone shutil.copytree function that would be compatible with any path object, local or cloud.

What's your opinion on this ?

PS: for now, I use the path.upload_from function, which works but only for files (and is not very similar to shutil.copy) and I am in a position where a copytree function that would not require me to code a custome function would be nice.

Thanks !

Clément

[pjbull]

Yeah, I definitely appreciate that copying (especially trees) is annoying when using the pathlib.Path object. I do wish the built-in had copy and copytree methods.

That said, I'm not inclined to develop, support, and maintain our own LocalPath object for a couple reasons. (1) It is in some sense out-of-scope since this is lib is designed to make working with cloud paths as easy as working with local paths (2) subclassing Path is actually much trickier than you might think (we did look at this and supporting os.PathLike make more sense), and (3) the simplicity of just getting either a standard pathlib.Path object or a CloudPath from AnyPath makes the API simpler for other devs to pick up and use this lib.

Like you mention, I could see having a shutils equivalent (e.g., clutils) that implemented copytree and copy in a way that handles both CloudPaths and Paths elegantly. Definitely easier to implement and support since it just routes to the appropriate functionality that already exists.

That said, I'm happy to keep this open for discussion and other opinions.

---

PS: for now, I use the path.upload_from function, which works but only for files

Separately, upload_from should work with directories as well. It is the currently supported API to upload a file or directory. If not, there is likely a separate bug worth filing as long as it is not a duplicate of #148

[jayqi]

Having shutils-equivalent functions is also my favored solution to this problem.

---

This is also partly related to some of the typing issues we've encountered with AnyPath (see #146). I think the right way to conceptualize the typing for AnyPath is that it's the intersection of the inferfaces of Path and CloudPath. That way, static type checking would complain if you tried to use copy or copytree on something instantiated with AnyPath.

[ClementPinard]

Thanks for the answer !

I do wish the built-in had copy and copytree methods.
Agreed, it was part of path.py, but was stripped down when converted into PathLib, in an effort to only keep things path related.

I also the method that would make CloudPath the closest to actual PathLib is the custom shutils module. Should this custome module be created, are they anymore method in shutilsthat we should care about if we really wanted to make CloudPath act as if it was an actual Path ?
https://docs.python.org/3/library/shutil.html

An interesting solution can be found here : https://pypi.org/project/libs3/

Feel like ignore_patterns would be a nice to have too.

PS: for now, I use the path.upload_from function, which works but only for files

Separately, upload_from should work with directories as well. It is the currently supported API to upload a file or directory. If not, there is likely a separate bug worth filing as long as it is not a duplicate of #148

Ha my bad, just assumed it was only for files, I feel dumb ! Thanks for that ! 😅