Repository: Add the as_path context manager

[sphuber]

The node repository interface intentionally does not provide access to
its file objects through filepaths on the file system. This is because,
for efficiency reasons, the content of a repository may not actually be
stored as individual files on a file system, but for example are stored
in an object store.

Therefore, the contents of the repository can only be retrieved as a
file-like object or read as a string or list of bytes into memory.
Certain use-cases require a file to be made available through a filepath.
An example is when it needs to be passed to an API that only accepts a
filepath, such as numpy.loadfromtxt.

Currently, the user will have to manually copy the content of the repo's
content to a temporary file on disk, and pass the temporary filepath.
This results in clients having to often resport to the following snippet:

import pathlib
import shutil
import tempfile

with tempfile.TemporaryDirectory() as tmp_path:

    # Copy the entire content to the temporary folder
    dirpath = pathlib.Path(tmp_path)
    node.base.repository.copy_tree(dirpath)

    # Or copy the content of a file. Should use streaming
    # to avoid reading everything into memory
    filepath = (dirpath / 'some_file.txt')
    with filepath.open('rb') as target:
        with node.base.repository.open('rb') as source:
            shutil.copyfileobj(source, target)

    # Now use `filepath` to library call, e.g.
    numpy.loadtxt(filepath)

This logic is now provided under the as_path context manager. This
will make it easy to access repository content as files on the local
file system. A warning is added to the docs explaining the inefficiency
of the content having to be read and written to a temporary directory
first, encouraging it only to be used when the alternative is not an
option.

[giovannipizzi]

While simple, this is not a good example IMHO, because it ends up anyway putting everything in memory in handle.read(), defeating the purpose and making it not clear why we have to go via the new interface.

We should thing at something simple but really streaming. E.g. just pseudo code:

CHUNK_SIZE = 65536
with single_file.open() as handle:
    while True:
        chunk = handle.read(CHUNK_SIZE)
        if not chunk: 
            break
        #process chunk here

Or something simple, like counting length:

CHUNK_SIZE = 65536

length = 0
with single_file.open() as handle:
    while True:
        chunk = handle.read(CHUNK_SIZE)
        if not chunk: 
            break
        length += len(chunk)

[sphuber]

I thought about that, but I find this example is quite complex and risks confusing the reader. Maybe something like this:

import shutil

# Copy a large file from repo to file on disk without loading in memory
with single_file.open(mode='rb') as source:
    with open('copy.txt', mode='wb') as target:
        shutil.copyfileobj(source, target)

Think this will be more intuitive and an actual use case for some users.

[giovannipizzi]

OK!

[giovannipizzi]

Same comment as before

[giovannipizzi]

Since you mention above generally shutil.copyfileobj, shouldn't we have an example of that here as well?

[sphuber]

But shutil.copyfileobj takes a file handle, not a filepath. Of course I could then do

with single_file.as_path() as filepath:
    with filepath.open() as source:
        with open('output.txt', 'wb') as target:
            shutil.copyfileobj(source, target)

That would be defeating the point. Or did you mean another method like shutil.copyfileobj but that uses filepaths?