ENH: pathlib -> universal_pathlib to enable cloud (and other filesystem) support

[akhanf]

This PR makes a relatively small change to the code that enables pybids to parse any filesystem backend that is supported by universal_pathlib (which is a subset of those supported by fsspec), by dropping in UPath instead of Path.

The list of filesystems currently supported (from https://github.com/fsspec/universal_pathlib) is:

• file: Local filesystem
• memory: Ephemeral filesystem in RAM
• az:, adl:, abfs: and abfss: Azure Storage (requires adlfs)
• data: RFC 2397 style data URLs (requires fsspec>=2023.12.2)
• github: GitHub repository filesystem
• http: and https: HTTP(S)-based filesystem
• hdfs: Hadoop distributed filesystem
• gs: and gcs: Google Cloud Storage (requires gcsfs)
• s3: and s3a: AWS S3 (requires s3fs to be installed)
• webdav, webdav+http: and webdav+https: WebDAV-based filesystem on top of
  HTTP(S) (requires webdav4[fsspec])

I realize there has been a previous spin-off to enable S3 support (https://github.com/nrdg/cloud_bids_layout) but that has been inactive for ~4 years, and the proposed solution here would do the job more seamlessly, by simply being able to consume paths that include a uri prefix to designate the filesystem (e.g. s3:// or gcs://). There was also an issue raised to request S3 support a while back that became stale and was closed #45.

E.g. one specific use-cases this PR could enable is applying pybids directly to openneuro datasets using the s3 uri:
layout = BIDSLayout('s3://openneuro.org/ds005360')

Filenames from the layout could then be consumed by any tool that support fsspec (pandas, dask, etc). Note: it looks like nibabel supports loading image from URLs, but does not yet support fsspec (nipy/nibabel#1029), so use-cases for nifti data might be limited currently.

My personal use-case is for BIDS microscopy data with OME-Zarr datasets, to allow BIDS apps to consume them directly from the cloud (e.g. using Dask or Zarr libraries).

Re: this PR, I believe it shouldn't change how pybids works if a prefix is not used (eg the tests currently pass), but someone more familiar with the pybids internals would have to confirm.

Keen to hear what others think!

[codecov]

Codecov Report

Attention: Patch coverage is 91.30435% with 2 lines in your changes missing coverage. Please review.

Project coverage is 89.80%. Comparing base (d442c8b) to head (80b9391).
Report is 5 commits behind head on master.

Files	Patch %	Lines
bids/layout/models.py	75.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1074   +/-   ##
=======================================
  Coverage   89.80%   89.80%           
=======================================
  Files          63       63           
  Lines        7177     7179    +2     
  Branches     1374     1374           
=======================================
+ Hits         6445     6447    +2     
  Misses        532      532           
  Partials      200      200           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[effigies]

I'll be honest, this makes me kind of nervous. PyBIDS is full of moving parts.

That said, the tests passing on pre-3.12 Python is surprising and suggests the authors really know what they're doing.

@adelavega thoughts?

[adelavega]

This is pretty neat! I share some of Chris's concerns though.

I wonder if we could hedge, by only using UPath for non-standard paths. Thus ensuring legacy behavior for 99% of users but enabling new paths. Maybe that's overkill though, not sure.

[effigies]

IDK how they did it, but on Python 3.8:

Python 3.8.17 | packaged by conda-forge | (default, Jun 16 2023, 07:06:00) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.12.3 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from pathlib import Path

In [2]: from upath import UPath

In [3]: isinstance(UPath('/tmp'), Path)
Out[3]: True

In [4]: Path
Out[4]: pathlib.Path

In [5]: Path('/tmp')
Out[5]: PosixPath('/tmp')

In [6]: UPath('/tmp')
Out[6]: PosixUPath('/tmp')

I'm kind of inclined to merge, release and then roll back if bug reports start coming in.

[adelavega]

I'm happy with that. It seems like an easy win

…

On Thu, Jul 25, 2024, 2:59 PM Chris Markiewicz ***@***.***> wrote: IDK how they did it, but on Python 3.8: Python 3.8.17 | packaged by conda-forge | (default, Jun 16 2023, 07:06:00) Type 'copyright', 'credits' or 'license' for more informationIPython 8.12.3 -- An enhanced Interactive Python. Type '?' for help. In [1]: from pathlib import Path In [2]: from upath import UPath In [3]: isinstance(UPath('/tmp'), Path)Out[3]: True In [4]: PathOut[4]: pathlib.Path In [5]: Path('/tmp')Out[5]: PosixPath('/tmp') In [6]: UPath('/tmp')Out[6]: PosixUPath('/tmp') I'm kind of inclined to merge, release and then roll back if bug reports start coming in. — Reply to this email directly, view it on GitHub <#1074 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVFLMA7INA653ENLTHQFOLZOFKKJAVCNFSM6AAAAABLKYTBSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJRGI4TKNBYGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>