-
Notifications
You must be signed in to change notification settings - Fork 244
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[fs] support hfs.ls on a bucket #14176
Commits on Jan 22, 2024
-
[fs] support hfs.ls on a bucket
Teaches `hfs.ls('gs://bucket/')` to list the files and directories at the top-level of the bucket. In `main` that command raises because this line of `_ls_no_glob` raises: ```python3 maybe_sb_and_t, maybe_contents = await asyncio.gather( self._size_bytes_and_time_modified_or_none(path), ls_as_dir() ) ``` In particular, `statfile` raises a cloud-specific, esoteric error about a malformed URL or empty object names: ```python3 async def _size_bytes_and_time_modified_or_none(self, path: str) -> Optional[Tuple[int, float]]: try: # Hadoop semantics: creation time is used if the object has no notion of last modification time. file_status = await self.afs.statfile(path) return (await file_status.size(), file_status.time_modified().timestamp()) except FileNotFoundError: return None ``` I decided to add a sub-class of `FileNotFoundError` which is self-describing: `IsABucketError`. I changed most methods to raise that error when given a bucket URL. The two interesting cases: 1. `isdir`. This raises an error but I could also see this returning `True`. A bucket is like a directory whose path/name is empty. 2. `isfile`. This returns False but I could also see this raising an error. This just seems convenient, we know the bucket is not a file so we should say so. --- Apparently `hfs.ls` had no current tests because the globbing system doesn't work with Azure https:// URLs. I fixed it to use `AsyncFSURL.with_new_path_component` which is resilient to Azure https weirdness. However, I had to change `with_new_path_component` to treat an empty path in a special way. I wanted this to hold: ``` actual = str(afs.parse_url('gs://bucket').with_new_path_component('bar')) expected = 'gs://bucket/bar' assert actual == expected ``` But `with_new_path_component` interacts badly with `GoogleAsyncFSURL.__str__` to return this: ``` 'gs://bucket//bar' ```
Dan King committedJan 22, 2024 Configuration menu - View commit details
-
Copy full SHA for 7543069 - Browse repository at this point
Copy the full SHA 7543069View commit details -
add and user with_new_path_components
Dan King committedJan 22, 2024 Configuration menu - View commit details
-
Copy full SHA for 5fa7f85 - Browse repository at this point
Copy the full SHA 5fa7f85View commit details -
Dan King committed
Jan 22, 2024 Configuration menu - View commit details
-
Copy full SHA for 4052520 - Browse repository at this point
Copy the full SHA 4052520View commit details -
Dan King committed
Jan 22, 2024 Configuration menu - View commit details
-
Copy full SHA for d37da3f - Browse repository at this point
Copy the full SHA d37da3fView commit details -
Dan King committed
Jan 22, 2024 Configuration menu - View commit details
-
Copy full SHA for b5cb06f - Browse repository at this point
Copy the full SHA b5cb06fView commit details -
Dan King committed
Jan 22, 2024 Configuration menu - View commit details
-
Copy full SHA for 3bf27b9 - Browse repository at this point
Copy the full SHA 3bf27b9View commit details -
use with_root_path in router_fs
Dan King committedJan 22, 2024 Configuration menu - View commit details
-
Copy full SHA for 2ace87b - Browse repository at this point
Copy the full SHA 2ace87bView commit details
Commits on Feb 2, 2024
-
use exit stacks and context managers for RouterFS
Dan King committedFeb 2, 2024 Configuration menu - View commit details
-
Copy full SHA for 75bf0b9 - Browse repository at this point
Copy the full SHA 75bf0b9View commit details -
Dan King committed
Feb 2, 2024 Configuration menu - View commit details
-
Copy full SHA for b24b709 - Browse repository at this point
Copy the full SHA b24b709View commit details -
Dan King committed
Feb 2, 2024 Configuration menu - View commit details
-
Copy full SHA for c94fa65 - Browse repository at this point
Copy the full SHA c94fa65View commit details -
fix override: error_if_bucket is keyword only
Dan King committedFeb 2, 2024 Configuration menu - View commit details
-
Copy full SHA for 61c0a2c - Browse repository at this point
Copy the full SHA 61c0a2cView commit details -
Dan King committed
Feb 2, 2024 Configuration menu - View commit details
-
Copy full SHA for 9ab8302 - Browse repository at this point
Copy the full SHA 9ab8302View commit details
Commits on Feb 5, 2024
-
also error if we try to create a bucket
Dan King committedFeb 5, 2024 Configuration menu - View commit details
-
Copy full SHA for 19e49b4 - Browse repository at this point
Copy the full SHA 19e49b4View commit details -
need to ensure something exists in that bucket
Dan King committedFeb 5, 2024 Configuration menu - View commit details
-
Copy full SHA for c459a33 - Browse repository at this point
Copy the full SHA c459a33View commit details