Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[query-service] teach query service to read MTs and Ts created by Spark #10184

Merged
merged 2 commits into from
Mar 11, 2021
Merged

[query-service] teach query service to read MTs and Ts created by Spark #10184

merged 2 commits into from
Mar 11, 2021

Commits on Mar 11, 2021

  1. [query-service] teach query service to read MTs and Ts created by Spark 

    Hail-on-Spark uses HadoopFS which emulates directories by creating size-zero files with
the name `gs://bucket/dirname/`. Note: the object name literally ends in a slash. Such files
should not be included in `listStatus` (they should always be empty anyway). Unfortunately,
my fix in hail-is#9914 was wrong because `GoogleStorageFileStatus` removes
the trailing slash. This prevented the path from matching `path`, which always ends in a `/`.
    Daniel King committed Mar 11, 2021
    Configuration menu
    Copy the full SHA
    8ef7f0c View commit details
    Browse the repository at this point in the history

  2. fix

    Daniel King committed Mar 11, 2021
    Configuration menu
    Copy the full SHA
    9c5328f View commit details
    Browse the repository at this point in the history