Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read catalog - get only positional coordinates by default #267

Closed
3 tasks done
nevencaplar opened this issue Apr 4, 2024 · 4 comments · Fixed by #543
Closed
3 tasks done

Read catalog - get only positional coordinates by default #267

nevencaplar opened this issue Apr 4, 2024 · 4 comments · Fixed by #543
Labels
enhancement New feature or request

Comments

@nevencaplar
Copy link
Member

Feature request

When reading a catalog with read_hipscat change the default to only read in ra, dec coordinates, i.e., spatial coordinates, and require user to specify further columns. This is to avoid extensive loading when exploring the datasets (e.g., loading 10, 20, 50 columns of a catalog when only looking at the number of objects or similar).

Before submitting
Please check the following:

  • I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
  • I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
  • If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.
@nevencaplar nevencaplar added the enhancement New feature or request label Apr 4, 2024
@hombit
Copy link
Contributor

hombit commented Apr 5, 2024

Theoretically dask-expr would allow us to load only those columns we actually use, but it would require to change some LSDB code, for at least 1) data loading, and 2) cross-matching.

@nevencaplar
Copy link
Member Author

@dougbrn Can you please link here the presentation that you did which showed that dask-expr is pretty limited in that regard. or was it your @hombit presentation?

@delucchi-cmu
Copy link
Contributor

STSci requested a default_columns kind of attribute, to accomplish a similar goal, but potentially with a few more columns (e.g. redshift and r_mag).

See astronomy-commons/hats#322

@nevencaplar
Copy link
Member Author

Connected with: #529

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants