Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save catalog to disk #61

Merged
merged 18 commits into from
Nov 13, 2023
Merged

Save catalog to disk #61

merged 18 commits into from
Nov 13, 2023

Conversation

camposandro
Copy link
Collaborator

Implements saving a LSDB catalog to disk in HiPSCat format. There is a notebook to compare the two possible scenarios we currently have to import a catalog (lsdb.from_dataframe() and the hipscat-import pipeline). Closes #50.

@camposandro camposandro self-assigned this Nov 6, 2023
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

Copy link

github-actions bot commented Nov 6, 2023

Before [aac09ca] After [3e0790c] Ratio Benchmark (Parameter)
55.0±1ms 58.7±2ms 1.07 benchmarks.time_kdtree_crossmatch

Click here to view all benchmarks.

Copy link

codecov bot commented Nov 6, 2023

Codecov Report

Merging #61 (76e6411) into main (d9e03f7) will increase coverage by 0.13%.
Report is 2 commits behind head on main.
The diff coverage is 98.73%.

@@            Coverage Diff             @@
##             main      #61      +/-   ##
==========================================
+ Coverage   99.00%   99.13%   +0.13%     
==========================================
  Files          18       21       +3     
  Lines         401      464      +63     
==========================================
+ Hits          397      460      +63     
  Misses          4        4              
Files Coverage Δ
src/lsdb/catalog/catalog.py 100.00% <100.00%> (ø)
src/lsdb/io/__init__.py 100.00% <100.00%> (ø)
src/lsdb/io/to_hipscat.py 100.00% <100.00%> (ø)
...lsdb/loaders/dataframe/dataframe_catalog_loader.py 100.00% <100.00%> (ø)
src/lsdb/loaders/dataframe/from_dataframe.py 100.00% <ø> (ø)
src/lsdb/types.py 100.00% <100.00%> (ø)
src/lsdb/dask/crossmatch_catalog_data.py 95.08% <50.00%> (+0.08%) ⬆️

📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

@camposandro camposandro force-pushed the issue/50/catalog-saving branch from 4e18445 to 8dab73a Compare November 6, 2023 21:45
@camposandro camposandro force-pushed the issue/50/catalog-saving branch from 7099032 to aa15afe Compare November 8, 2023 18:20
@camposandro camposandro marked this pull request as ready for review November 8, 2023 18:34
src/lsdb/catalog/catalog.py Outdated Show resolved Hide resolved
src/lsdb/catalog/catalog.py Outdated Show resolved Hide resolved
src/lsdb/catalog/catalog.py Show resolved Hide resolved
src/lsdb/catalog/catalog.py Outdated Show resolved Hide resolved
src/lsdb/catalog/catalog.py Outdated Show resolved Hide resolved
docs/notebooks/import_catalogs.ipynb Outdated Show resolved Hide resolved
src/lsdb/io/file_io.py Outdated Show resolved Hide resolved
docs/notebooks/import_catalogs.ipynb Outdated Show resolved Hide resolved
src/lsdb/io/to_hipscat.py Outdated Show resolved Hide resolved
src/lsdb/io/file_io.py Outdated Show resolved Hide resolved
@camposandro camposandro merged commit 03e93be into main Nov 13, 2023
13 of 14 checks passed
@camposandro camposandro deleted the issue/50/catalog-saving branch January 11, 2024 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Catalog saving
2 participants