Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for hdf5 #1163

Merged
merged 11 commits into from
Sep 21, 2021
Merged

add support for hdf5 #1163

merged 11 commits into from
Sep 21, 2021

Conversation

njzjz
Copy link
Member

@njzjz njzjz commented Sep 21, 2021

dpdata format will be provided later.

The advantage of hdf5 is that it only has one file for multi-systems containing many systems, so it will be easier to transfer between servers or share online.

File size

About the file size, take an example of methane_v1:

  • Original file: 320 MB
  • Compressed file (.tar.bz2): 182 MB
  • HDF5 file: 258 MB
  • Compressed HDF5 file (.tar.bz2): 183 MB

We can see HDF5 file will have no advantage if fully compressed, but it is smaller if not compressed. Also, compressing and decompressing a single file should be faster than compressing a lot of small files.

Usage

One can add a HDF5 file in the systems list in input JSON file.

@codecov-commenter
Copy link

codecov-commenter commented Sep 21, 2021

Codecov Report

Merging #1163 (000c768) into devel (077df3d) will decrease coverage by 11.38%.
The diff coverage is n/a.

❗ Current head 000c768 differs from pull request most recent head dc44906. Consider uploading reports for the commit dc44906 to get more accurate results
Impacted file tree graph

@@             Coverage Diff             @@
##            devel    #1163       +/-   ##
===========================================
- Coverage   75.66%   64.28%   -11.39%     
===========================================
  Files          89        5       -84     
  Lines        7048       14     -7034     
===========================================
- Hits         5333        9     -5324     
+ Misses       1715        5     -1710     
Impacted Files Coverage Δ
deepmd/entrypoints/config.py
deepmd/descriptor/__init__.py
deepmd/infer/deep_polar.py
deepmd/entrypoints/__init__.py
deepmd/descriptor/se_a.py
deepmd/infer/deep_eval.py
deepmd/entrypoints/doc.py
deepmd/utils/learning_rate.py
deepmd/descriptor/se.py
deepmd/__main__.py
... and 71 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 077df3d...dc44906. Read the comment docs.

@njzjz njzjz requested a review from amcadmus September 21, 2021 20:19
@amcadmus amcadmus merged commit 53f1567 into deepmodeling:devel Sep 21, 2021
@njzjz njzjz deleted the hdf5 branch September 22, 2021 04:39
njzjz added a commit to njzjz/dpdata that referenced this pull request Oct 8, 2021
amcadmus pushed a commit to deepmodeling/dpdata that referenced this pull request Oct 10, 2021
* add deepmd/hdf5 format

To support deepmodeling/deepmd-kit#1163.

* bugfix

* add an example
njzjz pushed a commit to njzjz/deepmd-kit that referenced this pull request Sep 21, 2023
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants