Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: refactor update_sel and save min_nbor_dist #3829

Merged
merged 14 commits into from
May 31, 2024

Conversation

njzjz
Copy link
Member

@njzjz njzjz commented May 28, 2024

Fix #3525. Fix #3544.

Summary by CodeRabbit

  • New Features
    • Enhanced update_sel method to accept additional parameters and return more detailed data, improving model selection and neighbor statistics.
  • Bug Fixes
    • Improved handling and processing of training data to enhance model accuracy.
  • Refactor
    • Updated method signatures and logic for consistency and better performance.
  • Chores
    • Removed unused hook method to streamline codebase.

njzjz added 6 commits May 28, 2024 06:14
Fix deepmodeling#3525. Fix deepmodeling#3544.

Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Copy link
Contributor

coderabbitai bot commented May 28, 2024

Walkthrough

Walkthrough

The changes across multiple files enhance the update_sel method by adding parameters for train_data and type_map, and updating its return type to include a tuple with updated local data and an optional float value. This modification aims to improve neighbor statistics calculations. Additionally, the UpdateSel.hook method was removed, affecting the control flow related to saving data to the model.

Changes

Files Change Summary
deepmd/dpmodel/descriptor/dpa1.py, deepmd/dpmodel/descriptor/dpa2.py, deepmd/dpmodel/descriptor/hybrid.py, deepmd/dpmodel/descriptor/make_base_descriptor.py, deepmd/dpmodel/descriptor/se_e2_a.py, deepmd/dpmodel/descriptor/se_r.py Updated update_sel method to accept train_data, type_map, and return a tuple with updated local data and optional float value.
deepmd/dpmodel/model/base_model.py, deepmd/dpmodel/model/dp_model.py, deepmd/dpmodel/model/spin_model.py Added min_nbor_dist attribute and related methods for model compression. Modified update_sel method signature.
deepmd/dpmodel/utils/update_sel.py, deepmd/pt/utils/update_sel.py Removed hook method from UpdateSel class.
deepmd/pt/entrypoints/main.py Modified train function to handle min_nbor_dist and updated model selection and training data processing logic.
deepmd/pt/model/descriptor/dpa1.py, deepmd/pt/model/descriptor/dpa2.py, deepmd/pt/model/descriptor/hybrid.py, deepmd/pt/model/descriptor/se_a.py, deepmd/pt/model/descriptor/se_r.py, deepmd/pt/model/descriptor/se_t.py Updated update_sel method to accept train_data, type_map, and return a tuple with updated local data and optional float value.
deepmd/pt/model/model/dp_model.py, deepmd/pt/model/model/dp_zbl_model.py, deepmd/pt/model/model/frozen.py, deepmd/pt/model/model/model.py, deepmd/pt/model/model/spin_model.py Modified update_sel method signature and added min_nbor_dist related methods.
deepmd/pt/utils/serialization.py Added serialization and deserialization support for min_nbor_dist variable.
deepmd/tf/descriptor/descriptor.py, deepmd/tf/descriptor/hybrid.py, deepmd/tf/descriptor/loc_frame.py, deepmd/tf/descriptor/se.py, deepmd/tf/descriptor/se_a_mask.py, deepmd/tf/descriptor/se_atten.py Updated update_sel method to accept train_data, type_map, and return a tuple with updated local data and optional float value.
deepmd/tf/entrypoints/compress.py Modified compress function to retrieve train_data and pass it to update_sel.get_min_nbor_dist.

Assessment against linked issues

Objective Addressed Explanation
Save to the model in UpdateSel.hook (#3525) The hook method was removed instead of implementing the save functionality.

Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 27ea218 and ecb7b66.

Files selected for processing (4)
  • deepmd/dpmodel/descriptor/se_t.py (2 hunks)
  • deepmd/pt/entrypoints/main.py (4 hunks)
  • deepmd/pt/model/descriptor/se_a.py (2 hunks)
  • deepmd/pt/model/descriptor/se_t.py (2 hunks)
Files skipped from review due to trivial changes (1)
  • deepmd/pt/model/descriptor/se_a.py
Files skipped from review as they are similar to previous changes (1)
  • deepmd/pt/entrypoints/main.py
Additional comments not posted (2)
deepmd/dpmodel/descriptor/se_t.py (1)

354-382: Refactor of update_sel method aligns with PR objectives.

The method now correctly handles the train_data and type_map parameters to perform neighbor statistics and update selections. The return type has been adjusted to include both the updated local data and the minimum neighbor distance, which is a useful addition for further processing.

deepmd/pt/model/descriptor/se_t.py (1)

330-358: Refactor of update_sel method aligns with PR objectives.

The method now correctly handles the train_data and type_map parameters to perform neighbor statistics and update selections. The return type has been adjusted to include both the updated local data and the minimum neighbor distance, which is a useful addition for further processing.


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

source/tests/pt/test_update_sel.py Fixed Show fixed Hide fixed
deepmd/pt/entrypoints/main.py Fixed Show fixed Hide fixed
Copy link

codecov bot commented May 28, 2024

Codecov Report

Attention: Patch coverage is 81.68317% with 37 lines in your changes missing coverage. Please review.

Project coverage is 82.62%. Comparing base (8de0aba) to head (ecb7b66).
Report is 127 commits behind head on devel.

Files with missing lines Patch % Lines
deepmd/dpmodel/descriptor/dpa2.py 28.57% 5 Missing ⚠️
deepmd/pt/entrypoints/main.py 58.33% 5 Missing ⚠️
deepmd/pt/model/descriptor/dpa2.py 28.57% 5 Missing ⚠️
deepmd/tf/model/pairtab.py 40.00% 3 Missing ⚠️
deepmd/dpmodel/descriptor/se_r.py 50.00% 2 Missing ⚠️
deepmd/dpmodel/descriptor/se_t.py 50.00% 2 Missing ⚠️
deepmd/pt/model/descriptor/se_r.py 50.00% 2 Missing ⚠️
deepmd/pt/model/descriptor/se_t.py 50.00% 2 Missing ⚠️
deepmd/pt/model/model/dp_zbl_model.py 50.00% 2 Missing ⚠️
deepmd/pt/utils/serialization.py 50.00% 2 Missing ⚠️
... and 6 more
Additional details and impacted files
@@            Coverage Diff             @@
##            devel    #3829      +/-   ##
==========================================
+ Coverage   82.53%   82.62%   +0.08%     
==========================================
  Files         515      515              
  Lines       49492    49563      +71     
  Branches     2985     2987       +2     
==========================================
+ Hits        40849    40949     +100     
+ Misses       7732     7703      -29     
  Partials      911      911              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

njzjz added 4 commits May 28, 2024 07:53
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Copy link
Collaborator

@wanghan-iapcm wanghan-iapcm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The UT for the dp model is necessary. or shall we provide consistency test for the behavior of update_sel across backends?

@njzjz
Copy link
Member Author

njzjz commented May 28, 2024

The UT for the dp model is necessary. or shall we provide consistency test for the behavior of update_sel across backends?

Good idea. The e2e and unit tests are not in conflict.

@njzjz
Copy link
Member Author

njzjz commented May 28, 2024

We haven't had fixtures for descriptors, though.

Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
@njzjz njzjz requested a review from wanghan-iapcm May 28, 2024 20:58
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 14

Outside diff range and nitpick comments (7)
deepmd/utils/update_sel.py (2)

86-115: Review the handling of empty type_map in get_nbor_stat. Consider adding a comment explaining why an empty type_map is set to None.


[!TIP]
Codebase Verification

The current unit tests for update_one_sel cover the auto and auto:1.5 cases but do not test the integer sel case or the warning mechanism for insufficient sel values. Please add test cases to cover these scenarios:

  • Integer sel input.

  • Warning mechanism when sel values are insufficient.

  • source/tests/tf/test_train.py

  • source/tests/pt/test_update_sel.py

  • source/tests/common/dpmodel/test_update_sel.py

Analysis chain

Line range hint 25-59: Ensure thorough testing of the update_one_sel method, especially the new logic handling different sel types and the warning mechanism.

Scripts Executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Description: Verify the unit tests cover all branches in `update_one_sel`.

# Test: Search for unit tests covering `update_one_sel`.
rg --type py "test_update_one_sel"

Length of output: 296



Script:

#!/bin/bash
# Extract the content of the test_update_one_sel functions from the identified test files.

# Extract from source/tests/tf/test_train.py
rg --type py -A 20 "def test_update_one_sel" source/tests/tf/test_train.py

# Extract from source/tests/pt/test_update_sel.py
rg --type py -A 20 "def test_update_one_sel" source/tests/pt/test_update_sel.py

# Extract from source/tests/common/dpmodel/test_update_sel.py
rg --type py -A 20 "def test_update_one_sel" source/tests/common/dpmodel/test_update_sel.py

Length of output: 3198

source/tests/common/dpmodel/test_update_sel.py (1)

42-71: The test test_update_sel_hybrid provides a comprehensive check of the update_sel functionality for hybrid descriptors. Consider adding more comments to explain the setup and expected outcomes for clarity.

doc/development/create-a-model-pt.md (1)

Line range hint 1-5: Consider adding a comma after "model" and correcting "trainner" to "trainer" for grammatical accuracy.

- If you'd like to create a new model that isn't covered by the existing DeePMD-kit library, but reuse DeePMD-kit's other efficient modules such as data processing, trainner, etc, you may want to read this section.
+ If you'd like to create a new model that isn't covered by the existing DeePMD-kit library, but reuse DeePMD-kit's other efficient modules such as data processing, trainer, etc., you may want to read this section.
source/tests/pt/test_update_sel.py (1)

16-20: Consider adding a docstring to the update_sel function to explain its purpose and parameters.

deepmd/tf/model/pairtab.py (1)

Line range hint 275-308: Consider adding error handling for cases where min_nbor_dist is None to prevent potential runtime errors in PairTabModel.

-    if min_nbor_dist is not None:
+    if min_nbor_dist is not None:
+        return local_jdata_cpy, min_nbor_dist
+    else:
+        # Handle the case where min_nbor_dist is None
+        log.error("min_nbor_dist is None, which may indicate an issue with neighbor statistics calculation.")
+        raise ValueError("Failed to calculate min_nbor_dist.")
source/tests/tf/test_train.py (1)

41-51: Refactor duplicated code in test_update_one_sel.

Consider extracting the repeated logic into a helper function to avoid code duplication and improve maintainability.

deepmd/tf/descriptor/se_atten.py Show resolved Hide resolved
deepmd/tf/model/model.py Show resolved Hide resolved
deepmd/pt/model/descriptor/hybrid.py Show resolved Hide resolved
deepmd/dpmodel/descriptor/se_r.py Show resolved Hide resolved
deepmd/dpmodel/descriptor/se_e2_a.py Show resolved Hide resolved
deepmd/tf/entrypoints/train.py Show resolved Hide resolved
deepmd/tf/model/frozen.py Show resolved Hide resolved
deepmd/tf/model/linear.py Show resolved Hide resolved
source/tests/tf/test_train.py Show resolved Hide resolved
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
@njzjz
Copy link
Member Author

njzjz commented May 30, 2024

@coderabbitai resolve

@wanghan-iapcm wanghan-iapcm added this pull request to the merge queue May 31, 2024
Merged via the queue into deepmodeling:devel with commit 3a7fbcf May 31, 2024
60 checks passed
mtaillefumier pushed a commit to mtaillefumier/deepmd-kit that referenced this pull request Sep 18, 2024
Fix deepmodeling#3525. Fix
deepmodeling#3544.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Enhanced `update_sel` method to accept additional parameters and
return more detailed data, improving model selection and neighbor
statistics.
- **Bug Fixes**
- Improved handling and processing of training data to enhance model
accuracy.
- **Refactor**
- Updated method signatures and logic for consistency and better
performance.
- **Chores**
  - Removed unused `hook` method to streamline codebase.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Co-authored-by: Han Wang <92130845+wanghan-iapcm@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

save to the model in UpdateSel.hook save to the model in UpdateSel.hook
3 participants