Enhance Numeric Data Inspection and Introduce Positive/Negative Filtering #217
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
for more information, see https://pre-commit.ci
include zero
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
|
@jalr4ever Please help me review this PR. |
jalr4ever
reviewed
Aug 26, 2024
|
Modified the code according to the suggestions in the code review, and all unit tests have passed. |
jalr4ever
approved these changes
Aug 27, 2024
Wh1isper
reviewed
Aug 27, 2024
| @@ -14,69 +14,132 @@ class NumericInspector(Inspector): | |||
|
|
|||
| This class is a subclass of `Inspector` and is designed to provide methods for inspecting | |||
| and analyzing numeric data. It includes methods for detecting int or float data type. | |||
|
|
|||
| In August 2024, we introduced a new feature that will continue to judge the positivity or | |||
There was a problem hiding this comment.
Perhaps we should indicate the PR and release version here, rather than the date?
There was a problem hiding this comment.
Good idea, I also have another branch in development, I'll release after merging another PR. Due to various reasons, we haven't released a new version for a long time :(
There was a problem hiding this comment.
Nerver mind, thanks for your work!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enhance NumericInspector and Implement PositiveNegativeFilter
Description
This PR introduces significant enhancements to the Synthetic Data Generator (SDG) framework, specifically in the
NumericInspectorclass and the addition of a newPositiveNegativeFilterclass. TheNumericInspectorhas been updated to support the identification of both positive and negative numeric columns, improving the quality of synthetic data generation. ThePositiveNegativeFilterclass is designed to filter data based on the positivity or negativity of values in specified columns, ensuring that the integrity of the data is maintained during processing.Key changes include:
NumericInspectorto classify columns as positive or negative based on defined thresholds.PositiveNegativeFilterto enforce positivity or negativity constraints on specified columns during data processing.Motivation and Context
The motivation behind these changes is to enhance the data quality assurance mechanisms within the SDG framework. By allowing the identification of positive and negative columns, we can ensure that the synthetic data generated meets specific criteria, which is crucial for various applications such as model training and data sharing. This change addresses the need for more robust data validation and filtering capabilities, ultimately leading to better performance and reliability of the generated synthetic data.
How has this been tested?
The changes have been thoroughly tested using a dedicated test suite. The following tests were performed:
NumericInspectorto ensure correct identification of positive and negative columns.PositiveNegativeFilterto verify that it correctly filters data based on the positivity and negativity of values in specified columns.Types of changes
Checklist: