Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fuzzy dedup #699

Merged
merged 100 commits into from
Nov 26, 2024
Merged

Fuzzy dedup #699

merged 100 commits into from
Nov 26, 2024

Conversation

Kibnelson
Copy link
Collaborator

Why are these changes needed?

Provide fuzzy dedup implementation for Python, Spark and Ray

Related issue number (if any).

#152
#79

blublinsky and others added 30 commits October 10, 2024 19:05
@blublinsky
added folder_transform
47f4526
@blublinsky
added folder_transform
5fd20a1
@blublinsky
added folder_transform
38b4725
@blublinsky
added folder_transform
a3abf21
@cmadam
Merge branch 'folder_transform' into fuzzy-dedup
d93a06c
@cmadam
Fuzzy dedup pure python implementation
af8475d 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Fuzzy dedup spark implementation
7f9b503 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@blublinsky
added folder_transform
3349521
@blublinsky
added folder_transform
0553edf
@blublinsky
added folder_transform
a53412e
@blublinsky
added folder_transform
9c3ace7
@blublinsky
added noop testing
7091a2e
@Kibnelson
Fuzzy dedup ray implementation
680c78a 
Signed-off-by: nelson <kibnelson@gmail.com>
@Kibnelson
Fixed bug in ray to distribute docs to remove file to all workers
0c31dc0 
Signed-off-by: nelson <kibnelson@gmail.com>
@cmadam
Merge with updated folder_transform branch
47d8fdf 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@blublinsky
added folder_transform
6ee6695
@blublinsky
added folder_transform
e7260ba
@blublinsky
added folder_transform
5856f3f
@blublinsky
added folder_transform
6519686
@blublinsky
added noop testing
c728224
@blublinsky
added noop Ray testing
6e2863a
@blublinsky
added noop Spark testing
3c9be57
@blublinsky
more data access simplifications
371a712
@cmadam
Renamed/refactored fuzzy dedup python orchestrator
680f313 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Rewrote cluster_analysis_transform as a folder_transform
c29d3bf 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Wrote get_duplicate_list_transform as a folder_transform
aada59e 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Added text preprocessing
2019d56 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Added python test data
9362803 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Added project admin tools
ddbd602 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Bug fix
4dac838 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Cleanup main entry point and local implementation of ray transforms
c4f889b 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Cleanup main entry point and local implementation of spark transforms
f3c5be0 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Cleanup main entry point and local implementation of spark transforms
4941d5b 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@touma-I touma-I self-requested a review November 15, 2024 19:01
@cmadam
Added documentation for python, ray, spark and kfp
9c82fe0 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@shahrokhDaijavad
Update README.md
ed4e9c1 
utils folder is one level up from the python folder
@shahrokhDaijavad
Copy link
Member

@cmadam The documentation looks really good! Thank you. I found one wrong link to the utils folder, and I changed it manually (without a change request). Sorry that it meant a full ci/cd test! If I find other problems, I will make a change request and won't change myself.

@touma-I touma-I self-requested a review November 19, 2024 01:59
shahrokhDaijavad
shahrokhDaijavad approved these changes Nov 20, 2024
View reviewed changes
Copy link
Member

@shahrokhDaijavad shahrokhDaijavad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve the three README files.

cmadam and others added 5 commits November 21, 2024 00:44
@cmadam
Reduce the amount of logging
0636d5f 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Cleanup KFP pipeline code
d58518b 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@touma-I
Merge branch 'dev' into fuzzy-dedup
ea2a269
@touma-I
added fdedup to build package for all transforms
280d105 
Signed-off-by: Maroun Touma <touma@us.ibm.com>
@cmadam
First draft of fdedup notebook
1a762e0 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@shahrokhDaijavad
Copy link
Member

@cmadam Thank you for the notebook. Looks good. Two comments:

  1. Your notebook is using the Ray runtime. To be consistent with other notebooks, can you please change it to use the Python runtime? (now that we have it because of your work on this PR)
  2. In the cell before setting up the parameters, can you briefly (in a few sentences) describe what some of the key parameters are?

@cmadam
Added sample ray fuzzy dedup jupyter notebook
ffebdc1 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Copy link
Collaborator

cmadam commented Nov 25, 2024

@cmadam Thank you for the notebook. Looks good. Two comments:

  1. Your notebook is using the Ray runtime. To be consistent with other notebooks, can you please change it to use the Python runtime? (now that we have it because of your work on this PR)
  2. In the cell before setting up the parameters, can you briefly (in a few sentences) describe what some of the key parameters are?

@shahrokhDaijavad : I am going to add documentation of the parameters that are used in the example notebook. Regarding the runtime, let me double-check with @touma-I, as my understanding was that he asked me to provide a notebook based on the ray runtime, not python.

@shahrokhDaijavad
Copy link
Member

Thanks, @cmadam. Sure. Maybe @touma-I wants this one with ray runtime as an example of how we can use ray.

@cmadam
Add jupyter notebooks for python, ray and spark fuzzy dedup
75fc4d1 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@cmadam
Add jupyter notebooks for python, ray and spark fuzzy dedup
edd5841 
Signed-off-by: Constantin M Adam <cmadam@us.ibm.com>
@shahrokhDaijavad
Copy link
Member

Wow! @cmadam This is outstanding! Three notebooks for the three runtimes.

@touma-I
create test branch for updating fdedup with latest release for 0.2.3.…
a62f92b 
…dev0

Signed-off-by: Maroun Touma <touma@us.ibm.com>
@touma-I
remove reference to noop transform project
7ae1f13 
Signed-off-by: Maroun Touma <touma@us.ibm.com>
@touma-I touma-I merged commit 2f80d9c into dev Nov 26, 2024
179 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cmadam cmadam cmadam left review comments

@revit13 revit13 revit13 left review comments

@touma-I touma-I touma-I approved these changes

@shahrokhDaijavad shahrokhDaijavad shahrokhDaijavad approved these changes

@daw3rd daw3rd daw3rd approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@Kibnelson @shahrokhDaijavad @cmadam @daw3rd @revit13 @touma-I @blublinsky