Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a URL clustering method to reduce the cardinality #268

Merged
merged 6 commits into from
Jun 27, 2022

Conversation

dxsup
Copy link
Member

@dxsup dxsup commented Jun 24, 2022

Description

  1. Add a new urlclustering package which contains a ClusteringMethod interface and two trivial implementations.
  2. Add an option for choosing which URL clustering method is used. alphabet is used by default.

Related Issue

#267

How Has This Been Tested?

UnitTest

=== RUN   TestAlphabeticClusteringMethod_Clustering
--- PASS: TestAlphabeticClusteringMethod_Clustering (0.00s)
=== RUN   TestAlphabeticClusteringMethod_ClusteringBaseline
--- PASS: TestAlphabeticClusteringMethod_ClusteringBaseline (0.00s)
=== RUN   Test_isAlphabetical
=== RUN   Test_isAlphabetical/alphabetical
=== RUN   Test_isAlphabetical/non-alphabetical
--- PASS: Test_isAlphabetical (0.00s)
    --- PASS: Test_isAlphabetical/alphabetical (0.00s)
    --- PASS: Test_isAlphabetical/non-alphabetical (0.00s)
=== RUN   TestNoParamClusteringMethod_Clustering
--- PASS: TestNoParamClusteringMethod_Clustering (0.00s)
PASS

Benchmark

goos: darwin
goarch: amd64
pkg: github.com/Kindling-project/kindling/collector/pkg/urlclustering
cpu: Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz
Benchmark_ClusteringBaseline
Benchmark_ClusteringBaseline-4   	  171873	      6979 ns/op
Benchmark_Clustering
Benchmark_Clustering-4           	  648603	      1991 ns/op
PASS

Daxin Wang added 3 commits June 23, 2022 20:59
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
@dxsup dxsup requested a review from NeJan2020 June 24, 2022 02:52
@dxsup dxsup changed the title Add a URL clustering method to reduce its cardinality Add a URL clustering method to reduce the cardinality Jun 24, 2022
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
@dxsup dxsup requested a review from hocktea214 June 24, 2022 07:57
Daxin Wang added 2 commits June 24, 2022 16:20
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
Copy link
Collaborator

@hocktea214 hocktea214 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an Alphabet version, it is able to merge some urls into the regular url.

@dxsup dxsup merged commit 7210fe0 into KindlingProject:main Jun 27, 2022
@dxsup dxsup deleted the feature/http-url-cluster branch June 27, 2022 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants