[WIP] Implement ngt Statistics API #2533

kpango · 2024-07-24T06:28:50Z

Description

Summary

This pull request introduces several significant updates and enhancements, primarily focusing on adding new API functionalities for index statistics and improving observability metrics.

The _test.go and pb.go diffs are just re-generated please ignore them.

Changes

Protobuf Updates:
- Added Info.Index.Statistics to apis/proto/v1/payload/payload.proto.
- Introduced IndexStatistics and IndexStatisticsDetail APIs in apis/proto/v1/vald/index.proto to return Info.Index.Statistics.
Handler Implementations:
- Implemented IndexStatistics and IndexStatisticsDetail API handlers in pkg/gateway/lb/handler/grpc/handler.go.
- Added the same API handlers in pkg/agent/core/ngt/service/ngt.go.
Algorithm Enhancements:
- Added GraphStatistics struct and GetGraphStatistics(m statisticsType) (stats *GraphStatistics, err error) function in internal/core/algorithm/ngt/ngt.go to enable retrieval of index statistics.
- Incorporated logic to obtain GraphStatistics post CreateIndex based on options in pkg/agent/core/ngt/service/ngt.go.
Metrics Export:
- Added export processing for GraphStatistics metrics in internal/observability/metrics/agent/core/ngt/ngt.go.
- Introduced type aliases for Int64ObservableGauge and Float64ObservableGauge in internal/observability/metrics/metrics.go.
Configuration and Formatting:
- Updated .github/workflows/format.yml to add ignore settings for Format and license.
- Enhanced Makefile under format/go to include formatting of artifacts generated from Proto.

Additional Notes

These changes enhance the functionality of the Vald project by providing detailed index statistics, which are crucial for monitoring and performance tuning.
The observability improvements will facilitate better metric tracking and reporting, contributing to overall system stability and performance.

Related Issue

Versions

Go Version: 1.22.5
Rust Version: 1.77.2
Docker Version: 20.10.8
Kubernetes Version: v1.30.3
NGT Version: 2.2.3

Checklist

I have read the CONTRIBUTING document and completed our CLA agreement.
I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer

coderabbitai · 2024-07-24T06:28:55Z

Important

Review skipped

Ignore keyword(s) in the title.

Ignored keywords (3)

WIP
DO NOT MERGE
DNM

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

vdaas-ci · 2024-07-24T06:29:17Z

[CHATOPS:HELP] ChatOps commands.

🙆‍♀️ /approve - approve
🍱 /format - format codes and add licenses
✅ /gen-test - generate test codes
🏷️ /label - add labels
🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

cloudflare-workers-and-pages · 2024-07-24T06:31:43Z

Deploying vald with Cloudflare Pages

Latest commit:	`714027d`
Status:	✅ Deploy successful!
Preview URL:	https://898c68e1.vald.pages.dev
Branch Preview URL:	https://feature-agent-ngt-statistics.vald.pages.dev

View logs

codecov · 2024-07-24T06:43:40Z

Codecov Report

Attention: Patch coverage is 0.94340% with 315 lines in your changes missing coverage. Please review.

Project coverage is 7.90%. Comparing base (76eb8d7) to head (e445bb1).
Report is 1 commits behind head on main.

Files	Patch %	Lines
apis/grpc/v1/payload/payload.pb.go	0.48%	205 Missing ⚠️
internal/core/algorithm/ngt/ngt.go	0.00%	62 Missing ⚠️
apis/grpc/v1/vald/index_vtproto.pb.go	0.00%	40 Missing ⚠️
apis/grpc/v1/payload/payload.pb.json.go	0.00%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #2533       +/-   ##
==========================================
- Coverage   18.43%   7.90%   -10.54%     
==========================================
  Files         566     162      -404     
  Lines       66342   35872    -30470     
==========================================
- Hits        12232    2835     -9397     
+ Misses      53320   32736    -20584     
+ Partials      790     301      -489

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This commit fixes the style issues introduced in 597f8e2 according to the output from Gofumpt and Prettier. Details: #2533

vdaas-ci · 2024-07-25T01:08:33Z

[WARNING:INTCFG] Changes in interal/config may require you to change Helm charts. Please check.

This commit fixes the style issues introduced in c460b18 according to the output from Gofumpt and Prettier. Details: #2533

github-actions · 2024-07-25T03:03:04Z

pkg/gateway/lb/handler/grpc/handler.go

+		return merged
+	}
+
+	var indegrees, outdegrees []int32


🚫 [golangci] _{reported by reviewdog 🐶}
Consider pre-allocating indegrees (prealloc)

github-actions · 2024-07-25T03:03:04Z

pkg/gateway/lb/handler/grpc/handler.go

+	}
+
+	var indegrees, outdegrees []int32
+	var indegreeCounts [][]int64


🚫 [golangci] _{reported by reviewdog 🐶}
Consider pre-allocating indegreeCounts (prealloc)

github-actions · 2024-07-25T03:03:04Z

pkg/gateway/lb/handler/grpc/handler.go

+
+	var indegrees, outdegrees []int32
+	var indegreeCounts [][]int64
+	var outdegreeHistograms, indegreeHistograms [][]uint64


🚫 [golangci] _{reported by reviewdog 🐶}
Consider pre-allocating outdegreeHistograms (prealloc)

github-actions · 2024-07-25T03:03:05Z

pkg/gateway/lb/handler/grpc/handler.go

+
+func mergeInfoIndexStatistics(
+	stats map[string]*payload.Info_Index_Statistics,
+) *payload.Info_Index_Statistics {


🚫 [golangci] _{reported by reviewdog 🐶}
mergeInfoIndexStatistics - result 0 (*github.com/vdaas/vald/apis/grpc/v1/payload.Info_Index_Statistics) is always nil (unparam)

github-actions · 2024-07-25T03:03:05Z

cmd/agent/core/faiss/main_test.go

+// 	type want struct{}
+// 	type test struct {
+// 		name       string
+// 		want       want


🚫 [golangci] _{reported by reviewdog 🐶}
Duplicate words (want) found (dupword)

github-actions · 2024-07-25T03:03:05Z

pkg/tools/benchmark/job/service/insert_test.go

+// 	type fields struct {
+// 		eg                 errgroup.Group
+// 		dataset            *config.BenchmarkDataset
+// 		jobType            jobType


🚫 [golangci] _{reported by reviewdog 🐶}
Duplicate words (jobType) found (dupword)

github-actions · 2024-07-25T03:03:05Z

pkg/tools/benchmark/job/service/job_test.go

+// 	type fields struct {
+// 		eg                 errgroup.Group
+// 		dataset            *config.BenchmarkDataset
+// 		jobType            jobType


🚫 [golangci] _{reported by reviewdog 🐶}
Duplicate words (jobType) found (dupword)

github-actions · 2024-07-25T03:03:05Z

pkg/tools/benchmark/job/service/job_test.go

+// 	type fields struct {
+// 		eg                 errgroup.Group
+// 		dataset            *config.BenchmarkDataset
+// 		jobType            jobType


🚫 [golangci] _{reported by reviewdog 🐶}
Duplicate words (jobType) found (dupword)

github-actions · 2024-07-25T03:03:05Z

internal/observability/metrics/agent/core/ngt/ngt.go

@@ -61,8 +158,8 @@ func New(n service.NGT) metrics.Metric {
 	}
 }

-func (n *ngtMetrics) View() ([]metrics.View, error) {
-	return []metrics.View{
+func (n *ngtMetrics) View() (mv []metrics.View, err error) {


🚫 [golangci] _{reported by reviewdog 🐶}
Function name: View, Cyclomatic Complexity: 2, Halstead Volume: 5766.49, Maintainability Index: 17 (maintidx)

github-actions · 2024-07-25T03:03:05Z

internal/observability/metrics/agent/core/ngt/ngt.go

 }

-func (n *ngtMetrics) Register(m metrics.Meter) error {
-	indexCount, err := m.Int64ObservableGauge(
+func (n *ngtMetrics) Register(m metrics.Meter) (err error) {


🚫 [golangci] _{reported by reviewdog 🐶}
Function name: Register, Cyclomatic Complexity: 51, Halstead Volume: 12248.73, Maintainability Index: 4 (maintidx)

Signed-off-by: kpango <kpango@vdaas.org>

kpango force-pushed the feature/agent/ngt-statistics-api branch from 917534d to ebd278c Compare July 24, 2024 06:31

github-actions bot added the priority/medium label Jul 24, 2024

kpango force-pushed the feature/agent/ngt-statistics-api branch 4 times, most recently from 164ac1a to 826cd8e Compare July 24, 2024 08:07

github-actions bot added area/discoverer area/agent/core/faiss area/agent/core/ngt labels Jul 24, 2024

kpango force-pushed the feature/agent/ngt-statistics-api branch 2 times, most recently from cfbaf2f to 2b3754d Compare July 24, 2024 08:41

github-actions bot added the area/makefile label Jul 24, 2024

kpango force-pushed the feature/agent/ngt-statistics-api branch from 99cfe7e to 597f8e2 Compare July 25, 2024 01:08

deepsource-autofix bot added a commit that referenced this pull request Jul 25, 2024

style: format code with Gofumpt and Prettier

2eb2623

This commit fixes the style issues introduced in 597f8e2 according to the output from Gofumpt and Prettier. Details: #2533

github-actions bot added the area/helm label Jul 25, 2024

kpango force-pushed the feature/agent/ngt-statistics-api branch 3 times, most recently from 6047dc0 to 3df1ba1 Compare July 25, 2024 01:49

github-actions bot added size/XXXL and removed size/XXL labels Jul 25, 2024

kpango force-pushed the feature/agent/ngt-statistics-api branch 4 times, most recently from 496be06 to c460b18 Compare July 25, 2024 02:22

deepsource-autofix bot added a commit that referenced this pull request Jul 25, 2024

style: format code with Gofumpt and Prettier

ff25721

This commit fixes the style issues introduced in c460b18 according to the output from Gofumpt and Prettier. Details: #2533

kpango force-pushed the feature/agent/ngt-statistics-api branch 2 times, most recently from 50f5c99 to bb96c49 Compare July 25, 2024 02:53

github-actions bot reviewed Jul 25, 2024

View reviewed changes

kpango force-pushed the feature/agent/ngt-statistics-api branch 7 times, most recently from 739fc7a to 6717995 Compare July 25, 2024 08:13

Implement ngt Statistics API

714027d

Signed-off-by: kpango <kpango@vdaas.org>

kpango force-pushed the feature/agent/ngt-statistics-api branch from 6717995 to 714027d Compare July 25, 2024 08:14

kpango closed this Jul 25, 2024

kpango deleted the feature/agent/ngt-statistics-api branch July 25, 2024 08:25

kpango restored the feature/agent/ngt-statistics-api branch July 25, 2024 08:25

github-actions bot mentioned this pull request Aug 1, 2024

Monthly review time metrics report: 2024-07-01..2024-07-31 #2555

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Implement ngt Statistics API #2533

[WIP] Implement ngt Statistics API #2533

kpango commented Jul 24, 2024 •

edited

Loading

coderabbitai bot commented Jul 24, 2024

Review skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

vdaas-ci commented Jul 24, 2024

cloudflare-workers-and-pages bot commented Jul 24, 2024 •

edited

Loading

codecov bot commented Jul 24, 2024 •

edited

Loading

vdaas-ci commented Jul 25, 2024

github-actions bot Jul 25, 2024

github-actions bot Jul 25, 2024

github-actions bot Jul 25, 2024

github-actions bot Jul 25, 2024

github-actions bot Jul 25, 2024

github-actions bot Jul 25, 2024

github-actions bot Jul 25, 2024

github-actions bot Jul 25, 2024

github-actions bot Jul 25, 2024

github-actions bot Jul 25, 2024

[WIP] Implement ngt Statistics API #2533

[WIP] Implement ngt Statistics API #2533

Conversation

kpango commented Jul 24, 2024 • edited Loading

Description

Summary

Changes

Additional Notes

Related Issue

Versions

Checklist

Special notes for your reviewer

coderabbitai bot commented Jul 24, 2024

Review skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

vdaas-ci commented Jul 24, 2024

cloudflare-workers-and-pages bot commented Jul 24, 2024 • edited Loading

Deploying vald with Cloudflare Pages

codecov bot commented Jul 24, 2024 • edited Loading

Codecov Report

vdaas-ci commented Jul 25, 2024

github-actions bot Jul 25, 2024

Choose a reason for hiding this comment

github-actions bot Jul 25, 2024

Choose a reason for hiding this comment

github-actions bot Jul 25, 2024

Choose a reason for hiding this comment

github-actions bot Jul 25, 2024

Choose a reason for hiding this comment

github-actions bot Jul 25, 2024

Choose a reason for hiding this comment

github-actions bot Jul 25, 2024

Choose a reason for hiding this comment

github-actions bot Jul 25, 2024

Choose a reason for hiding this comment

github-actions bot Jul 25, 2024

Choose a reason for hiding this comment

github-actions bot Jul 25, 2024

Choose a reason for hiding this comment

github-actions bot Jul 25, 2024

Choose a reason for hiding this comment

kpango commented Jul 24, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

cloudflare-workers-and-pages bot commented Jul 24, 2024 •

edited

Loading

codecov bot commented Jul 24, 2024 •

edited

Loading