Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](meta-service) Support querying and adjusting rpc qps limit on meta service #42413

Merged
merged 3 commits into from
Nov 13, 2024
Merged

[feature](meta-service) Support querying and adjusting rpc qps limit on meta service #42413

merged 3 commits into from
Nov 13, 2024

Conversation

TangSiyang2001
Copy link
Collaborator

@TangSiyang2001 TangSiyang2001 commented Oct 24, 2024 •
edited
Loading

Proposed changes

Usage

  1. adjust limit
curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?${params}
Entry Description
param uint64 qps_limit
behavior set qps_limit global default value
example curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000
Entry Description
param uint64 qps_limit, string rpc_name
behavior set RPC specific qps_limit
example curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster
Entry Description
param uint64 qps_limit, string rpc_name, string instance_id
behavior set instance qps_limit for specific RPC
example ccurl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster&instance_id="doris-0"
Entry Description
param uint64 qps_limit, string instance_id
behavior set global qps_limit for specific instance
example curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&instance_id="doris-0"
  1. query limit
Entry Description
param none
behavior query qps limit for all RPC interface
example curl http://ms_ip:ms_port/MetaService/http/v1/list_rate_limit

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions[bot]
github-actions bot reviewed Oct 24, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

cloud/src/rate-limiter/rate_limiter.h
@@ -19,10 +19,13 @@

#include <brpc/server.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'brpc/server.h' file not found [clang-diagnostic-error]

#include <brpc/server.h>
         ^

github-actions[bot]
github-actions bot reviewed Oct 25, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

cloud/test/rate_limiter_test.cpp
@@ -17,8 +17,11 @@

#include "rate-limiter/rate_limiter.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'rate-limiter/rate_limiter.h' file not found [clang-diagnostic-error]

#include "rate-limiter/rate_limiter.h"
         ^

@TangSiyang2001 TangSiyang2001 force-pushed the dynamic-ms-rate-limit branch 2 times, most recently from cb4d256 to 063ca50 Compare October 25, 2024 12:08
@TangSiyang2001
Copy link
Collaborator Author

run buildall

@TangSiyang2001
Copy link
Collaborator Author

run buildall

@TangSiyang2001
Copy link
Collaborator Author

run buildall

@TangSiyang2001 TangSiyang2001 marked this pull request as ready for review October 28, 2024 06:13
@TangSiyang2001 TangSiyang2001 marked this pull request as draft October 29, 2024 12:13
@TangSiyang2001
Copy link
Collaborator Author

run buildall

@TangSiyang2001
Copy link
Collaborator Author

run buildall

github-actions[bot]
github-actions bot reviewed Oct 31, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@TangSiyang2001
Copy link
Collaborator Author

run buildall

@TangSiyang2001 TangSiyang2001 marked this pull request as ready for review November 1, 2024 01:53
@gavinchou
Copy link
Contributor

Add some usage examples in description.

@TangSiyang2001
Copy link
Collaborator Author

run buildall

github-actions[bot]
github-actions bot reviewed Nov 5, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

gavinchou
gavinchou reviewed Nov 10, 2024
View reviewed changes
cloud/src/meta-service/meta_service_http.cpp Outdated
qps_limit_str, rpc_name, instance_id));
};

static auto parse_qps_limit =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems that abuse using lambdas

gavinchou
gavinchou reviewed Nov 10, 2024
View reviewed changes
cloud/src/meta-service/meta_service_http.cpp Outdated
});
};

// for 8 element in true table of params, register processor cb
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this part is confusing

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'll add some comments, just register different callback for different params pattern, and the "abuse" of lamda are just meant to adapt for this.

gavinchou
gavinchou requested changes Nov 10, 2024
View reviewed changes
@TangSiyang2001
Copy link
Collaborator Author

run buildall

@TangSiyang2001
Copy link
Collaborator Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 13, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@TangSiyang2001
Copy link
Collaborator Author

run compile

@TangSiyang2001
Copy link
Collaborator Author

run cloud_p0

@TangSiyang2001
Copy link
Collaborator Author

run p0

@TangSiyang2001
Copy link
Collaborator Author

run external

@gavinchou gavinchou merged commit 4783fc0 into apache:master Nov 13, 2024
26 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 13, 2024
@TangSiyang2001
[feature](meta-service) Support querying and adjusting rpc qps limit …
1e73ffe 
…on meta service (#42413)

## Proposed changes

Usage
1. adjust limit
```
curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?${params}
```
| Entry      | Description |
| ----------- | ----------- |
| param         | uint64 qps_limit  |
|behavior   | set qps_limit global default value |
|example|```curl
http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000```|

| Entry      | Description |
| ----------- | ----------- |
| param         | uint64 qps_limit, string rpc_name  |
|behavior   | set RPC specific qps_limit  |
|example|curl
http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster|

| Entry      | Description |
| ----------- | ----------- |
| param | uint64 qps_limit, string rpc_name, string instance_id |
|behavior   | set instance qps_limit for specific RPC  |
|example|```ccurl
http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster&instance_id="doris-0"```|

| Entry      | Description |
| ----------- | ----------- |
| param         | uint64 qps_limit, string instance_id  |
|behavior   | set global qps_limit for specific instance  |
|example|```curl
http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&instance_id="doris-0"```|

2. query limit

| Entry      | Description |
| ----------- | ----------- |
| param         | none  |
|behavior   | query qps limit for all RPC interface |
|example|```curl
http://ms_ip:ms_port/MetaService/http/v1/list_rate_limit```|
@github-actions github-actions bot mentioned this pull request Nov 13, 2024
Merged
dataroaring pushed a commit that referenced this pull request Nov 14, 2024
@TangSiyang2001 @dataroaring
[feature](meta-service) Support querying and adjusting rpc qps limit …
07f5b95 
…on meta service (#42413)

## Proposed changes

Usage
1. adjust limit
```
curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?${params}
```
| Entry      | Description |
| ----------- | ----------- |
| param         | uint64 qps_limit  |
|behavior   | set qps_limit global default value |
|example|```curl
http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000```|

| Entry      | Description |
| ----------- | ----------- |
| param         | uint64 qps_limit, string rpc_name  |
|behavior   | set RPC specific qps_limit  |
|example|curl
http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster|

| Entry      | Description |
| ----------- | ----------- |
| param | uint64 qps_limit, string rpc_name, string instance_id |
|behavior   | set instance qps_limit for specific RPC  |
|example|```ccurl
http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster&instance_id="doris-0"```|

| Entry      | Description |
| ----------- | ----------- |
| param         | uint64 qps_limit, string instance_id  |
|behavior   | set global qps_limit for specific instance  |
|example|```curl
http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&instance_id="doris-0"```|

2. query limit

| Entry      | Description |
| ----------- | ----------- |
| param         | none  |
|behavior   | query qps limit for all RPC interface |
|example|```curl
http://ms_ip:ms_port/MetaService/http/v1/list_rate_limit```|
gavinchou pushed a commit that referenced this pull request Nov 14, 2024
@github-actions @TangSiyang2001
branch-3.0: [feature](meta-service) Support querying and adjusting rp…
663845a 
…c qps limit on meta service #42413 (#43884)

Cherry-picked from #42413

Co-authored-by: Siyang Tang <tangsiyang2001@foxmail.com>
@TangSiyang2001 TangSiyang2001 mentioned this pull request Nov 20, 2024
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request Nov 21, 2024
@TangSiyang2001
[Fix](ms) Fix ci-uncaught ut coredump due to dynamic rate limit setti…
012620c 
…ng (#44362)

UT failed with coredump in PR #42413 but was not detected by CI. Fix the
problem.
github-actions bot pushed a commit that referenced this pull request Nov 21, 2024
@TangSiyang2001
[Fix](ms) Fix ci-uncaught ut coredump due to dynamic rate limit setti…
63cd07d 
…ng (#44362)

UT failed with coredump in PR #42413 but was not detected by CI. Fix the
problem.
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@gavinchou gavinchou gavinchou approved these changes

@luwei16 luwei16 luwei16 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.3-merged p0_c reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@TangSiyang2001 @doris-robot @gavinchou @luwei16 @dataroaring