Skip to content

Commit

Permalink
Merge latest changes from rate_limiting to master (#221)
Browse files Browse the repository at this point in the history
* Point to googleapi in service control client. (#91)

* Point to googleapi in service control client.

* Use git repository for service-control-client.

* Merge latest changes from master (#104)

* Get attributes from envoy config. (#87)

* Send all attributes.

* Remove unused const strings.

* Address comment.

* updated SHA to point to newer envoy with RDS API feature (#94)

* Disable travis on stable branches (#96)

* Publish debug binaries (no release yet) (#98)

* Copies the binary instead of linking for release (#102)

* Extract quota config from service config. (#101)

* Add metric_cost in config.

* Remove group rules.

* Call loadQuotaConfig in config::create.

* Update latest update from master branch (#106)

* Get attributes from envoy config. (#87)

* Send all attributes.

* Remove unused const strings.

* Address comment.

* updated SHA to point to newer envoy with RDS API feature (#94)

* Disable travis on stable branches (#96)

* Publish debug binaries (no release yet) (#98)

* Copies the binary instead of linking for release (#102)

* Added quota contoll without the service control client library (#93)

* Added quota contoll without the service control client library

* Applied code review

* Applied code review

* Resolve conflicts

* Resolve conflicts

* Fixed format error reported by script/check-style

* Fixed a bug at Aggregated::GetAuthToken that causes Segmentation Fault

* Changed usage of template funcion

* Applied latest changes from the repo

* Applied latest changes from the repo

* Applied latest changes from the repo

* Adde comments

* Updated log information

* Applied #101

* Changed metric_cost_map to metric_cost_vector

* Fixed test case compilation error

* Fixed test case compilation error

* Add unit test for quota config. (#108)

* Add unit test for quota config.

* Add comments.

* Update test specifics.

* Merge latest changes from master branch (#112)

* Get attributes from envoy config. (#87)

* Send all attributes.

* Remove unused const strings.

* Address comment.

* updated SHA to point to newer envoy with RDS API feature (#94)

* Disable travis on stable branches (#96)

* Publish debug binaries (no release yet) (#98)

* Copies the binary instead of linking for release (#102)

* Not to use api_key if its service is not actived. (#109)

* If QuotaControl service is not available, return utils::Status::OK (#113)

* If QuotaControl service is not available, return utils::Status::OK

* Updated comment

* Return HTTP status code 429 on google.rpc.Code.RESOURCE_EXHAUSTED (#119)

* Fixed incorrectly resolved conflicts (#123)

* Added unit test cases for rate limiting (#124)

* Fixed incorrectly resolved conflicts

* Added unit test cases for rate limiting

* Added unit test cases for rate limiting

* Added unit test cases for rate limiting

* Added unit test cases for rate limiting

* Added unit test cases for rate limiting

* Added unit test cases for rate limiting

* Rename response.http.code (#125) (#128)

* Added handling of error code QUOTA_SYSTEM_UNAVAILABLE (#148)

* Integrated service control client library with quota cache aggregation (#149)

* Fixed error on merge (#151)

* Integrated service control client library with quota cache aggregation

* Fixed error on merge

* Fixed the compatibility issue with the latest update on esp (#152)

* Removed copied proto files (#208)

* Set default allocate quota request timeout to 1sec and applied latest service control client library change (#211)

* Merged key_restriction related changes from master (#213)

* Merge latest changes from master branch (#217)

* Not call report if decodeHeaders is not called. (#150)

* Update mixerclient with sync-ed grpc write and fail-fast. (#155)

* Update mixerclient with sync-ed write and fail-fast.

* Update to latest test.

* Update again

* Update envoy to PR553 (#156)

* Update envoy to PR553

* Update libevent to 2.1.8

* Uses a specific version of the Shared Pipeline lib (#158)

* Update lyft/envoy commit Id to latest. (#161)

* Update lyft/envoy commit Id to latest.

* Remove the comment about pull request

* Add new line - will delete in next commit.

* Update repositories.bzl (#169)

* Always set response latency (#172)

* Update mixerclient to sync_transport change. (#178)

* Use opaque config to turn on/off forward attribute and mixer filter (#179)

* Modify mixer filter

* Swap defaults

* Make the filter decoder only

* cache mixer disabled decision

* Fix a bug in opaque config change and test it out (#182)

* Fix a bug and test it out

* Update filter type

* Update README.md

* Update mixer client to mixer api with gogoproto. (#184)

* Move .bazelrc to tools/bazel.rc (#186)

* Move .bazelrc to tools/bazel.rc

* Update Jenkinsfile with latest version of pipeline

* Support apikey based traffic restriction (#189)

* b/36368559 support apikey based traffic restriction

* Fixed code formatting

* Fix crash in unreachable/overloaded RDS (#190)

* Add mixer client end to end integration test. (#177)

* Add mixer client end to end integration test.

* Split some repositories into a separate file.

* use real mixer for fake mixer_server.

* Test repository

* use mixer bzl file.

* Use mixer repositories

* Not to use mixer repository.

* Add return line at the end of WORKSPACE.

* Fix broken link (#193)

* Make quota call (#192)

* hookup quota call

* Make quota call.

* Update indent.

* Update envoy and update configs (#195)

* Update envoy and update configs

* Use gcc-4.9 for travis

* Use bazel 0.4.5

* Fix SHA of lightstep-tracer-common

* Enable check cache and refactory mixer config loading  (#197)

* Refactory the mixer config loading.

* fix format

* Add integration test.

* updated README.md

* s/send/sent/

* Split into separate tests. (#201)

* Update README on how to enable check cache. (#204)

* Update README on how to enable check cache.

* Update the comment.

* build: support Envoy native Bazel build. (#210)

* build: support Envoy native Bazel build.

This patch switches the Envoy build from src/envoy/repositories.bzl to
using the upstream native build.

See envoyproxy/envoy#663 for the corresponding changes
on the Envoy side.

* Use Envoy master with BUILD.wip rename merged.

* Fix clang-format issues.

* Fixes bazel.rc issues (#212)

* Fixes bazel rc issues

* Update Jenkins to latest pipeline version

* Updated the commit id of cloudendpoints/service-control-client-cxx (#218)

* Update commitid of cloudendpoints/service-control-client-cxx repo (#220)
  • Loading branch information
mangchiandjjoe authored Apr 7, 2017
1 parent 987223f commit 1ef1b4a
Show file tree
Hide file tree
Showing 32 changed files with 1,145 additions and 51 deletions.
4 changes: 4 additions & 0 deletions contrib/endpoints/include/api_manager/method.h
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,10 @@ class MethodInfo {

// Get the names of url system parameters
virtual const std::set<std::string> &system_query_parameter_names() const = 0;

// Get quota metric cost vector
virtual const std::vector<std::pair<std::string, int>> &metric_cost_vector()
const = 0;
};

} // namespace api_manager
Expand Down
22 changes: 15 additions & 7 deletions contrib/endpoints/repositories.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -211,12 +211,9 @@ def googleapis_repositories(protobuf_repo="@protobuf_git//", bind=True):
#
################################################################################
#
licenses(["notice"])
load("{}:protobuf.bzl", "cc_proto_library")
exports_files(glob(["google/**"]))
load("@protobuf_git//:protobuf.bzl", "cc_proto_library")
cc_proto_library(
name = "servicecontrol",
Expand Down Expand Up @@ -259,9 +256,13 @@ cc_proto_library(
"google/api/log.proto",
"google/api/logging.proto",
"google/api/metric.proto",
"google/api/experimental/experimental.proto",
"google/api/experimental/authorization_config.proto",
"google/api/monitored_resource.proto",
"google/api/monitoring.proto",
"google/api/quota.proto",
"google/api/service.proto",
"google/api/source_info.proto",
"google/api/system_parameter.proto",
"google/api/usage.proto",
],
Expand Down Expand Up @@ -290,10 +291,9 @@ cc_proto_library(
)
""".format(protobuf_repo)


native.new_git_repository(
name = "googleapis_git",
commit = "db1d4547dc56a798915e0eb2c795585385922165",
commit = "2fe0050bd2a6d4c6ba798c0311f0b149b8997314",
remote = "https://github.com/googleapis/googleapis.git",
build_file_content = BUILD,
)
Expand Down Expand Up @@ -324,7 +324,7 @@ def servicecontrol_client_repositories(bind=True):

native.git_repository(
name = "servicecontrol_client_git",
commit = "d739d755365c6a13d0b4164506fd593f53932f5d",
commit = "3d1a30d9221e700542eeaaf20eab69faddb63894",
remote = "https://github.com/cloudendpoints/service-control-client-cxx.git",
)

Expand All @@ -333,3 +333,11 @@ def servicecontrol_client_repositories(bind=True):
name = "servicecontrol_client",
actual = "@servicecontrol_client_git//:service_control_client_lib",
)
native.bind(
name = "quotacontrol",
actual = "@servicecontrol_client_git//proto:quotacontrol",
)
native.bind(
name = "quotacontrol_genproto",
actual = "@servicecontrol_client_git//proto:quotacontrol_genproto",
)
2 changes: 2 additions & 0 deletions contrib/endpoints/src/api_manager/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,8 @@ cc_library(
"method_impl.cc",
"path_matcher.cc",
"path_matcher_node.cc",
"quota_control.cc",
"quota_control.h",
"request_handler.cc",
],
linkopts = select({
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ class ServiceAccountToken {
enum JWT_TOKEN_TYPE {
JWT_TOKEN_FOR_SERVICE_CONTROL = 0,
JWT_TOKEN_FOR_CLOUD_TRACING,
JWT_TOKEN_FOR_QUOTA_CONTROL,
JWT_TOKEN_TYPE_MAX,
};
// Set audience. Only calcualtes JWT token with specified audience.
Expand Down
3 changes: 3 additions & 0 deletions contrib/endpoints/src/api_manager/check_workflow.cc
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
#include "contrib/endpoints/src/api_manager/check_auth.h"
#include "contrib/endpoints/src/api_manager/check_service_control.h"
#include "contrib/endpoints/src/api_manager/fetch_metadata.h"
#include "contrib/endpoints/src/api_manager/quota_control.h"

using ::google::api_manager::utils::Status;

Expand All @@ -33,6 +34,8 @@ void CheckWorkflow::RegisterAll() {
Register(CheckAuth);
// Checks service control.
Register(CheckServiceControl);
// Quota control
Register(QuotaControl);
}

void CheckWorkflow::Register(CheckHandler handler) {
Expand Down
20 changes: 20 additions & 0 deletions contrib/endpoints/src/api_manager/config.cc
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,23 @@ MethodInfoImpl *Config::GetOrCreateMethodInfoImpl(const string &name,
return i->second.get();
}

bool Config::LoadQuotaRule(ApiManagerEnvInterface *env) {
for (const auto &rule : service_.quota().metric_rules()) {
auto method = utils::FindOrNull(method_map_, rule.selector());
if (method) {
for (auto &metric_cost : rule.metric_costs()) {
(*method)->add_metric_cost(metric_cost.first, metric_cost.second);
}
} else {
env->LogError("Metric rule with selector " + rule.selector() +
"is mismatched.");
return false;
}
}

return true;
}

bool Config::LoadHttpMethods(ApiManagerEnvInterface *env,
PathMatcherBuilder *pmb) {
std::set<std::string> all_urls, urls_with_options;
Expand Down Expand Up @@ -443,6 +460,9 @@ std::unique_ptr<Config> Config::Create(ApiManagerEnvInterface *env,
if (!config->LoadBackends(env)) {
return nullptr;
}
if (!config->LoadQuotaRule(env)) {
return nullptr;
}
return config;
}

Expand Down
3 changes: 3 additions & 0 deletions contrib/endpoints/src/api_manager/config.h
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
#include "contrib/endpoints/src/api_manager/method_impl.h"
#include "contrib/endpoints/src/api_manager/path_matcher.h"
#include "contrib/endpoints/src/api_manager/proto/server_config.pb.h"
#include "google/api/quota.pb.h"
#include "google/api/service.pb.h"

namespace google {
Expand Down Expand Up @@ -111,6 +112,8 @@ class Config {
// Load SystemParameters info to MethodInfo.
bool LoadSystemParameters(ApiManagerEnvInterface *env);

bool LoadQuotaRule(ApiManagerEnvInterface *env);

// Gets the MethodInfoImpl creating it if necessary
MethodInfoImpl *GetOrCreateMethodInfoImpl(const std::string &name,
const std::string &api_name,
Expand Down
84 changes: 84 additions & 0 deletions contrib/endpoints/src/api_manager/config_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -870,6 +870,90 @@ TEST(Config, TestCorsDisabled) {
ASSERT_EQ(nullptr, method1);
}

TEST(Config, TestInvalidMetricRules) {
MockApiManagerEnvironmentWithLog env;
// There is no http.rule or api.method to match the selector.
static const char config_text[] = R"(
name: "Service.Name"
quota {
metric_rules {
selector: "GetShelves"
metric_costs {
key: "test.googleapis.com/operation/read_book"
value: 100
}
}
}
)";

std::unique_ptr<Config> config = Config::Create(&env, config_text, "");
EXPECT_EQ(nullptr, config);
}

TEST(Config, TestMetricRules) {
MockApiManagerEnvironmentWithLog env;
static const char config_text[] = R"(
name: "Service.Name"
http {
rules {
selector: "DeleteShelf"
delete: "/shelves"
}
rules {
selector: "GetShelves"
get: "/shelves"
}
}
quota {
metric_rules {
selector: "GetShelves"
metric_costs {
key: "test.googleapis.com/operation/get_shelves"
value: 100
}
metric_costs {
key: "test.googleapis.com/operation/request"
value: 10
}
}
metric_rules {
selector: "DeleteShelf"
metric_costs {
key: "test.googleapis.com/operation/delete_shelves"
value: 200
}
}
}
)";

std::unique_ptr<Config> config = Config::Create(&env, config_text, "");
ASSERT_TRUE(config);

const MethodInfo *method1 = config->GetMethodInfo("GET", "/shelves");
ASSERT_NE(nullptr, method1);

std::vector<std::pair<std::string, int>> metric_cost_vector =
method1->metric_cost_vector();
std::sort(metric_cost_vector.begin(), metric_cost_vector.end());
ASSERT_EQ(2, metric_cost_vector.size());
ASSERT_EQ("test.googleapis.com/operation/get_shelves",
metric_cost_vector[0].first);
ASSERT_EQ(100, metric_cost_vector[0].second);

ASSERT_EQ("test.googleapis.com/operation/request",
metric_cost_vector[1].first);
ASSERT_EQ(10, metric_cost_vector[1].second);

const MethodInfo *method2 = config->GetMethodInfo("DELETE", "/shelves");
ASSERT_NE(nullptr, method1);

metric_cost_vector = method2->metric_cost_vector();
ASSERT_EQ(1, metric_cost_vector.size());
ASSERT_EQ("test.googleapis.com/operation/delete_shelves",
metric_cost_vector[0].first);
ASSERT_EQ(200, metric_cost_vector[0].second);
}

} // namespace

} // namespace api_manager
Expand Down
10 changes: 10 additions & 0 deletions contrib/endpoints/src/api_manager/context/request_context.cc
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,16 @@ void RequestContext::FillCheckRequestInfo(
request_->FindHeader(kXIosBundleId, &info->ios_bundle_id);
}

void RequestContext::FillAllocateQuotaRequestInfo(
service_control::QuotaRequestInfo *info) {
FillOperationInfo(info);

info->client_ip = request_->GetClientIP();
info->method_name = this->method_call_.method_info->name();
info->metric_cost_vector =
&this->method_call_.method_info->metric_cost_vector();
}

void RequestContext::FillReportRequestInfo(
Response *response, service_control::ReportRequestInfo *info) {
FillOperationInfo(info);
Expand Down
3 changes: 3 additions & 0 deletions contrib/endpoints/src/api_manager/context/request_context.h
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ class RequestContext {
// Fill CheckRequestInfo
void FillCheckRequestInfo(service_control::CheckRequestInfo *info);

// FillAllocateQuotaRequestInfo
void FillAllocateQuotaRequestInfo(service_control::QuotaRequestInfo *info);

// Fill ReportRequestInfo
void FillReportRequestInfo(Response *response,
service_control::ReportRequestInfo *info);
Expand Down
18 changes: 15 additions & 3 deletions contrib/endpoints/src/api_manager/method_impl.h
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
#include <map>
#include <memory>
#include <set>
#include <vector>

#include "contrib/endpoints/include/api_manager/method.h"
#include "contrib/endpoints/src/api_manager/utils/stl_util.h"
Expand Down Expand Up @@ -62,6 +63,10 @@ class MethodInfoImpl : public MethodInfo {

const std::string &backend_address() const { return backend_address_; }

const std::vector<std::pair<std::string, int>> &metric_cost_vector() const {
return metric_cost_vector_;
}

const std::string &rpc_method_full_name() const {
return rpc_method_full_name_;
}
Expand Down Expand Up @@ -90,6 +95,10 @@ class MethodInfoImpl : public MethodInfo {
url_query_parameters_[name].push_back(url_query_parameter);
}

void add_metric_cost(const std::string &metric, int64_t cost) {
metric_cost_vector_.push_back(std::make_pair(metric, cost));
}

// After add all system parameters, lookup some of them to cache
// their lookup results.
void process_system_parameters();
Expand Down Expand Up @@ -139,13 +148,13 @@ class MethodInfoImpl : public MethodInfo {
// such as API Key)?
bool allow_unregistered_calls_;
// Issuers to allowed audiences map.
std::map<std::string, std::set<std::string> > issuer_audiences_map_;
std::map<std::string, std::set<std::string>> issuer_audiences_map_;

// system parameter map of parameter name to http_header name.
std::map<std::string, std::vector<std::string> > http_header_parameters_;
std::map<std::string, std::vector<std::string>> http_header_parameters_;

// system parameter map of parameter name to url query parameter name.
std::map<std::string, std::vector<std::string> > url_query_parameters_;
std::map<std::string, std::vector<std::string>> url_query_parameters_;

// all the names of system query parameters
std::set<std::string> system_query_parameter_names_;
Expand Down Expand Up @@ -175,6 +184,9 @@ class MethodInfoImpl : public MethodInfo {

// Whether the response is streaming or not.
bool response_streaming_;

// map of metric and its cost
std::vector<std::pair<std::string, int>> metric_cost_vector_;
};

typedef std::unique_ptr<MethodInfoImpl> MethodInfoImplPtr;
Expand Down
2 changes: 2 additions & 0 deletions contrib/endpoints/src/api_manager/mock_method_info.h
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ class MockMethodInfo : public MethodInfo {
MOCK_CONST_METHOD0(response_streaming, bool());
MOCK_CONST_METHOD0(system_query_parameter_names,
const std::set<std::string>&());
MOCK_CONST_METHOD0(metric_cost_vector,
const std::vector<std::pair<std::string, int>>&());
};

} // namespace api_manager
Expand Down
18 changes: 18 additions & 0 deletions contrib/endpoints/src/api_manager/proto/server_config.proto
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,13 @@ message ServiceControlConfig {
// The intermediate reports for streaming calls should not be more frequent
// than this value (in seconds)
int32 intermediate_report_min_interval = 7;

// Quota aggregator config
QuotaAggregatorConfig quota_aggregator_config = 8;

// Timeout in milliseconds on service control allocate quota requests.
// If the value is <= 0, default timeout is 5000 milliseconds.
int32 quota_timeout_ms = 9;
}

// Check aggregator config
Expand All @@ -82,6 +89,17 @@ message CheckAggregatorConfig {
int32 response_expiration_ms = 3;
}

// Quota aggregator config
message QuotaAggregatorConfig {
// The maximum number of cache entries that can be kept in the aggregation
// cache. Cache is disabled when entries <= 0.
int32 cache_entries = 1;

// The maximum milliseconds before aggregated quota requests are refreshed to
// the server.
int32 refresh_interval_ms = 2;
}

// Report aggregator config
message ReportAggregatorConfig {
// The maximum number of cache entries that can be kept in the aggregation
Expand Down
Loading

0 comments on commit 1ef1b4a

Please sign in to comment.