feat(api): support metric API Prometheus format & add statistic metric api #2286

SunnyBoy-WYH · 2023-08-16T14:20:52Z

fix #1656 #1356 (will enhance it later)

术语：
统计指标：指【请求总数、成功数、失败数、平均响应时间、最大响应时间】五个指标

基础指标：指serverReporter中原有的gauges、counters、histograms等

思路：

1.增加原有Metric接口的prom格式返回：

# HELP hugegraph_info
# TYPE hugegraph_info untyped
hugegraph_info{version="0.69",} 1.0
# HELP org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_capacity
# TYPE org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_capacity gauge
org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_capacity 1000000
# HELP org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_expire
# TYPE org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_expire gauge
org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_expire 600000
# HELP org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_hits

2.增加statistics接口，返回【请求总数、成功数、失败数、平均响应时间、最大响应时间】五个指标，同时支持prop格式返回
json：

{
"metrics/GET":{
"FAILED_REQUEST":0,
"MEAN_RESPONSE_TIME":10.047648352712768,
"MAX_RESPONSE_TIME":27,
"SUCCESS_REQUEST":12,
"TOTAL_REQUEST":12
},
"metrics/statistics/GET":{
"FAILED_REQUEST":0,
"MEAN_RESPONSE_TIME":0.6234681965773761,
"MAX_RESPONSE_TIME":2,
"SUCCESS_REQUEST":3,
"TOTAL_REQUEST":3
}
}
prom格式：

# HELP hugegraph_info
# TYPE hugegraph_info untyped
hugegraph_info{version="0.69",} 1.0
# HELP metrics_GET
# TYPE metrics_GET gauge
metrics_GET{name=FAILED_REQUEST,} 0
metrics_GET{name=MEAN_RESPONSE_TIME,} 10.047648352712768
metrics_GET{name=MAX_RESPONSE_TIME,} 27
metrics_GET{name=SUCCESS_REQUEST,} 12
metrics_GET{name=TOTAL_REQUEST,} 12

3.需要讨论一下 statistics接口的数据是否要合并在Metric接口中，目前没有合并是因为，原有指标结果JSON中只包含counter、summary等五种标准指标；

另外分开还有个好处是，如果后续想要按照图、接口查询统计指标，比较方便定制

hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/AccessLogFilter.java

hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java

JackyYangPassion · 2023-08-16T14:43:50Z

hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java

    private static final Meter EXPECTED_ERROR_METER =
-                         MetricsUtil.registerMeter(API.class, "expected-error");
+            MetricsUtil.registerMeter(API.class, "expected-error");


JackyYangPassion · 2023-08-16T14:43:56Z

hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java

    private static final Meter UNKNOWN_ERROR_METER =
-                         MetricsUtil.registerMeter(API.class, "unknown-error");
+            MetricsUtil.registerMeter(API.class, "unknown-error");


hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java

JackyYangPassion · 2023-08-16T14:44:27Z

hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java

@@ -97,7 +95,7 @@ public static <R> R commit(HugeGraph g, Callable<R> callable) {
            SUCCEED_METER.mark();
            return result;
        } catch (IllegalArgumentException | NotFoundException |
-                 ForbiddenException e) {
+                ForbiddenException e) {


not addressed comment

hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java

JackyYangPassion · 2023-08-16T14:46:17Z

hugegraph-test/src/main/java/org/apache/hugegraph/api/MetricsApiTest.java

@@ -165,8 +190,8 @@ public void testMetricsBackend() {
                    String key = (String) e.getKey();
                    value = e.getValue();
                    Assert.assertTrue(String.format(
-                                      "Expect map value for key %s but got %s",
-                                      key, value),
+                                              "Expect map value for key %s but got %s",


prefer to keep the origin alignment

not addressed: prefer to keep the origin alignment

# Conflicts: # hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java

imbajin · 2023-08-23T03:36:38Z

@SunnyBoy-WYH seems CI failed? could check the CI status in your personal repo (submit a PR to your repo, u could check/test CI there)

SunnyBoy-WYH · 2023-08-23T03:47:50Z

@SunnyBoy-WYH seems CI failed? could check the CI status in your personal repo (submit a PR to your repo, u could check/test CI there)

Nice idea! ( when i push this pr, the local test passed, and a few days before i am looking for the ci problem,but ci info not enough.) i will try it again.

codecov · 2023-09-10T03:29:54Z

Codecov Report

Merging #2286 (6216976) into master (4ceef1a) will decrease coverage by 3.23%.
The diff coverage is 95.25%.

@@             Coverage Diff              @@
##             master    #2286      +/-   ##
============================================
- Coverage     68.26%   65.04%   -3.23%     
  Complexity      989      989              
============================================
  Files           500      503       +3     
  Lines         41516    41801     +285     
  Branches       5786     5804      +18     
============================================
- Hits          28342    27189    -1153     
- Misses        10415    11920    +1505     
+ Partials       2759     2692      -67

Files Changed	Coverage Δ
...pi/src/main/java/org/apache/hugegraph/api/API.java	`70.00% <ø> (ø)`
...java/org/apache/hugegraph/metrics/MetricsKeys.java	`90.90% <90.90%> (ø)`
...java/org/apache/hugegraph/metrics/MetricsUtil.java	`91.26% <93.81%> (+28.76%)`	⬆️
...a/org/apache/hugegraph/api/metrics/MetricsAPI.java	`89.79% <95.78%> (+24.00%)`	⬆️
...g/apache/hugegraph/api/filter/AccessLogFilter.java	`100.00% <100.00%> (ø)`
...va/org/apache/hugegraph/api/filter/PathFilter.java	`100.00% <100.00%> (ø)`

... and 69 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java

hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/AccessLogFilter.java

hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/PathFilter.java

hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/AccessLogFilter.java

hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/PathFilter.java

hugegraph-api/src/main/java/org/apache/hugegraph/api/metrics/MetricsAPI.java

hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsKeys.java

hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java

hugegraph-test/src/main/java/org/apache/hugegraph/api/MetricsApiTest.java

javeme

there are still some comments to be addressed, thanks