Add service label on metrics #1024

eloycoto · 2019-04-30T06:57:40Z

This commits adds support to have additional information on some metrics[0],
this commit enables users to know the service affected by the metric.

Due to this can be a problematic for Prometheus[1], to enable this a new ENV
variable(APICAST_EXTENDED_METRICS) is defined, so if users that have a few services
they can run the extended metrics without affecting Prometheus stability, if
they thousands of services the recommendation is to have this option disabled.

In case of the extended_metrics is disabled, all the service labels will use
"all" value and Prometheus performance will not be affected.

The new metrics list looks like this:

bash-4.2$ curl http://localhost:9421/metrics -s | grep service
total_response_time_seconds_bucket{service="all",le="00.100"} 91
total_response_time_seconds_bucket{service="all",le="00.200"} 189
total_response_time_seconds_bucket{service="all",le="00.300"} 220
total_response_time_seconds_bucket{service="all",le="00.400"} 220
total_response_time_seconds_bucket{service="all",le="00.500"} 222
total_response_time_seconds_bucket{service="all",le="00.750"} 223
total_response_time_seconds_bucket{service="all",le="01.000"} 223
total_response_time_seconds_bucket{service="all",le="01.500"} 224
total_response_time_seconds_bucket{service="all",le="02.000"} 224
total_response_time_seconds_bucket{service="all",le="03.000"} 224
total_response_time_seconds_bucket{service="all",le="04.000"} 224
total_response_time_seconds_bucket{service="all",le="05.000"} 224
total_response_time_seconds_bucket{service="all",le="10.000"} 224
total_response_time_seconds_bucket{service="all",le="+Inf"} 224
total_response_time_seconds_count{service="all"} 224
total_response_time_seconds_sum{service="all"} 33.616
upstream_response_time_seconds_bucket{service="all",le="00.100"} 93
upstream_response_time_seconds_bucket{service="all",le="00.200"} 190
upstream_response_time_seconds_bucket{service="all",le="00.300"} 220
upstream_response_time_seconds_bucket{service="all",le="00.400"} 220
upstream_response_time_seconds_bucket{service="all",le="00.500"} 223
upstream_response_time_seconds_bucket{service="all",le="00.750"} 224
upstream_response_time_seconds_bucket{service="all",le="01.000"} 224
upstream_response_time_seconds_bucket{service="all",le="01.500"} 224
upstream_response_time_seconds_bucket{service="all",le="02.000"} 224
upstream_response_time_seconds_bucket{service="all",le="03.000"} 224
upstream_response_time_seconds_bucket{service="all",le="04.000"} 224
upstream_response_time_seconds_bucket{service="all",le="05.000"} 224
upstream_response_time_seconds_bucket{service="all",le="10.000"} 224
upstream_response_time_seconds_bucket{service="all",le="+Inf"} 224
upstream_response_time_seconds_count{service="all"} 224
upstream_response_time_seconds_sum{service="all"} 32.226
upstream_status{status="200",service="all"} 224

[0] List of affected metrics:
total_response_time_seconds
upstream_response_time_seconds
upstream_status

[1] https://prometheus.io/docs/practices/naming/#labels

gateway/src/apicast/policy/nginx_metrics/nginx_metrics.lua

mikz · 2019-04-30T07:03:49Z

gateway/src/apicast/policy/nginx_metrics/nginx_metrics.lua

+    service = "all"
+  end
+  upstream_metrics.report(ngx.var.upstream_status, ngx.var.upstream_response_time, service)
+  report_req_response_time(service)


I know this is unrelated to this PR. But we probably should also introduce a counter of how many requests the gateway processed (and have a label for each service). So prometheus can detect requests per minute for example.

t/prometheus-metrics.t

mikz

I'll leave the approval on David, but this is excellent work! 👍

doc/parameters.md

davidor · 2019-04-30T09:41:12Z

spec/metrics/upstream_spec.lua

@@ -36,8 +37,8 @@ describe('upstream metrics', function()
    end)

    it('adds the latency to the histogram', function()
-      upstream_metrics.report(200, 0.1)
-      assert.stub(test_histogram.observe).was_called_with(test_histogram, 0.1)
+      upstream_metrics.report(200, 0.1, service_metric_name)


We should probably test with and without service. To make sure that both test cases work correctly.

Also, we should do the same for the status codes counter.

spec/metrics/upstream_spec.lua

gateway/src/apicast/metrics/upstream.lua

t/prometheus-metrics.t

davidor · 2019-04-30T09:58:48Z

@eloycoto the new metrics should be added to https://github.com/3scale/APIcast/blob/master/doc/prometheus-metrics.md

t/prometheus-metrics.t

eloycoto · 2019-04-30T17:32:24Z

I think that I found the issue with the test[0].

On the executor, where the phase log happens, the ngx.ctx.context is nil, so
it creates a new context and the old one is not updated.

APIcast/gateway/src/apicast/executor.lua

Lines 40 to 51 in 66168b7

    
           local function shared_build_context(executor) 
        
               local ctx = ngx.ctx or {} 
        
               local context = ctx.context 
        
               if not context then 
        
                   context = build_context(executor) 
        
                   ctx.context = context 
        
               end 
        
               return context 
        
           end

The main problem on log phase, if the batch policy or cache policy is in place
the context.service is not defined, so the Prometheus metrics raise an error
that the service.id cannot be found.

I debug all the code, and the issue is fixed if we add in this the following
code require('resty.ctx').apply(), if we add this the code is working
correctly.

APIcast/gateway/conf.d/apicast.conf

Lines 68 to 71 in 66168b7

    
           log_by_lua_block { 
        
             ngx.var.post_action_impact = ngx.var.request_time - ngx.var.original_request_time 
        
             require('apicast.executor'):log() 
        
           }

I think that it's safe enough, but I want to make sure that I do not introduce
a new issue. Also IMHO we should trigger a error message on
executor:shared_build_context that if the context does not exists maybe we
need to notice. (But I do not have the full overview of the project)

[0] https://github.com/3scale/APIcast/blob/master/t/apicast-policy-3scale-batcher.t#L403-L483

eloycoto · 2019-04-30T19:56:13Z

Added the patch on log phase in apicast.conf, now a different test is failing:

APIcast/t/apicast.t

Lines 621 to 631 in 66168b7

    
           === TEST 17: invalid service 
        
           The message is configurable and default status is 403. 
        
           --- http_config 
        
             lua_package_path "$TEST_NGINX_LUA_PATH"; 
        
           --- config 
        
             include $TEST_NGINX_APICAST_CONFIG; 
        
           --- request 
        
           GET /?user_key=value 
        
           --- error_code: 404 
        
           --- no_error_log 
        
           [error]

Should a request to invalid service reach log phase? Should be denied in the auth phase? @davidor any insight here?

davidor · 2019-05-02T08:53:11Z

@eloycoto The log phase will run even when the find_service policy cannot find a service for the request. In that case context.service is going to be nil. That explains why the test TEST 17: invalid service that you pasted above fails.

We'll need to take that case into account when updating the Prometheus metrics.

Regarding the 3scale batching problem, in that case, context.service should not be nil. We'll need to investigate if there's something wrong in that test or in the policy itself.

eloycoto · 2019-05-02T09:02:04Z

@davidor what about this commit? Is that correct?
190962e

CHANGELOG.md

gateway/src/apicast/policy/nginx_metrics/nginx_metrics.lua

porueesq

A couple of suggestions.

davidor · 2019-05-02T13:29:17Z

@davidor what about this commit? Is that correct?
190962e

@eloycoto and I have been investigating this. Adding require('resty.ctx').apply() to the log by lua block in gateway/conf.d/apicast.conf is not the solution. When starting APIcast with bin/apicast we saw that the context is shared correctly in all the phases, including the log one. There's only one test that fails and the reason is that it is the only one that defines its own rewrite_by_lua:

APIcast/t/apicast-policy-3scale-batcher.t

Line 453 in 66168b7

rewrite_by_lua_block {

Calling require('resty.ctx').apply() in that rewrite block fixes the issue.

t/prometheus-metrics.t

davidor · 2019-05-03T14:10:27Z

@eloycoto Sorry, I thought about this after approving the PR. Maybe we should use the service name (maybe system_name to guarantee uniqueness) or instead of the ID as the label of the metric.

I'm thinking about how this can be used to show data in something like Grafana. The ID will not give much information to the people operating APIcast. It's likely that they'll need to go to the admin portal to check which service that ID corresponds to. Maybe we can save them that step.

eloycoto · 2019-05-03T15:25:31Z

For the record, we discussed offline with @davidor:

If we use the ID, the number will never change.
There is a name that the users can update, but that means that the historical information will be lost.
There is a system_name variable that will work, but if you update the Name in the UI the system_name will not be updated, so it's like an old id.

We are going to keep the ID that it's the most reasonable one.

mikz · 2019-05-03T17:55:54Z

I think we discussed this earlier and kind of agreed using system name. The system name is immutable and should be used instead of an ID because it gives more context. It is immediately obvious in alerts and dashboards what service it is. Ids have zero context and it would be impossible for humans to decipher them without some lookup through 3scale API.

eloycoto · 2019-05-06T07:16:48Z

Changes to use system_name added in a new commit. Please review :-)

eloycoto · 2019-05-06T12:31:59Z

Hi,

Made some perfomance testing around the multiple services in the labels, it's
not a big change, I didn't notice any memory issue, is around 200req/sec slower
and adds a 2ms latency:

Config: https://gist.githubusercontent.com/eloycoto/37b23b7c74a21645e6a4c37f41197c53/raw/447eaf62434e50cd1e0253b622443f0d6de99d64/3scale-100-nodes.json
wrk setup:

function request()
   wrk.headers["Host"] = tostring(math.random(0,99))
   return wrk.format(nil, "/?user_key=foo")
end

Result with extended metrics:

root@ubuntu-s-4vcpu-8gb-ams3-01:~/APIcast# wrk  --connections 100 --threads 10 --duration 300 'http://172.18.0.3:8080/?user_key=foo' -s wrk-code.lua
Running 5m test @ http://172.18.0.3:8080/?user_key=foo
  10 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    27.49ms    7.93ms 141.59ms   72.14%
    Req/Sec   365.66     40.38     0.95k    75.36%
  1092416 requests in 5.00m, 234.36MB read
Requests/sec:   3640.26
Transfer/sec:    799.69KB

Upstream status metrics:

bash-4.2# curl http://localhost:9421/metrics -s | grep ^upstream_status
upstream_status{status="200",service="0"} 10772
upstream_status{status="200",service="1"} 10933
upstream_status{status="200",service="10"} 10410
upstream_status{status="200",service="11"} 11157
upstream_status{status="200",service="12"} 10465
upstream_status{status="200",service="13"} 11176
upstream_status{status="200",service="14"} 10485
upstream_status{status="200",service="15"} 10869
upstream_status{status="200",service="16"} 11551
upstream_status{status="200",service="17"} 11190
upstream_status{status="200",service="18"} 11373
upstream_status{status="200",service="19"} 10638
upstream_status{status="200",service="2"} 10568
upstream_status{status="200",service="20"} 10732
upstream_status{status="200",service="21"} 11071
upstream_status{status="200",service="22"} 11077
upstream_status{status="200",service="23"} 10724
upstream_status{status="200",service="24"} 10985
upstream_status{status="200",service="25"} 10688
upstream_status{status="200",service="26"} 10419
upstream_status{status="200",service="27"} 10913
upstream_status{status="200",service="28"} 11312
upstream_status{status="200",service="29"} 11288
upstream_status{status="200",service="3"} 11297
upstream_status{status="200",service="30"} 10395
upstream_status{status="200",service="31"} 10977
upstream_status{status="200",service="32"} 10810
upstream_status{status="200",service="33"} 11141
upstream_status{status="200",service="34"} 10920
upstream_status{status="200",service="35"} 11128
upstream_status{status="200",service="36"} 10456
upstream_status{status="200",service="37"} 10756
upstream_status{status="200",service="38"} 11159
upstream_status{status="200",service="39"} 10872
upstream_status{status="200",service="4"} 10773
upstream_status{status="200",service="40"} 10346
upstream_status{status="200",service="41"} 10553
upstream_status{status="200",service="42"} 11502
upstream_status{status="200",service="43"} 10724
upstream_status{status="200",service="44"} 10953
upstream_status{status="200",service="45"} 11275
upstream_status{status="200",service="46"} 10420
upstream_status{status="200",service="47"} 11296
upstream_status{status="200",service="48"} 11134
upstream_status{status="200",service="49"} 10559
upstream_status{status="200",service="5"} 10817
upstream_status{status="200",service="50"} 10998
upstream_status{status="200",service="51"} 10654
upstream_status{status="200",service="52"} 10794
upstream_status{status="200",service="53"} 10395
upstream_status{status="200",service="54"} 11651
upstream_status{status="200",service="55"} 10933
upstream_status{status="200",service="56"} 11145
upstream_status{status="200",service="57"} 10777
upstream_status{status="200",service="58"} 10043
upstream_status{status="200",service="59"} 11505
upstream_status{status="200",service="6"} 10780
upstream_status{status="200",service="60"} 10272
upstream_status{status="200",service="61"} 10658
upstream_status{status="200",service="62"} 10847
upstream_status{status="200",service="63"} 11367
upstream_status{status="200",service="64"} 11336
upstream_status{status="200",service="65"} 10528
upstream_status{status="200",service="66"} 10825
upstream_status{status="200",service="67"} 11279
upstream_status{status="200",service="68"} 11098
upstream_status{status="200",service="69"} 10708
upstream_status{status="200",service="7"} 11377
upstream_status{status="200",service="70"} 11280
upstream_status{status="200",service="71"} 11224
upstream_status{status="200",service="72"} 10878
upstream_status{status="200",service="73"} 10995
upstream_status{status="200",service="74"} 11010
upstream_status{status="200",service="75"} 10953
upstream_status{status="200",service="76"} 11471
upstream_status{status="200",service="77"} 10683
upstream_status{status="200",service="78"} 10551
upstream_status{status="200",service="79"} 10088
upstream_status{status="200",service="8"} 11209
upstream_status{status="200",service="80"} 11152
upstream_status{status="200",service="81"} 11418
upstream_status{status="200",service="82"} 10981
upstream_status{status="200",service="83"} 10945
upstream_status{status="200",service="84"} 10713
upstream_status{status="200",service="85"} 10293
upstream_status{status="200",service="86"} 11375
upstream_status{status="200",service="87"} 10748
upstream_status{status="200",service="88"} 10630
upstream_status{status="200",service="89"} 11301
upstream_status{status="200",service="9"} 11493
upstream_status{status="200",service="90"} 11578
upstream_status{status="200",service="91"} 10510
upstream_status{status="200",service="92"} 11049
upstream_status{status="200",service="93"} 11309
upstream_status{status="200",service="94"} 10555
upstream_status{status="200",service="95"} 10859
upstream_status{status="200",service="96"} 11179
upstream_status{status="200",service="97"} 10857
upstream_status{status="200",service="98"} 11392
upstream_status{status="200",service="99"} 10790
bash-4.2#

Without extended metrics:

wrk output:

root@ubuntu-s-4vcpu-8gb-ams3-01:~/APIcast# wrk  --connections 100 --threads 10 --duration 300 'http://172.18.0.3:8080/?user_key=foo' -s wrk-code.lua
Running 5m test @ http://172.18.0.3:8080/?user_key=foo
  10 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    25.81ms    7.61ms 118.59ms   72.33%
    Req/Sec   389.60     46.55     1.07k    79.60%
  1163878 requests in 5.00m, 249.69MB read
Requests/sec:   3878.50
Transfer/sec:    852.02KB
root@ubuntu-s-4vcpu-8gb-ams3-01:~/APIcast#

Metrics:

bash-4.2# curl http://localhost:9421/metrics -s | grep ^upstream_status
upstream_status{status="200",service=""} 1163966
bash-4.2#

mikz · 2019-05-07T06:38:11Z

@eloycoto I realised one shortcoming when using just system_name. It is no longer unique in multi-tenant environments. Would it be a big issue to have both system name and id? The granularity should be the same, so it shouldn't affect prometheus.

eloycoto · 2019-05-10T07:08:56Z

@mikz your comment makes total sense! I do not liked the system_name; a user can't change that and can be out of date quite quickly.

I would format as ${id}-${name}, with that users can use a custom legend based on ID or names, for legend format, they can use the following promQL query, and we allow users to have the correct names on that:

https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace

What do you think about this? Are you ok with that?

Regards

mikz · 2019-05-10T07:15:27Z

@eloycoto I actually thought about using two labels. One for id and one for system_name. So people do not have to bother using the label replace and just see the system_name as a hint for what the service is, but still be unique because of the ID. It is just additional metadata.
And for the ones that want, they can use the label_replace based on the id.

eloycoto · 2019-05-10T08:25:59Z

@mikz the problem if you add two labels, is that Prometheus matrix will be bigger, and the Prometheus performance will be affected, I'm happy with two labels if have different data, but having two labels for the same data, I'm a bit against

mikz · 2019-05-10T08:39:05Z

@eloycoto how it will be bigger? the number of combinations is the same if you make an identifier that contains both or if you have two identifiers and one is unique. Prometheus will not store anything in empty timelines.

This commits adds support to have additional information on some metrics[0], this commit enables users to know the service in the metric. Due to this can be a problematic for Prometheus[1], to enable this, a new ENV variable(`APICAST_EXTENDED_METRICS`) is defined. Users that have a few services can run the extended metrics without affecting Prometheus stability, if the APICast instance has thousands of services the recommendation is to have this option disabled. In case of the extended_metrics is disabled, all the service labels will use "all" value and Prometheus performance will not be affected. The new metrics list looks like this: ``` bash-4.2$ curl http://localhost:9421/metrics -s | grep service total_response_time_seconds_bucket{service="all",le="00.100"} 91 total_response_time_seconds_bucket{service="all",le="00.200"} 189 total_response_time_seconds_bucket{service="all",le="00.300"} 220 total_response_time_seconds_bucket{service="all",le="00.400"} 220 total_response_time_seconds_bucket{service="all",le="00.500"} 222 total_response_time_seconds_bucket{service="all",le="00.750"} 223 total_response_time_seconds_bucket{service="all",le="01.000"} 223 total_response_time_seconds_bucket{service="all",le="01.500"} 224 total_response_time_seconds_bucket{service="all",le="02.000"} 224 total_response_time_seconds_bucket{service="all",le="03.000"} 224 total_response_time_seconds_bucket{service="all",le="04.000"} 224 total_response_time_seconds_bucket{service="all",le="05.000"} 224 total_response_time_seconds_bucket{service="all",le="10.000"} 224 total_response_time_seconds_bucket{service="all",le="+Inf"} 224 total_response_time_seconds_count{service="all"} 224 total_response_time_seconds_sum{service="all"} 33.616 upstream_response_time_seconds_bucket{service="all",le="00.100"} 93 upstream_response_time_seconds_bucket{service="all",le="00.200"} 190 upstream_response_time_seconds_bucket{service="all",le="00.300"} 220 upstream_response_time_seconds_bucket{service="all",le="00.400"} 220 upstream_response_time_seconds_bucket{service="all",le="00.500"} 223 upstream_response_time_seconds_bucket{service="all",le="00.750"} 224 upstream_response_time_seconds_bucket{service="all",le="01.000"} 224 upstream_response_time_seconds_bucket{service="all",le="01.500"} 224 upstream_response_time_seconds_bucket{service="all",le="02.000"} 224 upstream_response_time_seconds_bucket{service="all",le="03.000"} 224 upstream_response_time_seconds_bucket{service="all",le="04.000"} 224 upstream_response_time_seconds_bucket{service="all",le="05.000"} 224 upstream_response_time_seconds_bucket{service="all",le="10.000"} 224 upstream_response_time_seconds_bucket{service="all",le="+Inf"} 224 upstream_response_time_seconds_count{service="all"} 224 upstream_response_time_seconds_sum{service="all"} 32.226 upstream_status{status="200",service="all"} 224 ``` [0] List of affected metrics: total_response_time_seconds upstream_response_time_seconds upstream_status [1] https://prometheus.io/docs/practices/naming/#labels Signed-off-by: Eloy Coto <eloy.coto@gmail.com>

This commit add the EXTENDED_Metrics integration test. Signed-off-by: Eloy Coto <eloy.coto@gmail.com>

Co-Authored-By: eloycoto <eloy.coto@gmail.com>

This commit change the behaviour of the metrics, now the metrics contains the service_id and the service_system_name labels. ``` bash-4.2$ curl http://localhost:9421/metrics -s | grep service total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="00.200"} 1 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="00.300"} 1 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="00.400"} 1 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="00.500"} 1 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="00.750"} 2 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="01.000"} 2 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="01.500"} 3 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="02.000"} 3 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="03.000"} 3 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="04.000"} 3 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="05.000"} 3 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="10.000"} 3 total_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="+Inf"} 3 total_response_time_seconds_count{service_id="2555417794444",service_system_name="api"} 3 total_response_time_seconds_sum{service_id="2555417794444",service_system_name="api"} 1.802 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="00.200"} 1 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="00.300"} 1 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="00.400"} 1 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="00.500"} 2 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="00.750"} 3 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="01.000"} 3 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="01.500"} 3 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="02.000"} 3 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="03.000"} 3 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="04.000"} 3 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="05.000"} 3 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="10.000"} 3 upstream_response_time_seconds_bucket{service_id="2555417794444",service_system_name="api",le="+Inf"} 3 upstream_response_time_seconds_count{service_id="2555417794444",service_system_name="api"} 3 upstream_response_time_seconds_sum{service_id="2555417794444",service_system_name="api"} 1.102 upstream_status{status="200",service_id="2555417794444",service_system_name="api"} 3 ``` Signed-off-by: Eloy Coto <eloy.coto@gmail.com>

eloycoto · 2019-05-13T07:48:18Z

@mikz latest changes now contains the service_id and the service_system_name labels per metric :-)

Regards.

mikz · 2019-05-13T08:08:12Z

@eloycoto amazing! 👍

davidor · 2019-05-13T08:23:38Z

@eloycoto Good job! 👍

…ween phases In this particular test, the context was not being shared properly between phases. The problem did not show up in this test, but it can cause problems in other tests that rely on having always the correct context in the log phases for example. Like it happened here: #1024 (comment)

eloycoto requested review from a team as code owners April 30, 2019 06:57

eloycoto force-pushed the THREESCALE-2150 branch from d1af596 to 5027c0b Compare April 30, 2019 06:59

mikz reviewed Apr 30, 2019

View reviewed changes

gateway/src/apicast/policy/nginx_metrics/nginx_metrics.lua Outdated Show resolved Hide resolved

mikz reviewed Apr 30, 2019

View reviewed changes

gateway/src/apicast/policy/nginx_metrics/nginx_metrics.lua Outdated Show resolved Hide resolved

mikz reviewed Apr 30, 2019

View reviewed changes

t/prometheus-metrics.t Outdated Show resolved Hide resolved

mikz reviewed Apr 30, 2019

View reviewed changes