Certain step values for a query return no results

[steveberryman]

We have this query: sum(increase(nodeproxy_http_response_time_histogram_seconds_bucket{app="349"}[10m])) by (le)

For some reason, when querying over a 24h period and the step is set to certain values (600 and 3600 are confirmed to work. 1800 and 2880 are confirmed to not work), no metrics are returned. The query works directly on prometheus or victoria metrics, just no through promxy.

Any thoughts? No errors in the logs (but possibly i can turn the logging level up?)

[jacksontj]

Very curious, I haven't seen that behavior before (and I've been using promxy for a while). I'd definitely be curious to see the debug or trace logging output from promxy. From my first throughts on it I can't think of why the step value would cause data to go missing, so I'm definitely interested :)

[steveberryman]

Here are some trace logs and curl requests/responses for you!
First request is step 3600. Works fine.
Second is step 2880. Returns no results in the response but seems to have results in the trace output.

$ curl 'http://localhost:8085/api/v1/query_range?start=1564328640&end=1564501440&step=3600&query=sum(increase(nodeproxy_http_response_time_histogram_seconds_bucket%7Bapp%3D"349"%7D%5B10m%5D))%20by%20(le)'
{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"le":"+Inf"},"values":[[1564328640,"900706"],[1564332240,"946727"],[1564335840,"976202"],[1564339440,"1038578"],[1564343040,"1044686"],[1564346640,"1131985"],[1564350240,"1167847"],[1564353840,"1150481"],[1564357440,"1243422"],[1564361040,"1317385"],[1564364640,"1423299"],[1564368240,"1430829"],[1564371840,"1417251"],[1564375440,"814139"],[1564379040,"594950"],[1564382640,"422275"],[1564386240,"327057"],[1564389840,"334167"],[1564393440,"337814"],[1564397040,"445100"],[1564400640,"449520"],[1564404240,"585377"],[1564407840,"761788"],[1564411440,"886062"],[1564415040,"943218"],[1564418640,"1072332"],[1564422240,"1054224"],[1564425840,"1053473"],[1564429440,"1156887"],[1564433040,"1194061"],[1564436640,"1174240"],[1564440240,"1139232"],[1564443840,"1230972"],[1564447440,"1421080"],[1564451040,"1531544"],[1564454640,"1549557"],[1564458240,"1372291"],[1564461840,"1000169"],[1564465440,"732119"],[1564469040,"535268"],[1564472640,"438176"],[1564476240,"399494"],[1564479840,"375091"],[1564483440,"487035"],[1564487040,"583542"],[1564490640,"698682"],[1564494240,"876755"],[1564497840,"882128"],[1564501440,"968841"]]},{"metric":{"le":"0.005"},"values":[[1564328640,"12500"],[1564332240,"13383"],[1564335840,"13379"],[1564339440,"12401"],[1564343040,"12049"],[1564346640,"11120"],[1564350240,"13051"],[1564353840,"12816"],[1564357440,"12161"],[1564361040,"11027"],[1564364640,"17777"],[1564368240,"16825"],[1564371840,"15552"],[1564375440,"17230"],[1564379040,"13810"],[1564382640,"9074"],[1564386240,"6849"],[1564389840,"5176"],[1564393440,"5155"],[1564397040,"5750"],[1564400640,"7325"],[1564404240,"11877"],[1564407840,"12024"],[1564411440,"14407"],[1564415040,"12542"],[1564418640,"12871"],[1564422240,"12772"],[1564425840,"11971"],[1564429440,"11587"],[1564433040,"11104"],[1564436640,"9706"],[1564440240,"11877"],[1564443840,"11446"],[1564447440,"8783"],[1564451040,"11136"],[1564454640,"10898"],[1564458240,"11466"],[1564461840,"12203"],[1564465440,"10432"],[1564469040,"7108"],[1564472640,"4628"],[1564476240,"4362"],[1564479840,"4647"],[1564483440,"5903"],[1564487040,"7594"],[1564490640,"9677"],[1564494240,"11779"],[1564497840,"11237"],[1564501440,"11392"]]},{"metric":{"le":"0.01"},"values":[[1564328640,"36288"],[1564332240,"40119"],[1564335840,"39745"],[1564339440,"45535"],[1564343040,"47647"],[1564346640,"41953"],[1564350240,"51241"],[1564353840,"49375"],[1564357440,"45946"],[1564361040,"48590"],[1564364640,"58940"],[1564368240,"58080"],[1564371840,"51630"],[1564375440,"34947"],[1564379040,"23966"],[1564382640,"15311"],[1564386240,"11346"],[1564389840,"11635"],[1564393440,"12400"],[1564397040,"17985"],[1564400640,"16855"],[1564404240,"24593"],[1564407840,"28634"],[1564411440,"38829"],[1564415040,"39355"],[1564418640,"45833"],[1564422240,"44300"],[1564425840,"42952"],[1564429440,"40855"],[1564433040,"38345"],[1564436640,"34489"],[1564440240,"37172"],[1564443840,"43045"],[1564447440,"51528"],[1564451040,"46247"],[1564454640,"46511"],[1564458240,"46748"],[1564461840,"29636"],[1564465440,"20503"],[1564469040,"12840"],[1564472640,"8409"],[1564476240,"8479"],[1564479840,"9314"],[1564483440,"14298"],[1564487040,"15384"],[1564490640,"22000"],[1564494240,"32048"],[1564497840,"29617"],[1564501440,"32891"]]},{"metric":{"le":"0.1"},"values":[[1564328640,"355289"],[1564332240,"383441"],[1564335840,"388465"],[1564339440,"423242"],[1564343040,"422938"],[1564346640,"429082"],[1564350240,"469354"],[1564353840,"438155"],[1564357440,"469358"],[1564361040,"472820"],[1564364640,"505228"],[1564368240,"493116"],[1564371840,"438669"],[1564375440,"297525"],[1564379040,"223981"],[1564382640,"143263"],[1564386240,"103174"],[1564389840,"134694"],[1564393440,"152256"],[1564397040,"208486"],[1564400640,"188251"],[1564404240,"241661"],[1564407840,"300128"],[1564411440,"360863"],[1564415040,"366177"],[1564418640,"418392"],[1564422240,"405699"],[1564425840,"395592"],[1564429440,"417416"],[1564433040,"395369"],[1564436640,"359862"],[1564440240,"398021"],[1564443840,"427201"],[1564447440,"461214"],[1564451040,"372600"],[1564454640,"455924"],[1564458240,"474661"],[1564461840,"334003"],[1564465440,"212585"],[1564469040,"154851"],[1564472640,"118326"],[1564476240,"109314"],[1564479840,"107980"],[1564483440,"168761"],[1564487040,"183342"],[1564490640,"239781"],[1564494240,"321900"],[1564497840,"312250"],[1564501440,"347161"]]},{"metric":{"le":"0.25"},"values":[[1564328640,"674991"],[1564332240,"715242"],[1564335840,"727093"],[1564339440,"772432"],[1564343040,"779900"],[1564346640,"829819"],[1564350240,"872000"],[1564353840,"823609"],[1564357440,"907161"],[1564361040,"927054"],[1564364640,"996855"],[1564368240,"992356"],[1564371840,"934826"],[1564375440,"600717"],[1564379040,"448432"],[1564382640,"306148"],[1564386240,"236506"],[1564389840,"247719"],[1564393440,"264540"],[1564397040,"345693"],[1564400640,"333292"],[1564404240,"446455"],[1564407840,"566595"],[1564411440,"671580"],[1564415040,"687488"],[1564418640,"792909"],[1564422240,"764988"],[1564425840,"756140"],[1564429440,"818998"],[1564433040,"811607"],[1564436640,"778140"],[1564440240,"801509"],[1564443840,"862641"],[1564447440,"919925"],[1564451040,"799632"],[1564454640,"985822"],[1564458240,"954168"],[1564461840,"703500"],[1564465440,"501703"],[1564469040,"373938"],[1564472640,"301645"],[1564476240,"275743"],[1564479840,"258420"],[1564483440,"353818"],[1564487040,"402009"],[1564490640,"486328"],[1564494240,"623258"],[1564497840,"613427"],[1564501440,"673260"]]},{"metric":{"le":"0.5"},"values":[[1564328640,"853901"],[1564332240,"897624"],[1564335840,"922860"],[1564339440,"976140"],[1564343040,"983774"],[1564346640,"1060893"],[1564350240,"1094715"],[1564353840,"1060957"],[1564357440,"1156453"],[1564361040,"1205441"],[1564364640,"1297490"],[1564368240,"1298873"],[1564371840,"1264817"],[1564375440,"772751"],[1564379040,"567104"],[1564382640,"399275"],[1564386240,"309301"],[1564389840,"316627"],[1564393440,"322818"],[1564397040,"423047"],[1564400640,"426182"],[1564404240,"560958"],[1564407840,"723898"],[1564411440,"843501"],[1564415040,"886703"],[1564418640,"1010937"],[1564422240,"987707"],[1564425840,"983459"],[1564429440,"1081606"],[1564433040,"1097579"],[1564436640,"1062933"],[1564440240,"1058882"],[1564443840,"1129055"],[1564447440,"1249042"],[1564451040,"1157350"],[1564454640,"1354673"],[1564458240,"1262890"],[1564461840,"936571"],[1564465440,"689480"],[1564469040,"505972"],[1564472640,"415012"],[1564476240,"379220"],[1564479840,"357950"],[1564483440,"468511"],[1564487040,"552452"],[1564490640,"659037"],[1564494240,"827036"],[1564497840,"819416"],[1564501440,"893553"]]},{"metric":{"le":"1"},"values":[[1564328640,"898772"],[1564332240,"944442"],[1564335840,"973524"],[1564339440,"1034525"],[1564343040,"1040370"],[1564346640,"1127488"],[1564350240,"1162504"],[1564353840,"1142690"],[1564357440,"1234397"],[1564361040,"1301073"],[1564364640,"1403056"],[1564368240,"1405641"],[1564371840,"1384645"],[1564375440,"812367"],[1564379040,"594166"],[1564382640,"420712"],[1564386240,"326227"],[1564389840,"333520"],[1564393440,"336857"],[1564397040,"444209"],[1564400640,"448651"],[1564404240,"584460"],[1564407840,"759771"],[1564411440,"883985"],[1564415040,"939708"],[1564418640,"1068421"],[1564422240,"1050611"],[1564425840,"1047735"],[1564429440,"1150901"],[1564433040,"1181957"],[1564436640,"1157829"],[1564440240,"1131999"],[1564443840,"1214661"],[1564447440,"1384410"],[1564451040,"1423306"],[1564454640,"1507853"],[1564458240,"1355737"],[1564461840,"994929"],[1564465440,"729686"],[1564469040,"533853"],[1564472640,"437014"],[1564476240,"398342"],[1564479840,"374066"],[1564483440,"486085"],[1564487040,"582140"],[1564490640,"696679"],[1564494240,"874301"],[1564497840,"879290"],[1564501440,"963904"]]},{"metric":{"le":"10"},"values":[[1564328640,"900701"],[1564332240,"946727"],[1564335840,"976194"],[1564339440,"1038566"],[1564343040,"1044686"],[1564346640,"1131985"],[1564350240,"1167843"],[1564353840,"1150479"],[1564357440,"1243417"],[1564361040,"1317384"],[1564364640,"1423297"],[1564368240,"1430755"],[1564371840,"1417215"],[1564375440,"814138"],[1564379040,"594949"],[1564382640,"421511"],[1564386240,"327055"],[1564389840,"334166"],[1564393440,"337814"],[1564397040,"445100"],[1564400640,"449520"],[1564404240,"585377"],[1564407840,"761785"],[1564411440,"886062"],[1564415040,"943217"],[1564418640,"1072329"],[1564422240,"1054224"],[1564425840,"1053472"],[1564429440,"1156887"],[1564433040,"1194043"],[1564436640,"1174238"],[1564440240,"1138799"],[1564443840,"1230970"],[1564447440,"1421023"],[1564451040,"1531312"],[1564454640,"1549017"],[1564458240,"1371788"],[1564461840,"1000168"],[1564465440,"732118"],[1564469040,"535261"],[1564472640,"438176"],[1564476240,"399493"],[1564479840,"375091"],[1564483440,"487035"],[1564487040,"583541"],[1564490640,"698682"],[1564494240,"876755"],[1564497840,"882128"],[1564501440,"968841"]]},{"metric":{"le":"120"},"values":[[1564328640,"900706"],[1564332240,"946727"],[1564335840,"976202"],[1564339440,"1038578"],[1564343040,"1044686"],[1564346640,"1131985"],[1564350240,"1167847"],[1564353840,"1150481"],[1564357440,"1243422"],[1564361040,"1317385"],[1564364640,"1423299"],[1564368240,"1430829"],[1564371840,"1417251"],[1564375440,"814139"],[1564379040,"594950"],[1564382640,"422275"],[1564386240,"327057"],[1564389840,"334167"],[1564393440,"337814"],[1564397040,"445100"],[1564400640,"449520"],[1564404240,"585377"],[1564407840,"761788"],[1564411440,"886062"],[1564415040,"943218"],[1564418640,"1072332"],[1564422240,"1054224"],[1564425840,"1053473"],[1564429440,"1156887"],[1564433040,"1194061"],[1564436640,"1174240"],[1564440240,"1139232"],[1564443840,"1230972"],[1564447440,"1421080"],[1564451040,"1531544"],[1564454640,"1549557"],[1564458240,"1372291"],[1564461840,"1000169"],[1564465440,"732119"],[1564469040,"535268"],[1564472640,"438176"],[1564476240,"399494"],[1564479840,"375091"],[1564483440,"487035"],[1564487040,"583542"],[1564490640,"698682"],[1564494240,"876755"],[1564497840,"882128"],[1564501440,"968841"]]},{"metric":{"le":"5"},"values":[[1564328640,"900701"],[1564332240,"946721"],[1564335840,"976185"],[1564339440,"1038524"],[1564343040,"1044682"],[1564346640,"1131981"],[1564350240,"1167837"],[1564353840,"1150474"],[1564357440,"1243181"],[1564361040,"1317290"],[1564364640,"1423041"],[1564368240,"1428107"],[1564371840,"1415555"],[1564375440,"814132"],[1564379040,"594945"],[1564382640,"421504"],[1564386240,"327053"],[1564389840,"334165"],[1564393440,"337814"],[1564397040,"445096"],[1564400640,"449520"],[1564404240,"585373"],[1564407840,"761784"],[1564411440,"886061"],[1564415040,"943217"],[1564418640,"1072173"],[1564422240,"1054177"],[1564425840,"1053470"],[1564429440,"1156877"],[1564433040,"1193315"],[1564436640,"1173760"],[1564440240,"1138241"],[1564443840,"1230710"],[1564447440,"1419569"],[1564451040,"1528305"],[1564454640,"1547141"],[1564458240,"1370588"],[1564461840,"1000166"],[1564465440,"732118"],[1564469040,"535223"],[1564472640,"438175"],[1564476240,"399489"],[1564479840,"375091"],[1564483440,"487035"],[1564487040,"583538"],[1564490640,"698679"],[1564494240,"876754"],[1564497840,"882128"],[1564501440,"968811"]]},{"metric":{"le":"60"},"values":[[1564328640,"900706"],[1564332240,"946727"],[1564335840,"976202"],[1564339440,"1038578"],[1564343040,"1044686"],[1564346640,"1131985"],[1564350240,"1167847"],[1564353840,"1150481"],[1564357440,"1243422"],[1564361040,"1317384"],[1564364640,"1423299"],[1564368240,"1430829"],[1564371840,"1417250"],[1564375440,"814139"],[1564379040,"594950"],[1564382640,"422275"],[1564386240,"327057"],[1564389840,"334167"],[1564393440,"337814"],[1564397040,"445100"],[1564400640,"449520"],[1564404240,"585377"],[1564407840,"761788"],[1564411440,"886062"],[1564415040,"943218"],[1564418640,"1072332"],[1564422240,"1054224"],[1564425840,"1053473"],[1564429440,"1156887"],[1564433040,"1194061"],[1564436640,"1174240"],[1564440240,"1139232"],[1564443840,"1230972"],[1564447440,"1421080"],[1564451040,"1531544"],[1564454640,"1549557"],[1564458240,"1372291"],[1564461840,"1000169"],[1564465440,"732119"],[1564469040,"535268"],[1564472640,"438176"],[1564476240,"399494"],[1564479840,"375091"],[1564483440,"487035"],[1564487040,"583542"],[1564490640,"698682"],[1564494240,"876755"],[1564497840,"882128"],[1564501440,"968841"]]}]}}

DEBU[2019-08-01T11:04:05Z] AggregateExpr sum by(le) (increase(nodeproxy_http_response_time_histogram_seconds_bucket{app="349"}[10m]))
DEBU[2019-08-01T11:04:05Z] http://prometheus-cache-do-nyc1-3abf.vpn.flyio.net:8428  api=QueryRange query="sum by(le) (increase(nodeproxy_http_response_time_histogram_seconds_bucket{app=\"349\"}[10m]))" r="{2019-07-28 15:39:00 +0000 UTC 2019-07-30 15:44:00 +0000 UTC 1h0m0s}"
DEBU[2019-08-01T11:04:05Z] http://prometheus-cache-do-nyc1-f31d.vpn.flyio.net:8428  api=QueryRange query="sum by(le) (increase(nodeproxy_http_response_time_histogram_seconds_bucket{app=\"349\"}[10m]))" r="{2019-07-28 15:39:00 +0000 UTC 2019-07-30 15:44:00 +0000 UTC 1h0m0s}"
TRAC[2019-08-01T11:04:06Z] http://prometheus-cache-do-nyc1-3abf.vpn.flyio.net:8428  api=QueryRange error="<nil>" query="sum by(le) (increase(nodeproxy_http_response_time_histogram_seconds_bucket{app=\"349\"}[10m]))" r="{2019-07-28 15:39:00 +0000 UTC 2019-07-30 15:44:00 +0000 UTC 1h0m0s}" took=255.926759ms value="{le=\"+Inf\", sg=\"victoriametrics\"} =>\n900706 @[1564328340]\n946727 @[1564331940]\n976202 @[1564335540]\n1038578 @[1564339140]\n1044686 @[1564342740]\n1131985 @[1564346340]\n1167847 @[1564349940]\n1150554 @[1564353540]\n1243422 @[1564357140]\n1317385 @[1564360740]\n1423299 @[1564364340]\n1430846 @[1564367940]\n1417251 @[1564371540]\n814194 @[1564375140]\n594950 @[1564378740]\n422275 @[1564382340]\n327057 @[1564385940]\n334221 @[1564389540]\n337814 @[1564393140]\n445100 @[1564396740]\n449625 @[1564400340]\n585377 @[1564403940]\n761788 @[1564407540]\n886062 @[1564411140]\n943438 @[1564414740]\n1072332 @[1564418340]\n1054224 @[1564421940]\n1053473 @[1564425540]\n1156887 @[1564429140]\n1194061 @[1564432740]\n1174240 @[1564436340]\n1139232 @[1564439940]\n1230972 @[1564443540]\n1421080 @[1564447140]\n1531544 @[1564450740]\n1549557 @[1564454340]\n1372291 @[1564457940]\n1000169 @[1564461540]\n732119 @[1564465140]\n535268 @[1564468740]\n438176 @[1564472340]\n399494 @[1564475940]\n375091 @[1564479540]\n487035 @[1564483140]\n583542 @[1564486740]\n698682 @[1564490340]\n876846 @[1564493940]\n882128 @[1564497540]\n968841 @[1564501140]\n{le=\"0.005\", sg=\"victoriametrics\"} =>\n12500 @[1564328340]\n13383 @[1564331940]\n13379 @[1564335540]\n12401 @[1564339140]\n12049 @[1564342740]\n11120 @[1564346340]\n13051 @[1564349940]\n12816 @[1564353540]\n12161 @[1564357140]\n11029 @[1564360740]\n17777 @[1564364340]\n16825 @[1564367940]\n15552 @[1564371540]\n17230 @[1564375140]\n13810 @[1564378740]\n9074 @[1564382340]\n6849 @[1564385940]\n5176 @[1564389540]\n5155 @[1564393140]\n5750 @[1564396740]\n7325 @[1564400340]\n11877 @[1564403940]\n12024 @[1564407540]\n14407 @[1564411140]\n12542 @[1564414740]\n12871 @[1564418340]\n12772 @[1564421940]\n11971 @[1564425540]\n11587 @[1564429140]\n11104 @[1564432740]\n9706 @[1564436340]\n11877 @[1564439940]\n11446 @[1564443540]\n8783 @[1564447140]\n11136 @[1564450740]\n10898 @[1564454340]\n11466 @[1564457940]\n12203 @[1564461540]\n10432 @[1564465140]\n7108 @[1564468740]\n4628 @[1564472340]\n4362 @[1564475940]\n4647 @[1564479540]\n5903 @[1564483140]\n7594 @[1564486740]\n9677 @[1564490340]\n11779 @[1564493940]\n11237 @[1564497540]\n11392 @[1564501140]\n{le=\"0.01\", sg=\"victoriametrics\"} =>\n36288 @[1564328340]\n40119 @[1564331940]\n39745 @[1564335540]\n45535 @[1564339140]\n47647 @[1564342740]\n41953 @[1564346340]\n51241 @[1564349940]\n49375 @[1564353540]\n45946 @[1564357140]\n48590 @[1564360740]\n58940 @[1564364340]\n58080 @[1564367940]\n51630 @[1564371540]\n34947 @[1564375140]\n23966 @[1564378740]\n15311 @[1564382340]\n11346 @[1564385940]\n11635 @[1564389540]\n12400 @[1564393140]\n17985 @[1564396740]\n16855 @[1564400340]\n24593 @[1564403940]\n28634 @[1564407540]\n38829 @[1564411140]\n39355 @[1564414740]\n45833 @[1564418340]\n44300 @[1564421940]\n42952 @[1564425540]\n40855 @[1564429140]\n38345 @[1564432740]\n34489 @[1564436340]\n37172 @[1564439940]\n43045 @[1564443540]\n51528 @[1564447140]\n46247 @[1564450740]\n46511 @[1564454340]\n46748 @[1564457940]\n29636 @[1564461540]\n20503 @[1564465140]\n12840 @[1564468740]\n8409 @[1564472340]\n8479 @[1564475940]\n9314 @[1564479540]\n14298 @[1564483140]\n15384 @[1564486740]\n22000 @[1564490340]\n32048 @[1564493940]\n29617 @[1564497540]\n32891 @[1564501140]\n{le=\"0.1\", sg=\"victoriametrics\"} =>\n355289 @[1564328340]\n383441 @[1564331940]\n388465 @[1564335540]\n423242 @[1564339140]\n422938 @[1564342740]\n429082 @[1564346340]\n469354 @[1564349940]\n438160 @[1564353540]\n469358 @[1564357140]\n472820 @[1564360740]\n505228 @[1564364340]\n493116 @[1564367940]\n438669 @[1564371540]\n297527 @[1564375140]\n223981 @[1564378740]\n143263 @[1564382340]\n103174 @[1564385940]\n134700 @[1564389540]\n152256 @[1564393140]\n208486 @[1564396740]\n188256 @[1564400340]\n241661 @[1564403940]\n300128 @[1564407540]\n360863 @[1564411140]\n366178 @[1564414740]\n418392 @[1564418340]\n405699 @[1564421940]\n395592 @[1564425540]\n417416 @[1564429140]\n395369 @[1564432740]\n359862 @[1564436340]\n398021 @[1564439940]\n427201 @[1564443540]\n461214 @[1564447140]\n372600 @[1564450740]\n455924 @[1564454340]\n474661 @[1564457940]\n334003 @[1564461540]\n212585 @[1564465140]\n154851 @[1564468740]\n118326 @[1564472340]\n109314 @[1564475940]\n107980 @[1564479540]\n168761 @[1564483140]\n183342 @[1564486740]\n239781 @[1564490340]\n321901 @[1564493940]\n312250 @[1564497540]\n347161 @[1564501140]\n{le=\"0.25\", sg=\"victoriametrics\"} =>\n674991 @[1564328340]\n715242 @[1564331940]\n727093 @[1564335540]\n772432 @[1564339140]\n779900 @[1564342740]\n829819 @[1564346340]\n872000 @[1564349940]\n823626 @[1564353540]\n907161 @[1564357140]\n927054 @[1564360740]\n996855 @[1564364340]\n992361 @[1564367940]\n934826 @[1564371540]\n600765 @[1564375140]\n448432 @[1564378740]\n306148 @[1564382340]\n236506 @[1564385940]\n247753 @[1564389540]\n264540 @[1564393140]\n345693 @[1564396740]\n333340 @[1564400340]\n446455 @[1564403940]\n566595 @[1564407540]\n671580 @[1564411140]\n687558 @[1564414740]\n792909 @[1564418340]\n764988 @[1564421940]\n756140 @[1564425540]\n818998 @[1564429140]\n811607 @[1564432740]\n778140 @[1564436340]\n801509 @[1564439940]\n862641 @[1564443540]\n919925 @[1564447140]\n799632 @[1564450740]\n985822 @[1564454340]\n954168 @[1564457940]\n703500 @[1564461540]\n501703 @[1564465140]\n373938 @[1564468740]\n301645 @[1564472340]\n275743 @[1564475940]\n258420 @[1564479540]\n353818 @[1564483140]\n402009 @[1564486740]\n486328 @[1564490340]\n623288 @[1564493940]\n613427 @[1564497540]\n673260 @[1564501140]\n{le=\"0.5\", sg=\"victoriametrics\"} =>\n853901 @[1564328340]\n897624 @[1564331940]\n922860 @[1564335540]\n976140 @[1564339140]\n983774 @[1564342740]\n1060893 @[1564346340]\n1094715 @[1564349940]\n1061006 @[1564353540]\n1156453 @[1564357140]\n1205441 @[1564360740]\n1297490 @[1564364340]\n1298887 @[1564367940]\n1264817 @[1564371540]\n772806 @[1564375140]\n567104 @[1564378740]\n399275 @[1564382340]\n309301 @[1564385940]\n316681 @[1564389540]\n322818 @[1564393140]\n423047 @[1564396740]\n426267 @[1564400340]\n560958 @[1564403940]\n723898 @[1564407540]\n843501 @[1564411140]\n886888 @[1564414740]\n1010937 @[1564418340]\n987707 @[1564421940]\n983459 @[1564425540]\n1081606 @[1564429140]\n1097579 @[1564432740]\n1062933 @[1564436340]\n1058882 @[1564439940]\n1129055 @[1564443540]\n1249042 @[1564447140]\n1157350 @[1564450740]\n1354673 @[1564454340]\n1262890 @[1564457940]\n936571 @[1564461540]\n689480 @[1564465140]\n505972 @[1564468740]\n415012 @[1564472340]\n379220 @[1564475940]\n357950 @[1564479540]\n468511 @[1564483140]\n552452 @[1564486740]\n659037 @[1564490340]\n827102 @[1564493940]\n819416 @[1564497540]\n893553 @[1564501140]\n{le=\"1\", sg=\"victoriametrics\"} =>\n898772 @[1564328340]\n944442 @[1564331940]\n973524 @[1564335540]\n1034525 @[1564339140]\n1040370 @[1564342740]\n1127488 @[1564346340]\n1162504 @[1564349940]\n1142762 @[1564353540]\n1234397 @[1564357140]\n1301073 @[1564360740]\n1403056 @[1564364340]\n1405658 @[1564367940]\n1384645 @[1564371540]\n812422 @[1564375140]\n594166 @[1564378740]\n420712 @[1564382340]\n326227 @[1564385940]\n333574 @[1564389540]\n336857 @[1564393140]\n444209 @[1564396740]\n448756 @[1564400340]\n584460 @[1564403940]\n759771 @[1564407540]\n883985 @[1564411140]\n939928 @[1564414740]\n1068421 @[1564418340]\n1050611 @[1564421940]\n1047735 @[1564425540]\n1150901 @[1564429140]\n1181957 @[1564432740]\n1157829 @[1564436340]\n1131999 @[1564439940]\n1214661 @[1564443540]\n1384410 @[1564447140]\n1423306 @[1564450740]\n1507853 @[1564454340]\n1355737 @[1564457940]\n994929 @[1564461540]\n729686 @[1564465140]\n533853 @[1564468740]\n437014 @[1564472340]\n398342 @[1564475940]\n374066 @[1564479540]\n486085 @[1564483140]\n582140 @[1564486740]\n696679 @[1564490340]\n874385 @[1564493940]\n879290 @[1564497540]\n963904 @[1564501140]\n{le=\"10\", sg=\"victoriametrics\"} =>\n900701 @[1564328340]\n946727 @[1564331940]\n976194 @[1564335540]\n1038566 @[1564339140]\n1044686 @[1564342740]\n1131985 @[1564346340]\n1167843 @[1564349940]\n1150552 @[1564353540]\n1243417 @[1564357140]\n1317384 @[1564360740]\n1423297 @[1564364340]\n1430772 @[1564367940]\n1417215 @[1564371540]\n814193 @[1564375140]\n594949 @[1564378740]\n421511 @[1564382340]\n327055 @[1564385940]\n334220 @[1564389540]\n337814 @[1564393140]\n445100 @[1564396740]\n449625 @[1564400340]\n585377 @[1564403940]\n761785 @[1564407540]\n886062 @[1564411140]\n943437 @[1564414740]\n1072329 @[1564418340]\n1054224 @[1564421940]\n1053472 @[1564425540]\n1156887 @[1564429140]\n1194043 @[1564432740]\n1174238 @[1564436340]\n1138799 @[1564439940]\n1230970 @[1564443540]\n1421023 @[1564447140]\n1531312 @[1564450740]\n1549017 @[1564454340]\n1371788 @[1564457940]\n1000168 @[1564461540]\n732118 @[1564465140]\n535261 @[1564468740]\n438176 @[1564472340]\n399493 @[1564475940]\n375091 @[1564479540]\n487035 @[1564483140]\n583541 @[1564486740]\n698682 @[1564490340]\n876846 @[1564493940]\n882128 @[1564497540]\n968841 @[1564501140]\n{le=\"120\", sg=\"victoriametrics\"} =>\n900706 @[1564328340]\n946727 @[1564331940]\n976202 @[1564335540]\n1038578 @[1564339140]\n1044686 @[1564342740]\n1131985 @[1564346340]\n1167847 @[1564349940]\n1150554 @[1564353540]\n1243422 @[1564357140]\n1317385 @[1564360740]\n1423299 @[1564364340]\n1430846 @[1564367940]\n1417251 @[1564371540]\n814194 @[1564375140]\n594950 @[1564378740]\n422275 @[1564382340]\n327057 @[1564385940]\n334221 @[1564389540]\n337814 @[1564393140]\n445100 @[1564396740]\n449625 @[1564400340]\n585377 @[1564403940]\n761788 @[1564407540]\n886062 @[1564411140]\n943438 @[1564414740]\n1072332 @[1564418340]\n1054224 @[1564421940]\n1053473 @[1564425540]\n1156887 @[1564429140]\n1194061 @[1564432740]\n1174240 @[1564436340]\n1139232 @[1564439940]\n1230972 @[1564443540]\n1421080 @[1564447140]\n1531544 @[1564450740]\n1549557 @[1564454340]\n1372291 @[1564457940]\n1000169 @[1564461540]\n732119 @[1564465140]\n535268 @[1564468740]\n438176 @[1564472340]\n399494 @[1564475940]\n375091 @[1564479540]\n487035 @[1564483140]\n583542 @[1564486740]\n698682 @[1564490340]\n876846 @[1564493940]\n882128 @[1564497540]\n968841 @[1564501140]\n{le=\"5\", sg=\"victoriametrics\"} =>\n900701 @[1564328340]\n946721 @[1564331940]\n976185 @[1564335540]\n1038524 @[1564339140]\n1044682 @[1564342740]\n1131981 @[1564346340]\n1167837 @[1564349940]\n1150547 @[1564353540]\n1243181 @[1564357140]\n1317290 @[1564360740]\n1423041 @[1564364340]\n1428124 @[1564367940]\n1415555 @[1564371540]\n814187 @[1564375140]\n594945 @[1564378740]\n421504 @[1564382340]\n327053 @[1564385940]\n334219 @[1564389540]\n337814 @[1564393140]\n445096 @[1564396740]\n449625 @[1564400340]\n585373 @[1564403940]\n761784 @[1564407540]\n886061 @[1564411140]\n943437 @[1564414740]\n1072173 @[1564418340]\n1054177 @[1564421940]\n1053470 @[1564425540]\n1156877 @[1564429140]\n1193315 @[1564432740]\n1173760 @[1564436340]\n1138241 @[1564439940]\n1230710 @[1564443540]\n1419569 @[1564447140]\n1528305 @[1564450740]\n1547141 @[1564454340]\n1370588 @[1564457940]\n1000166 @[1564461540]\n732118 @[1564465140]\n535223 @[1564468740]\n438175 @[1564472340]\n399489 @[1564475940]\n375091 @[1564479540]\n487035 @[1564483140]\n583538 @[1564486740]\n698679 @[1564490340]\n876845 @[1564493940]\n882128 @[1564497540]\n968811 @[1564501140]\n{le=\"60\", sg=\"victoriametrics\"} =>\n900706 @[1564328340]\n946727 @[1564331940]\n976202 @[1564335540]\n1038578 @[1564339140]\n1044686 @[1564342740]\n1131985 @[1564346340]\n1167847 @[1564349940]\n1150554 @[1564353540]\n1243422 @[1564357140]\n1317384 @[1564360740]\n1423299 @[1564364340]\n1430846 @[1564367940]\n1417250 @[1564371540]\n814194 @[1564375140]\n594950 @[1564378740]\n422275 @[1564382340]\n327057 @[1564385940]\n334221 @[1564389540]\n337814 @[1564393140]\n445100 @[1564396740]\n449625 @[1564400340]\n585377 @[1564403940]\n761788 @[1564407540]\n886062 @[1564411140]\n943438 @[1564414740]\n1072332 @[1564418340]\n1054224 @[1564421940]\n1053473 @[1564425540]\n1156887 @[1564429140]\n1194061 @[1564432740]\n1174240 @[1564436340]\n1139232 @[1564439940]\n1230972 @[1564443540]\n1421080 @[1564447140]\n1531544 @[1564450740]\n1549557 @[1564454340]\n1372291 @[1564457940]\n1000169 @[1564461540]\n732119 @[1564465140]\n535268 @[1564468740]\n438176 @[1564472340]\n399494 @[1564475940]\n375091 @[1564479540]\n487035 @[1564483140]\n583542 @[1564486740]\n698682 @[1564490340]\n876846 @[1564493940]\n882128 @[1564497540]\n968841 @[1564501140]" warnings="[]"
TRAC[2019-08-01T11:04:06Z] http://prometheus-cache-do-nyc1-f31d.vpn.flyio.net:8428  api=QueryRange error="<nil>" query="sum by(le) (increase(nodeproxy_http_response_time_histogram_seconds_bucket{app=\"349\"}[10m]))" r="{2019-07-28 15:39:00 +0000 UTC 2019-07-30 15:44:00 +0000 UTC 1h0m0s}" took=369.333721ms value="{le=\"+Inf\", sg=\"victoriametrics\"} =>\n900706 @[1564328340]\n946727 @[1564331940]\n976202 @[1564335540]\n1038578 @[1564339140]\n1044686 @[1564342740]\n1131985 @[1564346340]\n1167847 @[1564349940]\n1150481 @[1564353540]\n1243422 @[1564357140]\n1317385 @[1564360740]\n1423299 @[1564364340]\n1430829 @[1564367940]\n1417251 @[1564371540]\n814139 @[1564375140]\n594950 @[1564378740]\n422275 @[1564382340]\n327057 @[1564385940]\n334167 @[1564389540]\n337814 @[1564393140]\n445100 @[1564396740]\n449520 @[1564400340]\n585377 @[1564403940]\n761788 @[1564407540]\n886062 @[1564411140]\n943218 @[1564414740]\n1072332 @[1564418340]\n1054224 @[1564421940]\n1053473 @[1564425540]\n1156887 @[1564429140]\n1194061 @[1564432740]\n1174240 @[1564436340]\n1139232 @[1564439940]\n1230972 @[1564443540]\n1421080 @[1564447140]\n1531544 @[1564450740]\n1549557 @[1564454340]\n1372291 @[1564457940]\n1000169 @[1564461540]\n732119 @[1564465140]\n535268 @[1564468740]\n438176 @[1564472340]\n399494 @[1564475940]\n375091 @[1564479540]\n487035 @[1564483140]\n583542 @[1564486740]\n698682 @[1564490340]\n876755 @[1564493940]\n882128 @[1564497540]\n968841 @[1564501140]\n{le=\"0.005\", sg=\"victoriametrics\"} =>\n12500 @[1564328340]\n13383 @[1564331940]\n13379 @[1564335540]\n12401 @[1564339140]\n12049 @[1564342740]\n11120 @[1564346340]\n13051 @[1564349940]\n12816 @[1564353540]\n12161 @[1564357140]\n11027 @[1564360740]\n17777 @[1564364340]\n16825 @[1564367940]\n15552 @[1564371540]\n17230 @[1564375140]\n13810 @[1564378740]\n9074 @[1564382340]\n6849 @[1564385940]\n5176 @[1564389540]\n5155 @[1564393140]\n5750 @[1564396740]\n7325 @[1564400340]\n11877 @[1564403940]\n12024 @[1564407540]\n14407 @[1564411140]\n12542 @[1564414740]\n12871 @[1564418340]\n12772 @[1564421940]\n11971 @[1564425540]\n11587 @[1564429140]\n11104 @[1564432740]\n9706 @[1564436340]\n11877 @[1564439940]\n11446 @[1564443540]\n8783 @[1564447140]\n11136 @[1564450740]\n10898 @[1564454340]\n11466 @[1564457940]\n12203 @[1564461540]\n10432 @[1564465140]\n7108 @[1564468740]\n4628 @[1564472340]\n4362 @[1564475940]\n4647 @[1564479540]\n5903 @[1564483140]\n7594 @[1564486740]\n9677 @[1564490340]\n11779 @[1564493940]\n11237 @[1564497540]\n11392 @[1564501140]\n{le=\"0.01\", sg=\"victoriametrics\"} =>\n36288 @[1564328340]\n40119 @[1564331940]\n39745 @[1564335540]\n45535 @[1564339140]\n47647 @[1564342740]\n41953 @[1564346340]\n51241 @[1564349940]\n49375 @[1564353540]\n45946 @[1564357140]\n48590 @[1564360740]\n58940 @[1564364340]\n58080 @[1564367940]\n51630 @[1564371540]\n34947 @[1564375140]\n23966 @[1564378740]\n15311 @[1564382340]\n11346 @[1564385940]\n11635 @[1564389540]\n12400 @[1564393140]\n17985 @[1564396740]\n16855 @[1564400340]\n24593 @[1564403940]\n28634 @[1564407540]\n38829 @[1564411140]\n39355 @[1564414740]\n45833 @[1564418340]\n44300 @[1564421940]\n42952 @[1564425540]\n40855 @[1564429140]\n38345 @[1564432740]\n34489 @[1564436340]\n37172 @[1564439940]\n43045 @[1564443540]\n51528 @[1564447140]\n46247 @[1564450740]\n46511 @[1564454340]\n46748 @[1564457940]\n29636 @[1564461540]\n20503 @[1564465140]\n12840 @[1564468740]\n8409 @[1564472340]\n8479 @[1564475940]\n9314 @[1564479540]\n14298 @[1564483140]\n15384 @[1564486740]\n22000 @[1564490340]\n32048 @[1564493940]\n29617 @[1564497540]\n32891 @[1564501140]\n{le=\"0.1\", sg=\"victoriametrics\"} =>\n355289 @[1564328340]\n383441 @[1564331940]\n388465 @[1564335540]\n423242 @[1564339140]\n422938 @[1564342740]\n429082 @[1564346340]\n469354 @[1564349940]\n438155 @[1564353540]\n469358 @[1564357140]\n472820 @[1564360740]\n505228 @[1564364340]\n493116 @[1564367940]\n438669 @[1564371540]\n297525 @[1564375140]\n223981 @[1564378740]\n143263 @[1564382340]\n103174 @[1564385940]\n134694 @[1564389540]\n152256 @[1564393140]\n208486 @[1564396740]\n188251 @[1564400340]\n241661 @[1564403940]\n300128 @[1564407540]\n360863 @[1564411140]\n366177 @[1564414740]\n418392 @[1564418340]\n405699 @[1564421940]\n395592 @[1564425540]\n417416 @[1564429140]\n395369 @[1564432740]\n359862 @[1564436340]\n398021 @[1564439940]\n427201 @[1564443540]\n461214 @[1564447140]\n372600 @[1564450740]\n455924 @[1564454340]\n474661 @[1564457940]\n334003 @[1564461540]\n212585 @[1564465140]\n154851 @[1564468740]\n118326 @[1564472340]\n109314 @[1564475940]\n107980 @[1564479540]\n168761 @[1564483140]\n183342 @[1564486740]\n239781 @[1564490340]\n321900 @[1564493940]\n312250 @[1564497540]\n347161 @[1564501140]\n{le=\"0.25\", sg=\"victoriametrics\"} =>\n674991 @[1564328340]\n715242 @[1564331940]\n727093 @[1564335540]\n772432 @[1564339140]\n779900 @[1564342740]\n829819 @[1564346340]\n872000 @[1564349940]\n823609 @[1564353540]\n907161 @[1564357140]\n927054 @[1564360740]\n996855 @[1564364340]\n992356 @[1564367940]\n934826 @[1564371540]\n600717 @[1564375140]\n448432 @[1564378740]\n306148 @[1564382340]\n236506 @[1564385940]\n247719 @[1564389540]\n264540 @[1564393140]\n345693 @[1564396740]\n333292 @[1564400340]\n446455 @[1564403940]\n566595 @[1564407540]\n671580 @[1564411140]\n687488 @[1564414740]\n792909 @[1564418340]\n764988 @[1564421940]\n756140 @[1564425540]\n818998 @[1564429140]\n811607 @[1564432740]\n778140 @[1564436340]\n801509 @[1564439940]\n862641 @[1564443540]\n919925 @[1564447140]\n799632 @[1564450740]\n985822 @[1564454340]\n954168 @[1564457940]\n703500 @[1564461540]\n501703 @[1564465140]\n373938 @[1564468740]\n301645 @[1564472340]\n275743 @[1564475940]\n258420 @[1564479540]\n353818 @[1564483140]\n402009 @[1564486740]\n486328 @[1564490340]\n623258 @[1564493940]\n613427 @[1564497540]\n673260 @[1564501140]\n{le=\"0.5\", sg=\"victoriametrics\"} =>\n853901 @[1564328340]\n897624 @[1564331940]\n922860 @[1564335540]\n976140 @[1564339140]\n983774 @[1564342740]\n1060893 @[1564346340]\n1094715 @[1564349940]\n1060957 @[1564353540]\n1156453 @[1564357140]\n1205441 @[1564360740]\n1297490 @[1564364340]\n1298873 @[1564367940]\n1264817 @[1564371540]\n772751 @[1564375140]\n567104 @[1564378740]\n399275 @[1564382340]\n309301 @[1564385940]\n316627 @[1564389540]\n322818 @[1564393140]\n423047 @[1564396740]\n426182 @[1564400340]\n560958 @[1564403940]\n723898 @[1564407540]\n843501 @[1564411140]\n886703 @[1564414740]\n1010937 @[1564418340]\n987707 @[1564421940]\n983459 @[1564425540]\n1081606 @[1564429140]\n1097579 @[1564432740]\n1062933 @[1564436340]\n1058882 @[1564439940]\n1129055 @[1564443540]\n1249042 @[1564447140]\n1157350 @[1564450740]\n1354673 @[1564454340]\n1262890 @[1564457940]\n936571 @[1564461540]\n689480 @[1564465140]\n505972 @[1564468740]\n415012 @[1564472340]\n379220 @[1564475940]\n357950 @[1564479540]\n468511 @[1564483140]\n552452 @[1564486740]\n659037 @[1564490340]\n827036 @[1564493940]\n819416 @[1564497540]\n893553 @[1564501140]\n{le=\"1\", sg=\"victoriametrics\"} =>\n898772 @[1564328340]\n944442 @[1564331940]\n973524 @[1564335540]\n1034525 @[1564339140]\n1040370 @[1564342740]\n1127488 @[1564346340]\n1162504 @[1564349940]\n1142690 @[1564353540]\n1234397 @[1564357140]\n1301073 @[1564360740]\n1403056 @[1564364340]\n1405641 @[1564367940]\n1384645 @[1564371540]\n812367 @[1564375140]\n594166 @[1564378740]\n420712 @[1564382340]\n326227 @[1564385940]\n333520 @[1564389540]\n336857 @[1564393140]\n444209 @[1564396740]\n448651 @[1564400340]\n584460 @[1564403940]\n759771 @[1564407540]\n883985 @[1564411140]\n939708 @[1564414740]\n1068421 @[1564418340]\n1050611 @[1564421940]\n1047735 @[1564425540]\n1150901 @[1564429140]\n1181957 @[1564432740]\n1157829 @[1564436340]\n1131999 @[1564439940]\n1214661 @[1564443540]\n1384410 @[1564447140]\n1423306 @[1564450740]\n1507853 @[1564454340]\n1355737 @[1564457940]\n994929 @[1564461540]\n729686 @[1564465140]\n533853 @[1564468740]\n437014 @[1564472340]\n398342 @[1564475940]\n374066 @[1564479540]\n486085 @[1564483140]\n582140 @[1564486740]\n696679 @[1564490340]\n874301 @[1564493940]\n879290 @[1564497540]\n963904 @[1564501140]\n{le=\"10\", sg=\"victoriametrics\"} =>\n900701 @[1564328340]\n946727 @[1564331940]\n976194 @[1564335540]\n1038566 @[1564339140]\n1044686 @[1564342740]\n1131985 @[1564346340]\n1167843 @[1564349940]\n1150479 @[1564353540]\n1243417 @[1564357140]\n1317384 @[1564360740]\n1423297 @[1564364340]\n1430755 @[1564367940]\n1417215 @[1564371540]\n814138 @[1564375140]\n594949 @[1564378740]\n421511 @[1564382340]\n327055 @[1564385940]\n334166 @[1564389540]\n337814 @[1564393140]\n445100 @[1564396740]\n449520 @[1564400340]\n585377 @[1564403940]\n761785 @[1564407540]\n886062 @[1564411140]\n943217 @[1564414740]\n1072329 @[1564418340]\n1054224 @[1564421940]\n1053472 @[1564425540]\n1156887 @[1564429140]\n1194043 @[1564432740]\n1174238 @[1564436340]\n1138799 @[1564439940]\n1230970 @[1564443540]\n1421023 @[1564447140]\n1531312 @[1564450740]\n1549017 @[1564454340]\n1371788 @[1564457940]\n1000168 @[1564461540]\n732118 @[1564465140]\n535261 @[1564468740]\n438176 @[1564472340]\n399493 @[1564475940]\n375091 @[1564479540]\n487035 @[1564483140]\n583541 @[1564486740]\n698682 @[1564490340]\n876755 @[1564493940]\n882128 @[1564497540]\n968841 @[1564501140]\n{le=\"120\", sg=\"victoriametrics\"} =>\n900706 @[1564328340]\n946727 @[1564331940]\n976202 @[1564335540]\n1038578 @[1564339140]\n1044686 @[1564342740]\n1131985 @[1564346340]\n1167847 @[1564349940]\n1150481 @[1564353540]\n1243422 @[1564357140]\n1317385 @[1564360740]\n1423299 @[1564364340]\n1430829 @[1564367940]\n1417251 @[1564371540]\n814139 @[1564375140]\n594950 @[1564378740]\n422275 @[1564382340]\n327057 @[1564385940]\n334167 @[1564389540]\n337814 @[1564393140]\n445100 @[1564396740]\n449520 @[1564400340]\n585377 @[1564403940]\n761788 @[1564407540]\n886062 @[1564411140]\n943218 @[1564414740]\n1072332 @[1564418340]\n1054224 @[1564421940]\n1053473 @[1564425540]\n1156887 @[1564429140]\n1194061 @[1564432740]\n1174240 @[1564436340]\n1139232 @[1564439940]\n1230972 @[1564443540]\n1421080 @[1564447140]\n1531544 @[1564450740]\n1549557 @[1564454340]\n1372291 @[1564457940]\n1000169 @[1564461540]\n732119 @[1564465140]\n535268 @[1564468740]\n438176 @[1564472340]\n399494 @[1564475940]\n375091 @[1564479540]\n487035 @[1564483140]\n583542 @[1564486740]\n698682 @[1564490340]\n876755 @[1564493940]\n882128 @[1564497540]\n968841 @[1564501140]\n{le=\"5\", sg=\"victoriametrics\"} =>\n900701 @[1564328340]\n946721 @[1564331940]\n976185 @[1564335540]\n1038524 @[1564339140]\n1044682 @[1564342740]\n1131981 @[1564346340]\n1167837 @[1564349940]\n1150474 @[1564353540]\n1243181 @[1564357140]\n1317290 @[1564360740]\n1423041 @[1564364340]\n1428107 @[1564367940]\n1415555 @[1564371540]\n814132 @[1564375140]\n594945 @[1564378740]\n421504 @[1564382340]\n327053 @[1564385940]\n334165 @[1564389540]\n337814 @[1564393140]\n445096 @[1564396740]\n449520 @[1564400340]\n585373 @[1564403940]\n761784 @[1564407540]\n886061 @[1564411140]\n943217 @[1564414740]\n1072173 @[1564418340]\n1054177 @[1564421940]\n1053470 @[1564425540]\n1156877 @[1564429140]\n1193315 @[1564432740]\n1173760 @[1564436340]\n1138241 @[1564439940]\n1230710 @[1564443540]\n1419569 @[1564447140]\n1528305 @[1564450740]\n1547141 @[1564454340]\n1370588 @[1564457940]\n1000166 @[1564461540]\n732118 @[1564465140]\n535223 @[1564468740]\n438175 @[1564472340]\n399489 @[1564475940]\n375091 @[1564479540]\n487035 @[1564483140]\n583538 @[1564486740]\n698679 @[1564490340]\n876754 @[1564493940]\n882128 @[1564497540]\n968811 @[1564501140]\n{le=\"60\", sg=\"victoriametrics\"} =>\n900706 @[1564328340]\n946727 @[1564331940]\n976202 @[1564335540]\n1038578 @[1564339140]\n1044686 @[1564342740]\n1131985 @[1564346340]\n1167847 @[1564349940]\n1150481 @[1564353540]\n1243422 @[1564357140]\n1317384 @[1564360740]\n1423299 @[1564364340]\n1430829 @[1564367940]\n1417250 @[1564371540]\n814139 @[1564375140]\n594950 @[1564378740]\n422275 @[1564382340]\n327057 @[1564385940]\n334167 @[1564389540]\n337814 @[1564393140]\n445100 @[1564396740]\n449520 @[1564400340]\n585377 @[1564403940]\n761788 @[1564407540]\n886062 @[1564411140]\n943218 @[1564414740]\n1072332 @[1564418340]\n1054224 @[1564421940]\n1053473 @[1564425540]\n1156887 @[1564429140]\n1194061 @[1564432740]\n1174240 @[1564436340]\n1139232 @[1564439940]\n1230972 @[1564443540]\n1421080 @[1564447140]\n1531544 @[1564450740]\n1549557 @[1564454340]\n1372291 @[1564457940]\n1000169 @[1564461540]\n732119 @[1564465140]\n535268 @[1564468740]\n438176 @[1564472340]\n399494 @[1564475940]\n375091 @[1564479540]\n487035 @[1564483140]\n583542 @[1564486740]\n698682 @[1564490340]\n876755 @[1564493940]\n882128 @[1564497540]\n968841 @[1564501140]" warnings="[]"
127.0.0.1 - - [01/Aug/2019 11:04:06] "GET /api/v1/query_range HTTP/1.1 200 12343" 0.371989 query=sum%28increase%28nodeproxy_http_response_time_histogram_seconds_bucket%7Bapp%3D%22349%22%7D%5B10m%5D%29%29+by+%28le%29&start=1564328640&end=1564501440&step=3600

$ curl 'http://localhost:8085/api/v1/query_range?start=1564328640&end=1564501440&step=2880&query=sum(increase(nodeproxy_http_response_time_histogram_seconds_bucket%7Bapp%3D"349"%7D%5B10m%5D))%20by%20(le)'
{"status":"success","data":{"resultType":"matrix","result":[]}}

DEBU[2019-08-01T11:05:40Z] AggregateExpr sum by(le) (increase(nodeproxy_http_response_time_histogram_seconds_bucket{app="349"}[10m]))
DEBU[2019-08-01T11:05:40Z] http://prometheus-cache-do-nyc1-3abf.vpn.flyio.net:8428  api=QueryRange query="sum by(le) (increase(nodeproxy_http_response_time_histogram_seconds_bucket{app=\"349\"}[10m]))" r="{2019-07-28 15:39:00 +0000 UTC 2019-07-30 15:44:00 +0000 UTC 48m0s}"
DEBU[2019-08-01T11:05:40Z] http://prometheus-cache-do-nyc1-f31d.vpn.flyio.net:8428  api=QueryRange query="sum by(le) (increase(nodeproxy_http_response_time_histogram_seconds_bucket{app=\"349\"}[10m]))" r="{2019-07-28 15:39:00 +0000 UTC 2019-07-30 15:44:00 +0000 UTC 48m0s}"
TRAC[2019-08-01T11:05:40Z] http://prometheus-cache-do-nyc1-3abf.vpn.flyio.net:8428  api=QueryRange error="<nil>" query="sum by(le) (increase(nodeproxy_http_response_time_histogram_seconds_bucket{app=\"349\"}[10m]))" r="{2019-07-28 15:39:00 +0000 UTC 2019-07-30 15:44:00 +0000 UTC 48m0s}" took=89.025455ms value="{le=\"+Inf\", sg=\"victoriametrics\"} =>\n873738 @[1564326720]\n922178 @[1564329600]\n949835 @[1564332480]\n967461 @[1564335360]\n1008285 @[1564338240]\n1033779 @[1564341120]\n1081060 @[1564344000]\n1155936 @[1564346880]\n1182467 @[1564349760]\n1145209 @[1564352640]\n1179810 @[1564355520]\n1205672 @[1564358400]\n1358533 @[1564361280]\n1418970 @[1564364160]\n1461188 @[1564367040]\n1414650 @[1564369920]\n1098251 @[1564372800]\n762144 @[1564375680]\n608857 @[1564378560]\n472929 @[1564381440]\n391043 @[1564384320]\n301929 @[1564387200]\n324254 @[1564390080]\n336610 @[1564392960]\n400939 @[1564395840]\n415918 @[1564398720]\n459115 @[1564401600]\n608794 @[1564404480]\n761405 @[1564407360]\n875078 @[1564410240]\n911329 @[1564413120]\n992561 @[1564416000]\n1047418 @[1564418880]\n1052952 @[1564421760]\n1034798 @[1564424640]\n1146156 @[1564427520]\n1176370 @[1564430400]\n1196514 @[1564433280]\n1173729 @[1564436160]\n1134836 @[1564439040]\n1197579 @[1564441920]\n1238984 @[1564444800]\n1444360 @[1564447680]\n1504501 @[1564450560]\n1577352 @[1564453440]\n1595856 @[1564456320]\n1258926 @[1564459200]\n889190 @[1564462080]\n744182 @[1564464960]\n570554 @[1564467840]\n463235 @[1564470720]\n426141 @[1564473600]\n411845 @[1564476480]\n372788 @[1564479360]\n410977 @[1564482240]\n537828 @[1564485120]\n596095 @[1564488000]\n731126 @[1564490880]\n866019 @[1564493760]\n882853 @[1564496640]\n966610 @[1564499520]\n989019 @[1564502400]\n{le=\"0.005\", sg=\"victoriametrics\"} =>\n12639 @[1564326720]\n13224 @[1564329600]\n13250 @[1564332480]\n13175 @[1564335360]\n13570 @[1564338240]\n13387 @[1564341120]\n11851 @[1564344000]\n11293 @[1564346880]\n12432 @[1564349760]\n12794 @[1564352640]\n12344 @[1564355520]\n11540 @[1564358400]\n11718 @[1564361280]\n16909 @[1564364160]\n17429 @[1564367040]\n16460 @[1564369920]\n18710 @[1564372800]\n16675 @[1564375680]\n14001 @[1564378560]\n10485 @[1564381440]\n7774 @[1564384320]\n6188 @[1564387200]\n4816 @[1564390080]\n5101 @[1564392960]\n5259 @[1564395840]\n7200 @[1564398720]\n8825 @[1564401600]\n12513 @[1564404480]\n12252 @[1564407360]\n12093 @[1564410240]\n12148 @[1564413120]\n12684 @[1564416000]\n12144 @[1564418880]\n12948 @[1564421760]\n12148 @[1564424640]\n11262 @[1564427520]\n10976 @[1564430400]\n9852 @[1564433280]\n9731 @[1564436160]\n12331 @[1564439040]\n11437 @[1564441920]\n11211 @[1564444800]\n9635 @[1564447680]\n10869 @[1564450560]\n10754 @[1564453440]\n9223 @[1564456320]\n12386 @[1564459200]\n12081 @[1564462080]\n10265 @[1564464960]\n8410 @[1564467840]\n5767 @[1564470720]\n3844 @[1564473600]\n4313 @[1564476480]\n4796 @[1564479360]\n5545 @[1564482240]\n6497 @[1564485120]\n8079 @[1564488000]\n10322 @[1564490880]\n11657 @[1564493760]\n11269 @[1564496640]\n10253 @[1564499520]\n10928 @[1564502400]\n{le=\"0.01\", sg=\"victoriametrics\"} =>\n33996 @[1564326720]\n38218 @[1564329600]\n39512 @[1564332480]\n39428 @[1564335360]\n48000 @[1564338240]\n47683 @[1564341120]\n41520 @[1564344000]\n41970 @[1564346880]\n51130 @[1564349760]\n49578 @[1564352640]\n42302 @[1564355520]\n43267 @[1564358400]\n50461 @[1564361280]\n58303 @[1564364160]\n58743 @[1564367040]\n57542 @[1564369920]\n48140 @[1564372800]\n31347 @[1564375680]\n24707 @[1564378560]\n17488 @[1564381440]\n13000 @[1564384320]\n10038 @[1564387200]\n10312 @[1564390080]\n13023 @[1564392960]\n15298 @[1564395840]\n18694 @[1564398720]\n19903 @[1564401600]\n25011 @[1564404480]\n28446 @[1564407360]\n33324 @[1564410240]\n35404 @[1564413120]\n41638 @[1564416000]\n38885 @[1564418880]\n44021 @[1564421760]\n42322 @[1564424640]\n37568 @[1564427520]\n38801 @[1564430400]\n38127 @[1564433280]\n35624 @[1564436160]\n39392 @[1564439040]\n38689 @[1564441920]\n46639 @[1564444800]\n51485 @[1564447680]\n43296 @[1564450560]\n47460 @[1564453440]\n42178 @[1564456320]\n34811 @[1564459200]\n25204 @[1564462080]\n20955 @[1564464960]\n14321 @[1564467840]\n10471 @[1564470720]\n8896 @[1564473600]\n8710 @[1564476480]\n9254 @[1564479360]\n11496 @[1564482240]\n15176 @[1564485120]\n17820 @[1564488000]\n23992 @[1564490880]\n31424 @[1564493760]\n30365 @[1564496640]\n29631 @[1564499520]\n33551 @[1564502400]\n{le=\"0.1\", sg=\"victoriametrics\"} =>\n353411 @[1564326720]\n370070 @[1564329600]\n384241 @[1564332480]\n388375 @[1564335360]\n426540 @[1564338240]\n429446 @[1564341120]\n418227 @[1564344000]\n443277 @[1564346880]\n469621 @[1564349760]\n447029 @[1564352640]\n439507 @[1564355520]\n417005 @[1564358400]\n479953 @[1564361280]\n503831 @[1564364160]\n505224 @[1564367040]\n506134 @[1564369920]\n393669 @[1564372800]\n281747 @[1564375680]\n231114 @[1564378560]\n164992 @[1564381440]\n126971 @[1564384320]\n92894 @[1564387200]\n126594 @[1564390080]\n151847 @[1564392960]\n197299 @[1564395840]\n194091 @[1564398720]\n194841 @[1564401600]\n254227 @[1564404480]\n302245 @[1564407360]\n339669 @[1564410240]\n346799 @[1564413120]\n388155 @[1564416000]\n396396 @[1564418880]\n402597 @[1564421760]\n395027 @[1564424640]\n384869 @[1564427520]\n393060 @[1564430400]\n408275 @[1564433280]\n369112 @[1564436160]\n403714 @[1564439040]\n413045 @[1564441920]\n419731 @[1564444800]\n434079 @[1564447680]\n341648 @[1564450560]\n448060 @[1564453440]\n433207 @[1564456320]\n433147 @[1564459200]\n276114 @[1564462080]\n219014 @[1564464960]\n168390 @[1564467840]\n128615 @[1564470720]\n114247 @[1564473600]\n116473 @[1564476480]\n105493 @[1564479360]\n125514 @[1564482240]\n182548 @[1564485120]\n198821 @[1564488000]\n250880 @[1564490880]\n316277 @[1564493760]\n308627 @[1564496640]\n329225 @[1564499520]\n337752 @[1564502400]\n{le=\"0.25\", sg=\"victoriametrics\"} =>\n659291 @[1564326720]\n692348 @[1564329600]\n720480 @[1564332480]\n722795 @[1564335360]\n762978 @[1564338240]\n780749 @[1564341120]\n797442 @[1564344000]\n851782 @[1564346880]\n879227 @[1564349760]\n834756 @[1564352640]\n848879 @[1564355520]\n838076 @[1564358400]\n943498 @[1564361280]\n991277 @[1564364160]\n1009415 @[1564367040]\n1005856 @[1564369920]\n797167 @[1564372800]\n570145 @[1564375680]\n462264 @[1564378560]\n349631 @[1564381440]\n289336 @[1564384320]\n212211 @[1564387200]\n238544 @[1564390080]\n265091 @[1564392960]\n322907 @[1564395840]\n328129 @[1564398720]\n354463 @[1564401600]\n464082 @[1564404480]\n570049 @[1564407360]\n645129 @[1564410240]\n667419 @[1564413120]\n724293 @[1564416000]\n774617 @[1564418880]\n763052 @[1564421760]\n746362 @[1564424640]\n791552 @[1564427520]\n806410 @[1564430400]\n833835 @[1564433280]\n786015 @[1564436160]\n811756 @[1564439040]\n832055 @[1564441920]\n846846 @[1564444800]\n882511 @[1564447680]\n757521 @[1564450560]\n973701 @[1564453440]\n937391 @[1564456320]\n880963 @[1564459200]\n619607 @[1564462080]\n509507 @[1564464960]\n398536 @[1564467840]\n318125 @[1564470720]\n289966 @[1564473600]\n285534 @[1564476480]\n255415 @[1564479360]\n287887 @[1564482240]\n383198 @[1564485120]\n419365 @[1564488000]\n509597 @[1564490880]\n617181 @[1564493760]\n608396 @[1564496640]\n654104 @[1564499520]\n666501 @[1564502400]\n{le=\"0.5\", sg=\"victoriametrics\"} =>\n827671 @[1564326720]\n873614 @[1564329600]\n902252 @[1564332480]\n914032 @[1564335360]\n951893 @[1564338240]\n976370 @[1564341120]\n1015294 @[1564344000]\n1082457 @[1564346880]\n1105455 @[1564349760]\n1063894 @[1564352640]\n1093981 @[1564355520]\n1091574 @[1564358400]\n1234072 @[1564361280]\n1294336 @[1564364160]\n1319061 @[1564367040]\n1305919 @[1564369920]\n1032448 @[1564372800]\n725025 @[1564375680]\n581354 @[1564378560]\n449923 @[1564381440]\n372626 @[1564384320]\n282253 @[1564387200]\n306582 @[1564390080]\n322224 @[1564392960]\n384742 @[1564395840]\n399590 @[1564398720]\n440461 @[1564401600]\n583399 @[1564404480]\n725335 @[1564407360]\n829489 @[1564410240]\n861898 @[1564413120]\n932762 @[1564416000]\n984267 @[1564418880]\n985248 @[1564421760]\n968128 @[1564424640]\n1061646 @[1564427520]\n1088635 @[1564430400]\n1111664 @[1564433280]\n1066174 @[1564436160]\n1063815 @[1564439040]\n1100829 @[1564441920]\n1125406 @[1564444800]\n1235213 @[1564447680]\n1113824 @[1564450560]\n1351907 @[1564453440]\n1316517 @[1564456320]\n1162362 @[1564459200]\n837006 @[1564462080]\n700541 @[1564464960]\n540025 @[1564467840]\n437932 @[1564470720]\n402446 @[1564473600]\n391206 @[1564476480]\n355336 @[1564479360]\n392734 @[1564482240]\n513374 @[1564485120]\n565597 @[1564488000]\n689813 @[1564490880]\n817896 @[1564493760]\n815020 @[1564496640]\n890089 @[1564499520]\n902350 @[1564502400]\n{le=\"1\", sg=\"victoriametrics\"} =>\n871355 @[1564326720]\n919954 @[1564329600]\n947711 @[1564332480]\n964786 @[1564335360]\n1005417 @[1564338240]\n1030905 @[1564341120]\n1076889 @[1564344000]\n1150169 @[1564346880]\n1176087 @[1564349760]\n1138146 @[1564352640]\n1173148 @[1564355520]\n1189491 @[1564358400]\n1337772 @[1564361280]\n1401977 @[1564364160]\n1432492 @[1564367040]\n1397627 @[1564369920]\n1092107 @[1564372800]\n760561 @[1564375680]\n608092 @[1564378560]\n472053 @[1564381440]\n390262 @[1564384320]\n301078 @[1564387200]\n323627 @[1564390080]\n335678 @[1564392960]\n400052 @[1564395840]\n415199 @[1564398720]\n458425 @[1564401600]\n607753 @[1564404480]\n759839 @[1564407360]\n873180 @[1564410240]\n908731 @[1564413120]\n989248 @[1564416000]\n1042665 @[1564418880]\n1048809 @[1564421760]\n1030479 @[1564424640]\n1137952 @[1564427520]\n1166990 @[1564430400]\n1187804 @[1564433280]\n1157052 @[1564436160]\n1127888 @[1564439040]\n1184876 @[1564441920]\n1220547 @[1564444800]\n1406491 @[1564447680]\n1386526 @[1564450560]\n1531370 @[1564453440]\n1504435 @[1564456320]\n1247244 @[1564459200]\n886550 @[1564462080]\n741889 @[1564464960]\n569162 @[1564467840]\n461974 @[1564470720]\n424618 @[1564473600]\n410531 @[1564476480]\n371699 @[1564479360]\n409731 @[1564482240]\n536540 @[1564485120]\n594734 @[1564488000]\n729282 @[1564490880]\n863788 @[1564493760]\n879582 @[1564496640]\n961367 @[1564499520]\n982313 @[1564502400]\n{le=\"10\", sg=\"victoriametrics\"} =>\n873737 @[1564326720]\n922173 @[1564329600]\n949831 @[1564332480]\n967454 @[1564335360]\n1008284 @[1564338240]\n1033779 @[1564341120]\n1081060 @[1564344000]\n1155936 @[1564346880]\n1182462 @[1564349760]\n1145209 @[1564352640]\n1179809 @[1564355520]\n1205668 @[1564358400]\n1358515 @[1564361280]\n1418969 @[1564364160]\n1461184 @[1564367040]\n1414638 @[1564369920]\n1098251 @[1564372800]\n762144 @[1564375680]\n608856 @[1564378560]\n472928 @[1564381440]\n391043 @[1564384320]\n301928 @[1564387200]\n324254 @[1564390080]\n336609 @[1564392960]\n400939 @[1564395840]\n415915 @[1564398720]\n459114 @[1564401600]\n608794 @[1564404480]\n761401 @[1564407360]\n875078 @[1564410240]\n911329 @[1564413120]\n992558 @[1564416000]\n1047418 @[1564418880]\n1052952 @[1564421760]\n1034798 @[1564424640]\n1146156 @[1564427520]\n1176369 @[1564430400]\n1196512 @[1564433280]\n1173728 @[1564436160]\n1134836 @[1564439040]\n1197578 @[1564441920]\n1238983 @[1564444800]\n1444337 @[1564447680]\n1504269 @[1564450560]\n1577348 @[1564453440]\n1595442 @[1564456320]\n1258920 @[1564459200]\n889190 @[1564462080]\n744181 @[1564464960]\n570554 @[1564467840]\n463235 @[1564470720]\n426126 @[1564473600]\n411844 @[1564476480]\n372788 @[1564479360]\n410976 @[1564482240]\n537828 @[1564485120]\n596095 @[1564488000]\n731126 @[1564490880]\n866019 @[1564493760]\n882852 @[1564496640]\n966609 @[1564499520]\n989018 @[1564502400]\n{le=\"120\", sg=\"victoriametrics\"} =>\n873738 @[1564326720]\n922178 @[1564329600]\n949835 @[1564332480]\n967461 @[1564335360]\n1008285 @[1564338240]\n1033779 @[1564341120]\n1081060 @[1564344000]\n1155936 @[1564346880]\n1182467 @[1564349760]\n1145209 @[1564352640]\n1179810 @[1564355520]\n1205672 @[1564358400]\n1358533 @[1564361280]\n1418970 @[1564364160]\n1461188 @[1564367040]\n1414650 @[1564369920]\n1098251 @[1564372800]\n762144 @[1564375680]\n608857 @[1564378560]\n472929 @[1564381440]\n391043 @[1564384320]\n301929 @[1564387200]\n324254 @[1564390080]\n336610 @[1564392960]\n400939 @[1564395840]\n415918 @[1564398720]\n459115 @[1564401600]\n608794 @[1564404480]\n761405 @[1564407360]\n875078 @[1564410240]\n911329 @[1564413120]\n992561 @[1564416000]\n1047418 @[1564418880]\n1052952 @[1564421760]\n1034798 @[1564424640]\n1146156 @[1564427520]\n1176370 @[1564430400]\n1196514 @[1564433280]\n1173729 @[1564436160]\n1134836 @[1564439040]\n1197579 @[1564441920]\n1238984 @[1564444800]\n1444360 @[1564447680]\n1504501 @[1564450560]\n1577352 @[1564453440]\n1595856 @[1564456320]\n1258926 @[1564459200]\n889190 @[1564462080]\n744182 @[1564464960]\n570554 @[1564467840]\n463235 @[1564470720]\n426141 @[1564473600]\n411845 @[1564476480]\n372788 @[1564479360]\n410977 @[1564482240]\n537828 @[1564485120]\n596095 @[1564488000]\n731126 @[1564490880]\n866019 @[1564493760]\n882853 @[1564496640]\n966610 @[1564499520]\n989019 @[1564502400]\n{le=\"5\", sg=\"victoriametrics\"} =>\n873646 @[1564326720]\n922173 @[1564329600]\n949829 @[1564332480]\n967445 @[1564335360]\n1008283 @[1564338240]\n1033777 @[1564341120]\n1081058 @[1564344000]\n1155910 @[1564346880]\n1182460 @[1564349760]\n1145205 @[1564352640]\n1179749 @[1564355520]\n1205569 @[1564358400]\n1358250 @[1564361280]\n1418951 @[1564364160]\n1460538 @[1564367040]\n1414394 @[1564369920]\n1098251 @[1564372800]\n762141 @[1564375680]\n608853 @[1564378560]\n472928 @[1564381440]\n391041 @[1564384320]\n301923 @[1564387200]\n324253 @[1564390080]\n336609 @[1564392960]\n400939 @[1564395840]\n415914 @[1564398720]\n459112 @[1564401600]\n608794 @[1564404480]\n761400 @[1564407360]\n875072 @[1564410240]\n911326 @[1564413120]\n992556 @[1564416000]\n1047384 @[1564418880]\n1052905 @[1564421760]\n1034792 @[1564424640]\n1146141 @[1564427520]\n1176336 @[1564430400]\n1196126 @[1564433280]\n1171940 @[1564436160]\n1134787 @[1564439040]\n1197478 @[1564441920]\n1238906 @[1564444800]\n1443200 @[1564447680]\n1500312 @[1564450560]\n1576804 @[1564453440]\n1591478 @[1564456320]\n1258916 @[1564459200]\n889189 @[1564462080]\n744181 @[1564464960]\n570547 @[1564467840]\n463226 @[1564470720]\n426092 @[1564473600]\n411836 @[1564476480]\n372788 @[1564479360]\n410965 @[1564482240]\n537828 @[1564485120]\n596093 @[1564488000]\n731126 @[1564490880]\n866018 @[1564493760]\n882848 @[1564496640]\n966608 @[1564499520]\n989010 @[1564502400]\n{le=\"60\", sg=\"victoriametrics\"} =>\n873738 @[1564326720]\n922177 @[1564329600]\n949834 @[1564332480]\n967461 @[1564335360]\n1008285 @[1564338240]\n1033779 @[1564341120]\n1081060 @[1564344000]\n1155936 @[1564346880]\n1182467 @[1564349760]\n1145209 @[1564352640]\n1179810 @[1564355520]\n1205672 @[1564358400]\n1358533 @[1564361280]\n1418970 @[1564364160]\n1461188 @[1564367040]\n1414650 @[1564369920]\n1098251 @[1564372800]\n762144 @[1564375680]\n608857 @[1564378560]\n472929 @[1564381440]\n391043 @[1564384320]\n301929 @[1564387200]\n324254 @[1564390080]\n336609 @[1564392960]\n400939 @[1564395840]\n415916 @[1564398720]\n459115 @[1564401600]\n608794 @[1564404480]\n761405 @[1564407360]\n875078 @[1564410240]\n911329 @[1564413120]\n992561 @[1564416000]\n1047418 @[1564418880]\n1052952 @[1564421760]\n1034798 @[1564424640]\n1146156 @[1564427520]\n1176370 @[1564430400]\n1196514 @[1564433280]\n1173729 @[1564436160]\n1134836 @[1564439040]\n1197579 @[1564441920]\n1238984 @[1564444800]\n1444360 @[1564447680]\n1504501 @[1564450560]\n1577352 @[1564453440]\n1595856 @[1564456320]\n1258926 @[1564459200]\n889190 @[1564462080]\n744182 @[1564464960]\n570554 @[1564467840]\n463235 @[1564470720]\n426141 @[1564473600]\n411845 @[1564476480]\n372788 @[1564479360]\n410977 @[1564482240]\n537828 @[1564485120]\n596095 @[1564488000]\n731126 @[1564490880]\n866019 @[1564493760]\n882853 @[1564496640]\n966610 @[1564499520]\n989019 @[1564502400]" warnings="[]"
TRAC[2019-08-01T11:05:40Z] http://prometheus-cache-do-nyc1-f31d.vpn.flyio.net:8428  api=QueryRange error="<nil>" query="sum by(le) (increase(nodeproxy_http_response_time_histogram_seconds_bucket{app=\"349\"}[10m]))" r="{2019-07-28 15:39:00 +0000 UTC 2019-07-30 15:44:00 +0000 UTC 48m0s}" took=112.167844ms value="{le=\"+Inf\", sg=\"victoriametrics\"} =>\n873738 @[1564326720]\n922178 @[1564329600]\n949835 @[1564332480]\n967461 @[1564335360]\n1008167 @[1564338240]\n1033779 @[1564341120]\n1081060 @[1564344000]\n1155936 @[1564346880]\n1182715 @[1564349760]\n1145209 @[1564352640]\n1179738 @[1564355520]\n1203984 @[1564358400]\n1358533 @[1564361280]\n1418820 @[1564364160]\n1461188 @[1564367040]\n1414650 @[1564369920]\n1098251 @[1564372800]\n762144 @[1564375680]\n608818 @[1564378560]\n472929 @[1564381440]\n391043 @[1564384320]\n302082 @[1564387200]\n324329 @[1564390080]\n336610 @[1564392960]\n400939 @[1564395840]\n415918 @[1564398720]\n458436 @[1564401600]\n608794 @[1564404480]\n761405 @[1564407360]\n875078 @[1564410240]\n911329 @[1564413120]\n992561 @[1564416000]\n1047418 @[1564418880]\n1052952 @[1564421760]\n1034798 @[1564424640]\n1146156 @[1564427520]\n1176370 @[1564430400]\n1196514 @[1564433280]\n1173729 @[1564436160]\n1134836 @[1564439040]\n1197579 @[1564441920]\n1237639 @[1564444800]\n1444516 @[1564447680]\n1504501 @[1564450560]\n1577352 @[1564453440]\n1595856 @[1564456320]\n1258926 @[1564459200]\n889190 @[1564462080]\n744182 @[1564464960]\n570554 @[1564467840]\n463235 @[1564470720]\n426141 @[1564473600]\n411845 @[1564476480]\n372788 @[1564479360]\n410977 @[1564482240]\n537828 @[1564485120]\n595698 @[1564488000]\n730963 @[1564490880]\n866019 @[1564493760]\n882853 @[1564496640]\n966610 @[1564499520]\n989019 @[1564502400]\n{le=\"0.005\", sg=\"victoriametrics\"} =>\n12639 @[1564326720]\n13224 @[1564329600]\n13250 @[1564332480]\n13175 @[1564335360]\n13570 @[1564338240]\n13387 @[1564341120]\n11851 @[1564344000]\n11293 @[1564346880]\n12432 @[1564349760]\n12794 @[1564352640]\n12344 @[1564355520]\n11441 @[1564358400]\n11718 @[1564361280]\n16909 @[1564364160]\n17429 @[1564367040]\n16460 @[1564369920]\n18710 @[1564372800]\n16675 @[1564375680]\n14001 @[1564378560]\n10485 @[1564381440]\n7774 @[1564384320]\n6188 @[1564387200]\n4816 @[1564390080]\n5101 @[1564392960]\n5259 @[1564395840]\n7200 @[1564398720]\n8754 @[1564401600]\n12513 @[1564404480]\n12252 @[1564407360]\n12093 @[1564410240]\n12148 @[1564413120]\n12684 @[1564416000]\n12144 @[1564418880]\n12948 @[1564421760]\n12148 @[1564424640]\n11262 @[1564427520]\n10977 @[1564430400]\n9852 @[1564433280]\n9731 @[1564436160]\n12331 @[1564439040]\n11437 @[1564441920]\n11137 @[1564444800]\n9635 @[1564447680]\n10869 @[1564450560]\n10754 @[1564453440]\n9223 @[1564456320]\n12386 @[1564459200]\n12081 @[1564462080]\n10265 @[1564464960]\n8410 @[1564467840]\n5767 @[1564470720]\n3844 @[1564473600]\n4313 @[1564476480]\n4796 @[1564479360]\n5545 @[1564482240]\n6497 @[1564485120]\n8030 @[1564488000]\n10322 @[1564490880]\n11657 @[1564493760]\n11270 @[1564496640]\n10253 @[1564499520]\n10928 @[1564502400]\n{le=\"0.01\", sg=\"victoriametrics\"} =>\n33996 @[1564326720]\n38218 @[1564329600]\n39512 @[1564332480]\n39428 @[1564335360]\n48000 @[1564338240]\n47683 @[1564341120]\n41520 @[1564344000]\n41970 @[1564346880]\n51130 @[1564349760]\n49578 @[1564352640]\n42302 @[1564355520]\n42942 @[1564358400]\n50461 @[1564361280]\n58303 @[1564364160]\n58743 @[1564367040]\n57542 @[1564369920]\n48140 @[1564372800]\n31347 @[1564375680]\n24707 @[1564378560]\n17488 @[1564381440]\n13000 @[1564384320]\n10038 @[1564387200]\n10312 @[1564390080]\n13023 @[1564392960]\n15298 @[1564395840]\n18694 @[1564398720]\n19784 @[1564401600]\n25011 @[1564404480]\n28446 @[1564407360]\n33324 @[1564410240]\n35404 @[1564413120]\n41638 @[1564416000]\n38885 @[1564418880]\n44021 @[1564421760]\n42322 @[1564424640]\n37568 @[1564427520]\n38842 @[1564430400]\n38127 @[1564433280]\n35624 @[1564436160]\n39392 @[1564439040]\n38689 @[1564441920]\n46381 @[1564444800]\n51485 @[1564447680]\n43296 @[1564450560]\n47460 @[1564453440]\n42178 @[1564456320]\n34811 @[1564459200]\n25204 @[1564462080]\n20955 @[1564464960]\n14321 @[1564467840]\n10471 @[1564470720]\n8896 @[1564473600]\n8710 @[1564476480]\n9257 @[1564479360]\n11496 @[1564482240]\n15176 @[1564485120]\n17707 @[1564488000]\n23992 @[1564490880]\n31424 @[1564493760]\n30365 @[1564496640]\n29631 @[1564499520]\n33551 @[1564502400]\n{le=\"0.1\", sg=\"victoriametrics\"} =>\n353411 @[1564326720]\n370070 @[1564329600]\n384241 @[1564332480]\n388375 @[1564335360]\n426533 @[1564338240]\n429446 @[1564341120]\n418228 @[1564344000]\n443275 @[1564346880]\n469625 @[1564349760]\n447029 @[1564352640]\n439503 @[1564355520]\n415533 @[1564358400]\n479953 @[1564361280]\n503831 @[1564364160]\n505224 @[1564367040]\n506134 @[1564369920]\n393669 @[1564372800]\n281747 @[1564375680]\n231109 @[1564378560]\n164992 @[1564381440]\n126971 @[1564384320]\n92895 @[1564387200]\n126598 @[1564390080]\n151847 @[1564392960]\n197299 @[1564395840]\n194091 @[1564398720]\n194190 @[1564401600]\n254227 @[1564404480]\n302245 @[1564407360]\n339669 @[1564410240]\n346799 @[1564413120]\n388155 @[1564416000]\n396396 @[1564418880]\n402597 @[1564421760]\n395027 @[1564424640]\n384869 @[1564427520]\n393060 @[1564430400]\n408275 @[1564433280]\n369112 @[1564436160]\n403714 @[1564439040]\n413045 @[1564441920]\n418509 @[1564444800]\n434082 @[1564447680]\n341648 @[1564450560]\n448060 @[1564453440]\n433207 @[1564456320]\n433147 @[1564459200]\n276114 @[1564462080]\n219014 @[1564464960]\n168390 @[1564467840]\n128615 @[1564470720]\n114247 @[1564473600]\n116473 @[1564476480]\n105493 @[1564479360]\n125514 @[1564482240]\n182548 @[1564485120]\n198056 @[1564488000]\n250855 @[1564490880]\n316277 @[1564493760]\n308627 @[1564496640]\n329225 @[1564499520]\n337752 @[1564502400]\n{le=\"0.25\", sg=\"victoriametrics\"} =>\n659291 @[1564326720]\n692348 @[1564329600]\n720480 @[1564332480]\n722795 @[1564335360]\n762905 @[1564338240]\n780749 @[1564341120]\n797442 @[1564344000]\n851781 @[1564346880]\n879286 @[1564349760]\n834756 @[1564352640]\n848845 @[1564355520]\n835914 @[1564358400]\n943498 @[1564361280]\n991262 @[1564364160]\n1009415 @[1564367040]\n1005856 @[1564369920]\n797167 @[1564372800]\n570145 @[1564375680]\n462241 @[1564378560]\n349631 @[1564381440]\n289336 @[1564384320]\n212274 @[1564387200]\n238590 @[1564390080]\n265091 @[1564392960]\n322907 @[1564395840]\n328129 @[1564398720]\n353546 @[1564401600]\n464082 @[1564404480]\n570049 @[1564407360]\n645129 @[1564410240]\n667419 @[1564413120]\n724293 @[1564416000]\n774617 @[1564418880]\n763052 @[1564421760]\n746362 @[1564424640]\n791552 @[1564427520]\n806410 @[1564430400]\n833835 @[1564433280]\n786015 @[1564436160]\n811756 @[1564439040]\n832055 @[1564441920]\n845016 @[1564444800]\n882566 @[1564447680]\n757521 @[1564450560]\n973701 @[1564453440]\n937391 @[1564456320]\n880963 @[1564459200]\n619607 @[1564462080]\n509507 @[1564464960]\n398536 @[1564467840]\n318125 @[1564470720]\n289966 @[1564473600]\n285534 @[1564476480]\n255415 @[1564479360]\n287887 @[1564482240]\n383198 @[1564485120]\n418332 @[1564488000]\n509451 @[1564490880]\n617181 @[1564493760]\n608396 @[1564496640]\n654104 @[1564499520]\n666501 @[1564502400]\n{le=\"0.5\", sg=\"victoriametrics\"} =>\n827671 @[1564326720]\n873614 @[1564329600]\n902252 @[1564332480]\n914032 @[1564335360]\n951781 @[1564338240]\n976370 @[1564341120]\n1015294 @[1564344000]\n1082457 @[1564346880]\n1105645 @[1564349760]\n1063894 @[1564352640]\n1093915 @[1564355520]\n1089653 @[1564358400]\n1234072 @[1564361280]\n1294252 @[1564364160]\n1319061 @[1564367040]\n1305919 @[1564369920]\n1032448 @[1564372800]\n725025 @[1564375680]\n581318 @[1564378560]\n449923 @[1564381440]\n372626 @[1564384320]\n282387 @[1564387200]\n306652 @[1564390080]\n322224 @[1564392960]\n384742 @[1564395840]\n399590 @[1564398720]\n439736 @[1564401600]\n583399 @[1564404480]\n725335 @[1564407360]\n829489 @[1564410240]\n861898 @[1564413120]\n932762 @[1564416000]\n984267 @[1564418880]\n985248 @[1564421760]\n968128 @[1564424640]\n1061646 @[1564427520]\n1088635 @[1564430400]\n1111664 @[1564433280]\n1066174 @[1564436160]\n1063815 @[1564439040]\n1100829 @[1564441920]\n1123804 @[1564444800]\n1235353 @[1564447680]\n1113824 @[1564450560]\n1351907 @[1564453440]\n1316517 @[1564456320]\n1162362 @[1564459200]\n837006 @[1564462080]\n700541 @[1564464960]\n540025 @[1564467840]\n437932 @[1564470720]\n402446 @[1564473600]\n391206 @[1564476480]\n355336 @[1564479360]\n392734 @[1564482240]\n513374 @[1564485120]\n564977 @[1564488000]\n689653 @[1564490880]\n817896 @[1564493760]\n815020 @[1564496640]\n890089 @[1564499520]\n902350 @[1564502400]\n{le=\"1\", sg=\"victoriametrics\"} =>\n871355 @[1564326720]\n919954 @[1564329600]\n947711 @[1564332480]\n964786 @[1564335360]\n1005300 @[1564338240]\n1030905 @[1564341120]\n1076889 @[1564344000]\n1150169 @[1564346880]\n1176331 @[1564349760]\n1138146 @[1564352640]\n1173076 @[1564355520]\n1187789 @[1564358400]\n1337772 @[1564361280]\n1401828 @[1564364160]\n1432492 @[1564367040]\n1397627 @[1564369920]\n1092107 @[1564372800]\n760561 @[1564375680]\n608053 @[1564378560]\n472053 @[1564381440]\n390262 @[1564384320]\n301231 @[1564387200]\n323702 @[1564390080]\n335678 @[1564392960]\n400052 @[1564395840]\n415199 @[1564398720]\n457743 @[1564401600]\n607753 @[1564404480]\n759839 @[1564407360]\n873180 @[1564410240]\n908731 @[1564413120]\n989248 @[1564416000]\n1042665 @[1564418880]\n1048809 @[1564421760]\n1030479 @[1564424640]\n1137952 @[1564427520]\n1167006 @[1564430400]\n1187804 @[1564433280]\n1157052 @[1564436160]\n1127888 @[1564439040]\n1184876 @[1564441920]\n1219191 @[1564444800]\n1406647 @[1564447680]\n1386526 @[1564450560]\n1531370 @[1564453440]\n1504435 @[1564456320]\n1247244 @[1564459200]\n886550 @[1564462080]\n741889 @[1564464960]\n569162 @[1564467840]\n461974 @[1564470720]\n424618 @[1564473600]\n410531 @[1564476480]\n371699 @[1564479360]\n409731 @[1564482240]\n536540 @[1564485120]\n594330 @[1564488000]\n729119 @[1564490880]\n863788 @[1564493760]\n879582 @[1564496640]\n961367 @[1564499520]\n982313 @[1564502400]\n{le=\"10\", sg=\"victoriametrics\"} =>\n873737 @[1564326720]\n922173 @[1564329600]\n949831 @[1564332480]\n967454 @[1564335360]\n1008166 @[1564338240]\n1033779 @[1564341120]\n1081060 @[1564344000]\n1155936 @[1564346880]\n1182710 @[1564349760]\n1145209 @[1564352640]\n1179737 @[1564355520]\n1203980 @[1564358400]\n1358515 @[1564361280]\n1418819 @[1564364160]\n1461184 @[1564367040]\n1414638 @[1564369920]\n1098251 @[1564372800]\n762144 @[1564375680]\n608817 @[1564378560]\n472928 @[1564381440]\n391043 @[1564384320]\n302081 @[1564387200]\n324329 @[1564390080]\n336609 @[1564392960]\n400939 @[1564395840]\n415915 @[1564398720]\n458435 @[1564401600]\n608794 @[1564404480]\n761401 @[1564407360]\n875078 @[1564410240]\n911329 @[1564413120]\n992558 @[1564416000]\n1047418 @[1564418880]\n1052952 @[1564421760]\n1034798 @[1564424640]\n1146156 @[1564427520]\n1176369 @[1564430400]\n1196512 @[1564433280]\n1173728 @[1564436160]\n1134836 @[1564439040]\n1197578 @[1564441920]\n1237638 @[1564444800]\n1444493 @[1564447680]\n1504269 @[1564450560]\n1577348 @[1564453440]\n1595442 @[1564456320]\n1258920 @[1564459200]\n889190 @[1564462080]\n744181 @[1564464960]\n570554 @[1564467840]\n463235 @[1564470720]\n426126 @[1564473600]\n411844 @[1564476480]\n372788 @[1564479360]\n410976 @[1564482240]\n537828 @[1564485120]\n595698 @[1564488000]\n730963 @[1564490880]\n866019 @[1564493760]\n882852 @[1564496640]\n966609 @[1564499520]\n989018 @[1564502400]\n{le=\"120\", sg=\"victoriametrics\"} =>\n873738 @[1564326720]\n922178 @[1564329600]\n949835 @[1564332480]\n967461 @[1564335360]\n1008167 @[1564338240]\n1033779 @[1564341120]\n1081060 @[1564344000]\n1155936 @[1564346880]\n1182715 @[1564349760]\n1145209 @[1564352640]\n1179738 @[1564355520]\n1203984 @[1564358400]\n1358533 @[1564361280]\n1418820 @[1564364160]\n1461188 @[1564367040]\n1414650 @[1564369920]\n1098251 @[1564372800]\n762144 @[1564375680]\n608818 @[1564378560]\n472929 @[1564381440]\n391043 @[1564384320]\n302082 @[1564387200]\n324329 @[1564390080]\n336610 @[1564392960]\n400939 @[1564395840]\n415918 @[1564398720]\n458436 @[1564401600]\n608794 @[1564404480]\n761405 @[1564407360]\n875078 @[1564410240]\n911329 @[1564413120]\n992561 @[1564416000]\n1047418 @[1564418880]\n1052952 @[1564421760]\n1034798 @[1564424640]\n1146156 @[1564427520]\n1176370 @[1564430400]\n1196514 @[1564433280]\n1173729 @[1564436160]\n1134836 @[1564439040]\n1197579 @[1564441920]\n1237639 @[1564444800]\n1444516 @[1564447680]\n1504501 @[1564450560]\n1577352 @[1564453440]\n1595856 @[1564456320]\n1258926 @[1564459200]\n889190 @[1564462080]\n744182 @[1564464960]\n570554 @[1564467840]\n463235 @[1564470720]\n426141 @[1564473600]\n411845 @[1564476480]\n372788 @[1564479360]\n410977 @[1564482240]\n537828 @[1564485120]\n595698 @[1564488000]\n730963 @[1564490880]\n866019 @[1564493760]\n882853 @[1564496640]\n966610 @[1564499520]\n989019 @[1564502400]\n{le=\"5\", sg=\"victoriametrics\"} =>\n873646 @[1564326720]\n922173 @[1564329600]\n949829 @[1564332480]\n967445 @[1564335360]\n1008165 @[1564338240]\n1033777 @[1564341120]\n1081058 @[1564344000]\n1155910 @[1564346880]\n1182708 @[1564349760]\n1145205 @[1564352640]\n1179677 @[1564355520]\n1203881 @[1564358400]\n1358250 @[1564361280]\n1418801 @[1564364160]\n1460538 @[1564367040]\n1414394 @[1564369920]\n1098251 @[1564372800]\n762141 @[1564375680]\n608814 @[1564378560]\n472928 @[1564381440]\n391041 @[1564384320]\n302076 @[1564387200]\n324328 @[1564390080]\n336609 @[1564392960]\n400939 @[1564395840]\n415914 @[1564398720]\n458433 @[1564401600]\n608794 @[1564404480]\n761400 @[1564407360]\n875072 @[1564410240]\n911326 @[1564413120]\n992556 @[1564416000]\n1047384 @[1564418880]\n1052905 @[1564421760]\n1034792 @[1564424640]\n1146141 @[1564427520]\n1176336 @[1564430400]\n1196126 @[1564433280]\n1171940 @[1564436160]\n1134787 @[1564439040]\n1197478 @[1564441920]\n1237561 @[1564444800]\n1443356 @[1564447680]\n1500312 @[1564450560]\n1576804 @[1564453440]\n1591478 @[1564456320]\n1258916 @[1564459200]\n889189 @[1564462080]\n744181 @[1564464960]\n570547 @[1564467840]\n463226 @[1564470720]\n426092 @[1564473600]\n411836 @[1564476480]\n372788 @[1564479360]\n410965 @[1564482240]\n537828 @[1564485120]\n595696 @[1564488000]\n730963 @[1564490880]\n866018 @[1564493760]\n882848 @[1564496640]\n966608 @[1564499520]\n989010 @[1564502400]\n{le=\"60\", sg=\"victoriametrics\"} =>\n873738 @[1564326720]\n922177 @[1564329600]\n949834 @[1564332480]\n967461 @[1564335360]\n1008167 @[1564338240]\n1033779 @[1564341120]\n1081060 @[1564344000]\n1155936 @[1564346880]\n1182715 @[1564349760]\n1145209 @[1564352640]\n1179738 @[1564355520]\n1203984 @[1564358400]\n1358533 @[1564361280]\n1418820 @[1564364160]\n1461188 @[1564367040]\n1414650 @[1564369920]\n1098251 @[1564372800]\n762144 @[1564375680]\n608818 @[1564378560]\n472929 @[1564381440]\n391043 @[1564384320]\n302082 @[1564387200]\n324329 @[1564390080]\n336609 @[1564392960]\n400939 @[1564395840]\n415916 @[1564398720]\n458436 @[1564401600]\n608794 @[1564404480]\n761405 @[1564407360]\n875078 @[1564410240]\n911329 @[1564413120]\n992561 @[1564416000]\n1047418 @[1564418880]\n1052952 @[1564421760]\n1034798 @[1564424640]\n1146156 @[1564427520]\n1176370 @[1564430400]\n1196514 @[1564433280]\n1173729 @[1564436160]\n1134836 @[1564439040]\n1197579 @[1564441920]\n1237639 @[1564444800]\n1444516 @[1564447680]\n1504501 @[1564450560]\n1577352 @[1564453440]\n1595856 @[1564456320]\n1258926 @[1564459200]\n889190 @[1564462080]\n744182 @[1564464960]\n570554 @[1564467840]\n463235 @[1564470720]\n426141 @[1564473600]\n411845 @[1564476480]\n372788 @[1564479360]\n410977 @[1564482240]\n537828 @[1564485120]\n595698 @[1564488000]\n730963 @[1564490880]\n866019 @[1564493760]\n882853 @[1564496640]\n966610 @[1564499520]\n989019 @[1564502400]" warnings="[]"
127.0.0.1 - - [01/Aug/2019 11:05:40] "GET /api/v1/query_range HTTP/1.1 200 63" 0.116111 start=1564328640&end=1564501440&step=2880&query=sum%28increase%28nodeproxy_http_response_time_histogram_seconds_bucket%7Bapp%3D%22349%22%7D%5B10m%5D%29%29+by+%28le%29```

[jacksontj]

@steveberryman that is very curious, I have attempted reproducing locally and am unable to do so. So a few things that I can think of that would help me debug. (1) Could you share your config file? (2) Could attach a tcpdump of the promxy host during this query that would help me with a more accurate repro scenario. (3) what version of promxy are you seeing this on (I'm assuming master or v0.0.46 but figured I should check).

[jacksontj]

@steveberryman assuming you are still able to repro this issue, it would be very helpful if you could provide the above requested data. At this point I can't repro this issue :(

[steveberryman]

Hi! So sorry! I was on vacation and then forgot about this ticket.

Attached is a pcap of a number of requests. First two are the queries mentioned above - one working, and one not with the same time ranges. Then there are two more queries with more recent time ranges which both seem to work for some reason!

Also attached is the config file.

I can confirm we're using promxy 0.0.46

promxy_pcap_config.zip

[jacksontj]

Using the data returned in your pcap I'm able to reproduce the problem (creating stub servers that return the data in the tcpdump etc.). I attempted to reproduce the same behavior with prometheus downstream but was unable to reproduce. The issue here is with how VM returns to query_range requests (upstream issue: VictoriaMetrics/VictoriaMetrics#171).

An attempted short explanation:

query_range responses should line up their start and step with where the start/step of the call was-- in this case they don't line up so points are at the wrong "time" in the response-- when the promql engine tries to find the points it determines they are "too old" (since they are so out of position). Shorter steps (depending on the offset of the downstream) mean that there are points within the tolerance (stalenessDelta).

TLDR; VM isn't returning to query_range properly so it looks like the data isn't there (since its at the wrong time). Unfortunately this will require a fix/change in VM to resolve as there isn't really an issue in promxy :/

[steveberryman]

Ah! Thanks for investigating so thoroughly!

…

-- Steve Berryman From: Thomas Jackson <notifications@github.com> <notifications@github.com> Reply: jacksontj/promxy <reply@reply.github.com> <reply@reply.github.com> Date: September 3, 2019 at 7:32:13 PM To: jacksontj/promxy <promxy@noreply.github.com> <promxy@noreply.github.com> CC: Steve Berryman <steve@berryman.org.uk> <steve@berryman.org.uk>, Mention <mention@noreply.github.com> <mention@noreply.github.com> Subject: Re: [jacksontj/promxy] Certain step values for a query return no results (#202) Using the data returned in your pcap I'm able to reproduce the problem

(creating stub servers that return the data in the tcpdump etc.). I attempted to reproduce the same behavior with prometheus downstream but was unable to reproduce. The issue here is with how VM returns to query_range requests (upstream issue: VictoriaMetrics/VictoriaMetrics#171 <VictoriaMetrics/VictoriaMetrics#171>). An attempted short explanation: query_range responses should line up their start and step with where the start/step of the call was-- in this case they don't line up so points are at the wrong "time" in the response-- when the promql engine tries to find the points it determines they are "too old" (since they are so out of position). Shorter steps (depending on the offset of the downstream) mean that there are points within the tolerance (stalenessDelta). TLDR; VM isn't returning to query_range properly so it looks like the data isn't there (since its at the wrong time). Unfortunately this will require a fix/change in VM to resolve as there isn't really an issue in promxy :/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#202?email_source=notifications&email_token=AAALSJR433GTCCNCXPPQSJ3QH2US3A5CNFSM4IH6Q2C2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5ZEPAA#issuecomment-527583104>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAALSJQDHHSH5OLKKD7VSH3QH2US3ANCNFSM4IH6Q2CQ> .

[jacksontj]

I've added a query_params option in #216 to allow for a workaround. If you add the new query_params option example from here: https://github.com/jacksontj/promxy/blob/master/cmd/promxy/config.yaml#L50