Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

store: More than 2000 Connections/s to S3 Backend #1492

Closed
muryoutaisuu opened this issue Sep 4, 2019 · 4 comments
Closed

store: More than 2000 Connections/s to S3 Backend #1492

muryoutaisuu opened this issue Sep 4, 2019 · 4 comments
Labels
component: store question

Comments

@muryoutaisuu
Copy link
Contributor

In case of issues related to exact bucket implementation, please ping corresponded maintainer from list here: https://github.com/thanos-io/thanos/blob/master/docs/storage.md

pinging @bwplotka because of S3 backend

Thanos, Prometheus and Golang version used

prometheus --version
prometheus, version 2.12.0 (branch: HEAD, revision: 43acd0e2e93f9f70c49b2267efa0124f1e759e86)
  build user:       root@7a9dbdbe0cc7
  build date:       20190818-13:53:16
  go version:       go1.12.8

thanos --version
thanos, version 0.7.0 (branch: HEAD, revision: c6eaf68bec32aefd280318eaef6121d6ddb20d10)
  build user:       root@aa355daaf987
  build date:       20190902-15:31:30
  go version:       go1.12.5

What happened

Sometimes, Thanos Store is making more than 2000 connections per second to S3 backend. The firewall blocks any connections after that for 1 minute (DDOS prevention).
It occurrs as infrequent spikes and I can't yet relate to any event. I did not see any queries being done on Grafana or Thanos Query component shortly before or at the moment of those spikes, that could have caused them.

I updated on 03. September 2019 ~19:00 GMT+02:00 from v0.6.1 to Thanos 0.7.0, hoping that the newly merged

#1302 Thanos now efficiently reuses HTTP keep-alive connections

would fix this issue.

What you expected to happen

To not spike that much.

How to reproduce it (as minimally and precisely as possible):

I can't relate any event, so I'm not sure what to do to reproduce them.

Thanos Store Startup Configuration

/appl/thanos/bin/thanos store --data-dir=/appl/thanos/data-cache --grpc-address=0.0.0.0:7023 --http-address=0.0.0.0:7007 --log.level=debug --objstore.config-file=/appl/thanos/.s3.yml

Contents of --objstore.config-file

---
config:
  access_key: <REDACTED>
  bucket: p-prometheus-thanos
  encrypt_sse: false
  endpoint: <REDACTED_ON_PREMISE>
  http_config:
    idle_conn_timeout: 0s
    insecure_skip_verify: true
  insecure: false
  put_user_metadata: {}
  secret_key: <REDACTED>
  signature_version2: false
  trace:
    enable: false
type: S3

Full logs to relevant components

All Thanos components are started via systemd. Log output is transferred to /var/log/thanos.log via rsyslog. Hence the logs may also contain logs from Thanos Compactor & Query components.

Apparently the 2000/s connection threshold was reached at 13:04:11, but I can't see anything suspicious in the logs.

Logs 

Sep  4 13:04:01 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:01.825817391Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:1316 postingsTouched:23049 postingsTouchedSizeSum:811328 postingsToFetch:0 postingsFetched:23049 postingsFetchedSizeSum:993152 postingsFetchCount:321 postingsFetchDurationSum:46153336980 seriesTouched:179783 seriesTouchedSizeSum:13566197 seriesFetched:179783 seriesFetchedSizeSum:97120224 seriesFetchCount:609 seriesFetchDurationSum:701108188367 chunksTouched:1288460 chunksTouchedSizeSum:116520069 chunksFetched:1288460 chunksFetchedSizeSum:191147868 chunksFetchCount:916 chunksFetchDurationSum:107864552498 getAllDuration:1063268359278 mergedSeriesCount:1 mergedChunksCount:32 mergeDuration:0}" err="rpc error: code = Unknown desc = send series response: rpc error: code = Internal desc = transport: transport: the stream is done or WriteHeader was already called"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.079653872Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 filtered out;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.086267514Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 queried;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.086699708Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567573140000 maxt=1567595055000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1567569600000-1567584000000 Resolution: 0"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.086880897Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567573140000 maxt=1567595055000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1567569600000-1567584000000 Resolution: 0"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.086963687Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567573140000 maxt=1567595055000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1567569600000-1567584000000 Resolution: 0"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.087027686Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567573140000 maxt=1567595055000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1567569600000-1567584000000 Resolution: 0"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.087291432Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:8 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:137693 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:4021}" err=null
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.611764342Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 filtered out"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.634286054Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 filtered out;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.646902717Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 filtered out;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.651589353Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 filtered out"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.659394176Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 filtered out"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.690827191Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 filtered out"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.694419501Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 filtered out"
Sep  4 13:04:02 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:02.719835135Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 filtered out"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.616888164Z caller=bucket.go:325 msg="error parsing block range" block=01DKW2PYFZ47Q954ZB76V5M78P err="read meta.json: open /appl/thanos/data-cache/01DKW2PYFZ47Q954ZB76V5M78P/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.616990476Z caller=bucket.go:325 msg="error parsing block range" block=01DKW3KJNF6C5MSTPRD4BSGZ8S err="read meta.json: open /appl/thanos/data-cache/01DKW3KJNF6C5MSTPRD4BSGZ8S/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.618902951Z caller=bucket.go:325 msg="error parsing block range" block=01DKW4E7E5QGWCF4BH7SWCFWGW err="read meta.json: open /appl/thanos/data-cache/01DKW4E7E5QGWCF4BH7SWCFWGW/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.618973884Z caller=bucket.go:325 msg="error parsing block range" block=01DKW56NPXANNEV0CMEX6K581X err="read meta.json: open /appl/thanos/data-cache/01DKW56NPXANNEV0CMEX6K581X/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.620408223Z caller=bucket.go:325 msg="error parsing block range" block=01DKW6181JVVW9H6QAVRAN74MJ err="read meta.json: open /appl/thanos/data-cache/01DKW6181JVVW9H6QAVRAN74MJ/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.62043776Z caller=bucket.go:325 msg="error parsing block range" block=01DKW6T8EM706WXN387FF3DSG6 err="read meta.json: open /appl/thanos/data-cache/01DKW6T8EM706WXN387FF3DSG6/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.622715992Z caller=bucket.go:325 msg="error parsing block range" block=01DKW7N21EF08YXVPDWE5WXSBY err="read meta.json: open /appl/thanos/data-cache/01DKW7N21EF08YXVPDWE5WXSBY/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.625997967Z caller=bucket.go:325 msg="error parsing block range" block=01DKW8E0CG7E66CT73MZH8MHGW err="read meta.json: open /appl/thanos/data-cache/01DKW8E0CG7E66CT73MZH8MHGW/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.626773435Z caller=bucket.go:325 msg="error parsing block range" block=01DKW97F57VPNCMFXM11SF34QJ err="read meta.json: open /appl/thanos/data-cache/01DKW97F57VPNCMFXM11SF34QJ/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.626800456Z caller=bucket.go:325 msg="error parsing block range" block=01DKWA11VMW67NPPYH93EA49AR err="read meta.json: open /appl/thanos/data-cache/01DKWA11VMW67NPPYH93EA49AR/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.628235615Z caller=bucket.go:325 msg="error parsing block range" block=01DKWAT5S8EBW0P67WMT4TZ19M err="read meta.json: open /appl/thanos/data-cache/01DKWAT5S8EBW0P67WMT4TZ19M/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.628262633Z caller=bucket.go:325 msg="error parsing block range" block=01DKWBKGN2X8NRMS0A3HW4XSXR err="read meta.json: open /appl/thanos/data-cache/01DKWBKGN2X8NRMS0A3HW4XSXR/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.62967897Z caller=bucket.go:325 msg="error parsing block range" block=01DKWCCP679PYV9VJE3S3E30EK err="read meta.json: open /appl/thanos/data-cache/01DKWCCP679PYV9VJE3S3E30EK/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.629729405Z caller=bucket.go:325 msg="error parsing block range" block=01DKWD4C7VR4M0CZMGZZ1KWEVC err="read meta.json: open /appl/thanos/data-cache/01DKWD4C7VR4M0CZMGZZ1KWEVC/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.631502923Z caller=bucket.go:325 msg="error parsing block range" block=01DKWDY4ZKRQDQ5KAWKTYE1TN5 err="read meta.json: open /appl/thanos/data-cache/01DKWDY4ZKRQDQ5KAWKTYE1TN5/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.631536558Z caller=bucket.go:325 msg="error parsing block range" block=01DKWEQ56MF8SX4F2QD8W4PTEJ err="read meta.json: open /appl/thanos/data-cache/01DKWEQ56MF8SX4F2QD8W4PTEJ/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.633667658Z caller=bucket.go:325 msg="error parsing block range" block=01DKWFH0HJ81Y22YM8K78F0MC5 err="read meta.json: open /appl/thanos/data-cache/01DKWFH0HJ81Y22YM8K78F0MC5/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.63370579Z caller=bucket.go:325 msg="error parsing block range" block=01DKWG9DPNKD2SAN3R07XZZJDV err="read meta.json: open /appl/thanos/data-cache/01DKWG9DPNKD2SAN3R07XZZJDV/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.637667472Z caller=bucket.go:325 msg="error parsing block range" block=01DKWH42KAQBFTWDMN531PHX36 err="read meta.json: open /appl/thanos/data-cache/01DKWH42KAQBFTWDMN531PHX36/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.637716685Z caller=bucket.go:325 msg="error parsing block range" block=01DKWHZ4YQYP67A0CAWP8FTEN7 err="read meta.json: open /appl/thanos/data-cache/01DKWHZ4YQYP67A0CAWP8FTEN7/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.638883728Z caller=bucket.go:325 msg="error parsing block range" block=01DKWJTWZRQMFHBP243E0N7DEJ err="read meta.json: open /appl/thanos/data-cache/01DKWJTWZRQMFHBP243E0N7DEJ/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.644267818Z caller=bucket.go:325 msg="error parsing block range" block=01DKWMDCF5JJH41N7DWF5YNHMC err="read meta.json: open /appl/thanos/data-cache/01DKWMDCF5JJH41N7DWF5YNHMC/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.644314139Z caller=bucket.go:325 msg="error parsing block range" block=01DKWN6SFNBCMHCZ172YF4P11P err="read meta.json: open /appl/thanos/data-cache/01DKWN6SFNBCMHCZ172YF4P11P/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.647090016Z caller=bucket.go:325 msg="error parsing block range" block=01DKWNZR9KYA4M8MZ1V99X3YB7 err="read meta.json: open /appl/thanos/data-cache/01DKWNZR9KYA4M8MZ1V99X3YB7/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.647137734Z caller=bucket.go:325 msg="error parsing block range" block=01DKWPSDRF734C4BG069MXE97V err="read meta.json: open /appl/thanos/data-cache/01DKWPSDRF734C4BG069MXE97V/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.647556818Z caller=bucket.go:325 msg="error parsing block range" block=01DKWQK8BP3652876G14BPZDT7 err="read meta.json: open /appl/thanos/data-cache/01DKWQK8BP3652876G14BPZDT7/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.647588262Z caller=bucket.go:325 msg="error parsing block range" block=01DKWRBNRDSENA8JBGTVXZWMFS err="read meta.json: open /appl/thanos/data-cache/01DKWRBNRDSENA8JBGTVXZWMFS/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.648839926Z caller=bucket.go:325 msg="error parsing block range" block=01DKWS41V3BZSAWERTW35EW7X3 err="read meta.json: open /appl/thanos/data-cache/01DKWS41V3BZSAWERTW35EW7X3/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.648880977Z caller=bucket.go:325 msg="error parsing block range" block=01DKWTKTJPNKTC591D98BF55PQ err="read meta.json: open /appl/thanos/data-cache/01DKWTKTJPNKTC591D98BF55PQ/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.64891254Z caller=bucket.go:325 msg="error parsing block range" block=01DKWVBER27PNTDGQ93PQJ1TBB err="read meta.json: open /appl/thanos/data-cache/01DKWVBER27PNTDGQ93PQJ1TBB/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.649270332Z caller=bucket.go:325 msg="error parsing block range" block=01DKWW48DEPE75JH5FX9SKZ3MW err="read meta.json: open /appl/thanos/data-cache/01DKWW48DEPE75JH5FX9SKZ3MW/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.651908257Z caller=bucket.go:325 msg="error parsing block range" block=01DKWWX5T37068WERGJ8T5DHZ9 err="read meta.json: open /appl/thanos/data-cache/01DKWWX5T37068WERGJ8T5DHZ9/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.652890668Z caller=bucket.go:325 msg="error parsing block range" block=01DKWXQ1Y2BJ33ETSXP3NSGPF9 err="read meta.json: open /appl/thanos/data-cache/01DKWXQ1Y2BJ33ETSXP3NSGPF9/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.652934816Z caller=bucket.go:325 msg="error parsing block range" block=01DKWYEX7527S9D7C22MDYB99W err="read meta.json: open /appl/thanos/data-cache/01DKWYEX7527S9D7C22MDYB99W/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.654246285Z caller=bucket.go:325 msg="error parsing block range" block=01DKWZ740QB25EMS2GS3RVFPVA err="read meta.json: open /appl/thanos/data-cache/01DKWZ740QB25EMS2GS3RVFPVA/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.654281239Z caller=bucket.go:325 msg="error parsing block range" block=01DKWZYR36D28VCRPZB6R2F4ZS err="read meta.json: open /appl/thanos/data-cache/01DKWZYR36D28VCRPZB6R2F4ZS/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.654741348Z caller=bucket.go:325 msg="error parsing block range" block=01DKX0PTHJTKS9FT7CGSTEECZP err="read meta.json: open /appl/thanos/data-cache/01DKX0PTHJTKS9FT7CGSTEECZP/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.654775989Z caller=bucket.go:325 msg="error parsing block range" block=01DKX1GPGMK05FH6X8PKJR6MDC err="read meta.json: open /appl/thanos/data-cache/01DKX1GPGMK05FH6X8PKJR6MDC/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.656274794Z caller=bucket.go:325 msg="error parsing block range" block=01DKX29VWKQFWTK0DT11AF1Y3Y err="read meta.json: open /appl/thanos/data-cache/01DKX29VWKQFWTK0DT11AF1Y3Y/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.662909148Z caller=bucket.go:325 msg="error parsing block range" block=01DKX3T9CDAJYJE4F7MNNEQB7K err="read meta.json: open /appl/thanos/data-cache/01DKX3T9CDAJYJE4F7MNNEQB7K/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.662967223Z caller=bucket.go:325 msg="error parsing block range" block=01DKX4KAH53E00FBXMN20M8TRV err="read meta.json: open /appl/thanos/data-cache/01DKX4KAH53E00FBXMN20M8TRV/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.664456651Z caller=bucket.go:325 msg="error parsing block range" block=01DKX5D4VQJP86K6GZQ59SB883 err="read meta.json: open /appl/thanos/data-cache/01DKX5D4VQJP86K6GZQ59SB883/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.664497005Z caller=bucket.go:325 msg="error parsing block range" block=01DKX65JY49W2227YAM3ETPSVB err="read meta.json: open /appl/thanos/data-cache/01DKX65JY49W2227YAM3ETPSVB/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.665737909Z caller=bucket.go:325 msg="error parsing block range" block=01DKX6YJHYKJAM9KNWWV1PG10Y err="read meta.json: open /appl/thanos/data-cache/01DKX6YJHYKJAM9KNWWV1PG10Y/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.665790354Z caller=bucket.go:325 msg="error parsing block range" block=01DKX7NK7VBWTP461Z2MREK6RA err="read meta.json: open /appl/thanos/data-cache/01DKX7NK7VBWTP461Z2MREK6RA/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.667483023Z caller=bucket.go:325 msg="error parsing block range" block=01DKX8H4SRSHV2CGDD92MVYGK3 err="read meta.json: open /appl/thanos/data-cache/01DKX8H4SRSHV2CGDD92MVYGK3/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.667571432Z caller=bucket.go:325 msg="error parsing block range" block=01DKX97Z7JQAF6TWM60FH4YQRA err="read meta.json: open /appl/thanos/data-cache/01DKX97Z7JQAF6TWM60FH4YQRA/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.6774169Z caller=bucket.go:325 msg="error parsing block range" block=01DKXATRSXTFZNA05WH09SQKGF err="read meta.json: open /appl/thanos/data-cache/01DKXATRSXTFZNA05WH09SQKGF/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.6784887Z caller=bucket.go:325 msg="error parsing block range" block=01DKXBP3MG04T0Q3VHYKFE1A4K err="read meta.json: open /appl/thanos/data-cache/01DKXBP3MG04T0Q3VHYKFE1A4K/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.678567294Z caller=bucket.go:325 msg="error parsing block range" block=01DKXCCXSFJKJJZYKFJQ856C2Z err="read meta.json: open /appl/thanos/data-cache/01DKXCCXSFJKJJZYKFJQ856C2Z/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.6800205Z caller=bucket.go:325 msg="error parsing block range" block=01DKXD8JHY0D6G85QDS32E0SJC err="read meta.json: open /appl/thanos/data-cache/01DKXD8JHY0D6G85QDS32E0SJC/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.680094276Z caller=bucket.go:325 msg="error parsing block range" block=01DKXE0858KVS3S99C0ZPK2TTF err="read meta.json: open /appl/thanos/data-cache/01DKXE0858KVS3S99C0ZPK2TTF/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.681733611Z caller=bucket.go:325 msg="error parsing block range" block=01DKXEWKHWE3EYHRWNWKFEJVFD err="read meta.json: open /appl/thanos/data-cache/01DKXEWKHWE3EYHRWNWKFEJVFD/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.681820288Z caller=bucket.go:325 msg="error parsing block range" block=01DKXFKQ2X2FDRF4GF9AEPMVE1 err="read meta.json: open /appl/thanos/data-cache/01DKXFKQ2X2FDRF4GF9AEPMVE1/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.683000294Z caller=bucket.go:325 msg="error parsing block range" block=01DKXG0GB0VK3SA10Y3SVPKX9Q err="read meta.json: open /appl/thanos/data-cache/01DKXG0GB0VK3SA10Y3SVPKX9Q/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.683101741Z caller=bucket.go:325 msg="error parsing block range" block=01DKXGG5SSD0VNK42ZXMV18R2Q err="read meta.json: open /appl/thanos/data-cache/01DKXGG5SSD0VNK42ZXMV18R2Q/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.685522779Z caller=bucket.go:325 msg="error parsing block range" block=01DKXH8ETY19VS5CVMW4M35Y47 err="read meta.json: open /appl/thanos/data-cache/01DKXH8ETY19VS5CVMW4M35Y47/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.687184281Z caller=bucket.go:325 msg="error parsing block range" block=01DKXJ496KH0PKWXPCRJQMVB95 err="read meta.json: open /appl/thanos/data-cache/01DKXJ496KH0PKWXPCRJQMVB95/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.687248467Z caller=bucket.go:325 msg="error parsing block range" block=01DKXJV8G3G0T2TZDQS9D90RWF err="read meta.json: open /appl/thanos/data-cache/01DKXJV8G3G0T2TZDQS9D90RWF/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.687490253Z caller=bucket.go:325 msg="error parsing block range" block=01DKXK8HG3BBBC5A78762G28V7 err="read meta.json: open /appl/thanos/data-cache/01DKXK8HG3BBBC5A78762G28V7/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.687526483Z caller=bucket.go:325 msg="error parsing block range" block=01DKXKQTF87NP5D7CV6FVCGBG5 err="read meta.json: open /appl/thanos/data-cache/01DKXKQTF87NP5D7CV6FVCGBG5/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.687551313Z caller=bucket.go:325 msg="error parsing block range" block=01DKXME248RYMBZYHSR72EWFJY err="read meta.json: open /appl/thanos/data-cache/01DKXME248RYMBZYHSR72EWFJY/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.688002804Z caller=bucket.go:325 msg="error parsing block range" block=01DKXNAQFVVJZ975CJS8DQJCVY err="read meta.json: open /appl/thanos/data-cache/01DKXNAQFVVJZ975CJS8DQJCVY/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.688040545Z caller=bucket.go:325 msg="error parsing block range" block=01DKXP14JD4P862G7Z33G7S552 err="read meta.json: open /appl/thanos/data-cache/01DKXP14JD4P862G7Z33G7S552/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.689238619Z caller=bucket.go:325 msg="error parsing block range" block=01DKXPX990NN6YVTA0YAR5BWDC err="read meta.json: open /appl/thanos/data-cache/01DKXPX990NN6YVTA0YAR5BWDC/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.68928131Z caller=bucket.go:325 msg="error parsing block range" block=01DKXQKKC38ZDRKMWJ6Z8E4R5K err="read meta.json: open /appl/thanos/data-cache/01DKXQKKC38ZDRKMWJ6Z8E4R5K/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.692860232Z caller=bucket.go:325 msg="error parsing block range" block=01DKXRFZWVHH6AT74XBFV9FF24 err="read meta.json: open /appl/thanos/data-cache/01DKXRFZWVHH6AT74XBFV9FF24/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.692919257Z caller=bucket.go:325 msg="error parsing block range" block=01DKXS736C8A2X4K1EV4GKG2WQ err="read meta.json: open /appl/thanos/data-cache/01DKXS736C8A2X4K1EV4GKG2WQ/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.694294313Z caller=bucket.go:325 msg="error parsing block range" block=01DKXSKQMY1GCGZK9BCF00VHTQ err="read meta.json: open /appl/thanos/data-cache/01DKXSKQMY1GCGZK9BCF00VHTQ/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.694348663Z caller=bucket.go:325 msg="error parsing block range" block=01DKXT3VW53TK5Y399NXR7WVZJ err="read meta.json: open /appl/thanos/data-cache/01DKXT3VW53TK5Y399NXR7WVZJ/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.694399894Z caller=bucket.go:325 msg="error parsing block range" block=01DKXTSR486MKAFX7KSSDCARDJ err="read meta.json: open /appl/thanos/data-cache/01DKXTSR486MKAFX7KSSDCARDJ/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.695490936Z caller=bucket.go:325 msg="error parsing block range" block=01DKXVP4DBSGSDTTT7PW77K836 err="read meta.json: open /appl/thanos/data-cache/01DKXVP4DBSGSDTTT7PW77K836/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.69554648Z caller=bucket.go:325 msg="error parsing block range" block=01DKXWD00FR17MKB8813X1WEWQ err="read meta.json: open /appl/thanos/data-cache/01DKXWD00FR17MKB8813X1WEWQ/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.695583925Z caller=bucket.go:325 msg="error parsing block range" block=01DKXWPNW2PSM79076BP1VFVBV err="read meta.json: open /appl/thanos/data-cache/01DKXWPNW2PSM79076BP1VFVBV/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.69633682Z caller=bucket.go:325 msg="error parsing block range" block=01DKXX8Q9EY4YZ2Q0W9TRXNNH0 err="read meta.json: open /appl/thanos/data-cache/01DKXX8Q9EY4YZ2Q0W9TRXNNH0/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.696397194Z caller=bucket.go:325 msg="error parsing block range" block=01DKXY08X7ZP15FPPHV5KKYBG9 err="read meta.json: open /appl/thanos/data-cache/01DKXY08X7ZP15FPPHV5KKYBG9/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.697324163Z caller=bucket.go:325 msg="error parsing block range" block=01DKXYDMYPFTJSZQWD0S6Z06PT err="read meta.json: open /appl/thanos/data-cache/01DKXYDMYPFTJSZQWD0S6Z06PT/meta.json: no such file or directory"
Sep  4 13:04:03 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:03.759713191Z caller=bucket.go:325 msg="error parsing block range" block=01DKXYX6PHKZ94564H3BKHSZRT err="download meta.json: get file: The specified key does not exist."
Sep  4 13:04:04 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:04.535192631Z caller=storeset.go:223 component=storeset msg="updating healthy stores" externalLabelOccurrencesInStores="map[string]int{\"\":1, \"{prometheus_server=\\\"p1-prom-mlsu001\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu002\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu101\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu102\\\"}\":1}"
Sep  4 13:04:05 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:05.721605303Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 filtered out"
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.36082669Z caller=proxy.go:265 msg="store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 queried;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.361426887Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=458 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1556690400000-1567584000000 Resolution: 0"
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.363022904Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=388 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1556776800000-1567584000000 Resolution: 0"
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.364267779Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=232 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1563840000000-1567591200000 Resolution: 0"
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.364869119Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=233 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1563840000000-1567591200000 Resolution: 0"
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.368839678Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:1311 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:3159031 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:161251}" err=null
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.607700923Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567584000000 queried;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.608107262Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=458 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1556690400000-1567584000000 Resolution: 0"
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.6095158Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=388 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1556776800000-1567584000000 Resolution: 0"
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.610668258Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=232 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1563840000000-1567591200000 Resolution: 0"
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.611323305Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=233 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1563840000000-1567591200000 Resolution: 0"
Sep  4 13:04:07 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:07.615449801Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:1311 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:3523156 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:137761}" err=null
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.542597492Z caller=storeset.go:223 component=storeset msg="updating healthy stores" externalLabelOccurrencesInStores="map[string]int{\"\":1, \"{prometheus_server=\\\"p1-prom-mlsu001\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu002\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu101\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu102\\\"}\":1}"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.934740449Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567591200000 queried;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.935459551Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595160000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.935571154Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595160000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.935620727Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595160000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.935666245Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595160000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.967809946Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567591200000 queried"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.968355993Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595160000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.968481427Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595160000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.968548639Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595160000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.968588354Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595160000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.978423886Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:6 postingsTouched:4 postingsTouchedSizeSum:2480 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:2 seriesTouchedSizeSum:144 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:2 chunksTouchedSizeSum:909 chunksFetched:2 chunksFetchedSizeSum:32000 chunksFetchCount:2 chunksFetchDurationSum:80589161 getAllDuration:42625891 mergedSeriesCount:2 mergedChunksCount:2 mergeDuration:42707}" err=null
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.981945814Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567591200000 queried"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.982554651Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595055000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.982654482Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595055000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.982699082Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595055000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:09 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:09.982747247Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595055000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.016340826Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:6 postingsTouched:4 postingsTouchedSizeSum:2480 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:2 seriesTouchedSizeSum:144 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:2 chunksTouchedSizeSum:261 chunksFetched:2 chunksFetchedSizeSum:32000 chunksFetchCount:2 chunksFetchDurationSum:84542553 getAllDuration:47616219 mergedSeriesCount:2 mergedChunksCount:2 mergeDuration:49307}" err=null
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.065048993Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567591200000 queried;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.065805146Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595070000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.065981684Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595070000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.0660504Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595070000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.066090992Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595070000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.098948841Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:6 postingsTouched:40 postingsTouchedSizeSum:9256 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:34 seriesTouchedSizeSum:2856 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:34 chunksTouchedSizeSum:4440 chunksFetched:34 chunksFetchedSizeSum:48915 chunksFetchCount:2 chunksFetchDurationSum:60042460 getAllDuration:32608881 mergedSeriesCount:34 mergedChunksCount:34 mergeDuration:185313}" err=null
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.219408921Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:6 postingsTouched:40 postingsTouchedSizeSum:9864 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 seriesTouched:34 seriesTouchedSizeSum:2856 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:34 chunksTouchedSizeSum:4480 chunksFetched:34 chunksFetchedSizeSum:49248 chunksFetchCount:2 chunksFetchDurationSum:281613520 getAllDuration:236330596 mergedSeriesCount:34 mergedChunksCount:34 mergeDuration:188678}" err=null
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.505663312Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567591200000 queried;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.506087566Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595060000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.506220789Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595060000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.506327288Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595060000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.506377206Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595060000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.513850756Z caller=proxy.go:265 msg="store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567591200000 queried;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.514514925Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583900000 maxt=1567595200000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.514605393Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583900000 maxt=1567595200000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.514663865Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583900000 maxt=1567595200000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.514750824Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583900000 maxt=1567595200000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.590729551Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:6 postingsTouched:8 postingsTouchedSizeSum:92328 postingsToFetch:0 postingsFetched:2 postingsFetchedSizeSum:33160 postingsFetchCount:2 postingsFetchDurationSum:73810116 seriesTouched:12 seriesTouchedSizeSum:1020 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:12 chunksTouchedSizeSum:2469 chunksFetched:12 chunksFetchedSizeSum:38273 chunksFetchCount:2 chunksFetchDurationSum:159309916 getAllDuration:84120099 mergedSeriesCount:12 mergedChunksCount:12 mergeDuration:146592}" err=null
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.610679696Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:6 postingsTouched:8 postingsTouchedSizeSum:92328 postingsToFetch:0 postingsFetched:2 postingsFetchedSizeSum:33160 postingsFetchCount:2 postingsFetchDurationSum:138558400 seriesTouched:12 seriesTouchedSizeSum:1020 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:12 chunksTouchedSizeSum:1578 chunksFetched:12 chunksFetchedSizeSum:37314 chunksFetchCount:2 chunksFetchDurationSum:166602806 getAllDuration:95761588 mergedSeriesCount:12 mergedChunksCount:12 mergeDuration:94376}" err=null
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.656010866Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567591200000 queried"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.656460737Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595080000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.656547587Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595080000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.656595884Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595080000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.656693012Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595080000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.696182006Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567591200000 queried;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.696698902Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595060000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.696814577Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=1 MaximumResolution=0 mint=1567583940000 maxt=1567595060000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1567576800000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.696866799Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595060000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.696920254Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=2 MaximumResolution=0 mint=1567583940000 maxt=1567595060000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1567576800000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.747435031Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567591200000 queried;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.748106809Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=458 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu002\"}" spans="Range: 1556690400000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.749503802Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=388 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu001\"}" spans="Range: 1556776800000-1567584000000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.750819581Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=232 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu101\"}" spans="Range: 1563840000000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.751536348Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=233 MaximumResolution=0 mint=-62167219200000 maxt=253402300799000 lset="{prometheus_server=\"p1-prom-mlsu102\"}" spans="Range: 1563840000000-1567591200000 Resolution: 0"
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.790720575Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:6 postingsTouched:8 postingsTouchedSizeSum:33096 postingsToFetch:0 postingsFetched:2 postingsFetchedSizeSum:8808 postingsFetchCount:2 postingsFetchDurationSum:97910953 seriesTouched:10 seriesTouchedSizeSum:850 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:10 chunksTouchedSizeSum:1334 chunksFetched:10 chunksFetchedSizeSum:559184 chunksFetchCount:2 chunksFetchDurationSum:148653400 getAllDuration:93655875 mergedSeriesCount:10 mergedChunksCount:10 mergeDuration:80011}" err=null
Sep  4 13:04:10 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:10.901238208Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:6 postingsTouched:14 postingsTouchedSizeSum:260248 postingsToFetch:0 postingsFetched:2 postingsFetchedSizeSum:62168 postingsFetchCount:2 postingsFetchDurationSum:145268553 seriesTouched:6 seriesTouchedSizeSum:510 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:6 chunksTouchedSizeSum:783 chunksFetched:6 chunksFetchedSizeSum:34126 chunksFetchCount:2 chunksFetchDurationSum:289129437 getAllDuration:244399430 mergedSeriesCount:6 mergedChunksCount:6 mergeDuration:80737}" err=null
Sep  4 13:04:11 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:11.432347989Z caller=proxy.go:265 msg="store Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried;store Addr: localhost:7023 LabelSets:  Mint: 1556690400000 Maxt: 1567591200000 filtered out;store Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu001:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu001\" ] Mint: 1567144800000 Maxt: 9223372036854775807 queried;store Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807 queried"
Sep  4 13:04:14 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:14.534581499Z caller=storeset.go:223 component=storeset msg="updating healthy stores" externalLabelOccurrencesInStores="map[string]int{\"\":1, \"{prometheus_server=\\\"p1-prom-mlsu001\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu002\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu101\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu102\\\"}\":1}"
Sep  4 13:04:17 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:17.743340617Z caller=proxy.go:409 err="failed to receive any data from Addr: p1-prom-mlsu002:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu002\" ] Mint: 1567144800000 Maxt: 9223372036854775807: context canceled" msg="returning partial response"
Sep  4 13:04:17 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:17.743431999Z caller=proxy.go:409 err="failed to receive any data from Addr: p1-prom-mlsu101:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu101\" ] Mint: 1567152000000 Maxt: 9223372036854775807: context canceled" msg="returning partial response"
Sep  4 13:04:17 p1-thanos-alsu001 thanos: level=warn ts=2019-09-04T11:04:17.743451208Z caller=proxy.go:409 err="failed to receive any data from Addr: p1-prom-mlsu102:7020 LabelSets: [name:\"prometheus_server\" value:\"p1-prom-mlsu102\" ] Mint: 1567152000000 Maxt: 9223372036854775807: context canceled" msg="returning partial response"
Sep  4 13:04:19 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:19.534640935Z caller=storeset.go:223 component=storeset msg="updating healthy stores" externalLabelOccurrencesInStores="map[string]int{\"\":1, \"{prometheus_server=\\\"p1-prom-mlsu001\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu002\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu101\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu102\\\"}\":1}"
Sep  4 13:04:24 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:24.534704177Z caller=storeset.go:223 component=storeset msg="updating healthy stores" externalLabelOccurrencesInStores="map[string]int{\"\":1, \"{prometheus_server=\\\"p1-prom-mlsu001\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu002\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu101\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu102\\\"}\":1}"
Sep  4 13:04:29 p1-thanos-alsu001 thanos: level=debug ts=2019-09-04T11:04:29.53792269Z caller=storeset.go:223 component=storeset msg="updating healthy stores" externalLabelOccurrencesInStores="map[string]int{\"\":1, \"{prometheus_server=\\\"p1-prom-mlsu001\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu002\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu101\\\"}\":1, \"{prometheus_server=\\\"p1-prom-mlsu102\\\"}\":1}"

Anything else we need to know

If this is of relevance: We do not downsample the data and we keep it for 7 months.

Thanos Compact Systemd startup command (on same server):

/appl/thanos/bin/thanos compact --data-dir=/appl/thanos/data-compactor --http-address=0.0.0.0:7008 --log.level=debug --objstore.config-file=/appl/thanos/.s3.yml --retention.resolution-raw=210d --wait

When querying

rate(thanos_objstore_bucket_operations_total{instance=~"p1-thanos-alsu001.*7007"}[30s])

I've around ~700/s, nothing near 2000/s.

When doing a tcpdump (sadly not at 13:04) I see lots of TCP ACK with NOOP & Timestamp Options, but no content really. Not sure, if those connections are causing the problem...?

Have you heard/encountered something familiar?
I'm not really sure how to further investigate. If you need further data, please tell me so and I'll see to it.

Environment:

  • OS (e.g. from /etc/os-release): RHEL7.6
  • Kernel (e.g. uname -a): Linux p1-thanos-alsu001 3.10.0-957.1.3.el7.x86_64 Initial structure and block shipper #1 SMP Thu Nov 15 17:36:42 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Others:
@GiedriusS
Copy link
Member

GiedriusS commented Sep 12, 2019 •
edited
Loading

Hi there, it seems like your index cache is too small. Could you check the index cache metrics and see if there are lots of evictions? If yes, then you might need to increase the size of it. Unfortunately, our current simple LRU based index cache doesn't work well under lots of pressure since it doesn't track what items are being constantly added and removed.

@muryoutaisuu
Copy link
Contributor Author

Hi, thanks for looking into it. In fact I had a lot of those message. I have now doubled both values and set:

--index-cache-size=500MB --chunk-pool-size=4GB

I once read in slack:

sum it up (how large index-cache is for biggest blocks for 2w and assume how many you want to cache) and well -> you have the value

But I don't really know where I can see the size of the index-cache for biggest blocks. So I'm just going with the try and fail method.

@GiedriusS
Copy link
Member

That seems like a pretty good rule of thumb. They are talking about the sizes of index.cache.json files that are in the remote object storage.

Did increasing its size help you with this problem?

@muryoutaisuu
Copy link
Contributor Author

Hi, yes it seems setting higher values for caching did indeed help with this issue.
Thank you very much for helping. If you're ever in Switzerland, I'll invite you to some drinks 🍻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component: store question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@GiedriusS @muryoutaisuu