Kibana logs [warning][collector-set][plugins][usageCollection] StatusCodeError: Request Timeout after 30000ms after being idle for a few hours

[bhavyarm]

Kibana version: 7.11.0 BC4

Elasticsearch version: 7.11.0 BC4

Server OS version: darwin_x86_64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): from staging

Describe the bug: I had my kibana running in my local overnight and noticed these logs the next morning.
Please note I don't have rollups, monitoring or fleet set up in my local (that could be a red herring)

log   [04:30:02.052] [warning][collector-set][plugins][usageCollection] StatusCodeError: Request Timeout after 30000ms
    at /Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:397:9
    at Timeout.<anonymous> (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:429:7)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7) {
  status: undefined,
  displayName: 'RequestTimeout',
  body: undefined
}
  log   [04:30:02.105] [warning][collector-set][plugins][usageCollection] Unable to fetch data from monitoringTelemetry collector
  log   [04:30:02.106] [warning][collector-set][plugins][usageCollection] StatusCodeError: Request Timeout after 30000ms
    at /Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:397:9
    at Timeout.<anonymous> (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:429:7)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7) {
  status: undefined,
  displayName: 'RequestTimeout',
  body: undefined
}
  log   [04:30:02.107] [warning][collector-set][plugins][usageCollection] Unable to fetch data from monitoringTelemetry collector
  log   [04:30:02.107] [error][index][plugins][security][session] Failed to retrieve session value: Request Timeout after 30000ms
  log   [04:30:02.108] [error][authentication][plugins][security] StatusCodeError: Request Timeout after 30000ms
    at /Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:397:9
    at Timeout.<anonymous> (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:429:7)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7) {
  status: undefined,
  displayName: 'RequestTimeout',
  body: undefined
}
  log   [04:30:02.117] [warning][collector-set][plugins][usageCollection] StatusCodeError: Request Timeout after 30000ms
    at /Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:397:9
    at Timeout.<anonymous> (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:429:7)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7) {
  status: undefined,
  displayName: 'RequestTimeout',
  body: undefined
}
  log   [04:30:02.118] [warning][collector-set][plugins][usageCollection] Unable to fetch data from rollups collector
  log   [04:30:02.118] [warning][collector-set][plugins][usageCollection] TimeoutError: Request timed out
    at ClientRequest.onTimeout (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/@elastic/elasticsearch/lib/Connection.js:109:16)
    at ClientRequest.emit (events.js:315:20)
    at Socket.emitRequestTimeout (_http_client.js:784:9)
    at Object.onceWrapper (events.js:421:28)
    at Socket.emit (events.js:327:22)
    at Socket._onTimeout (net.js:483:8)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7) {
  meta: {
    body: null,
    statusCode: null,
    headers: null,
    meta: {
      context: null,
      request: [Object],
      name: 'elasticsearch-js',
      connection: [Object],
      attempts: 0,
      aborted: false
    }
  },
  isBoom: true,
  isServer: true,
  data: null,
  output: {
    statusCode: 503,
    payload: {
      statusCode: 503,
      error: 'Service Unavailable',
      message: 'Request timed out'
    },
    headers: {}
  },
  [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/esUnavailable'
}
  log   [04:30:02.165] [warning][collector-set][plugins][usageCollection] Unable to fetch data from fleet collector
 error  [22:02:10.900]  Error: Internal Server Error
    at HapiResponseAdapter.toError (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/src/core/server/http/router/response_adapter.js:121:19)
    at HapiResponseAdapter.toHapiResponse (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/src/core/server/http/router/response_adapter.js:75:19)
    at HapiResponseAdapter.handle (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/src/core/server/http/router/response_adapter.js:70:17)
    at Object.interceptAuth [as authenticate] (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/src/core/server/http/lifecycle/auth.js:93:36)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at runNextTicks (internal/process/task_queues.js:62:3)
    at listOnTimeout (internal/timers.js:523:9)
    at processTimers (internal/timers.js:497:7)
    at module.exports.internals.Manager.execute (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/@hapi/hapi/lib/toolkit.js:45:28)
    at module.exports.internals.Auth._authenticate (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/@hapi/hapi/lib/auth.js:256:30)
    at Request._lifecycle (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/@hapi/hapi/lib/request.js:312:32)
    at Request._execute (/Users/bhavyarajumandya/Desktop/BC4_7.11.0/kibana-7.11.0-darwin-x86_64/node_modules/@hapi/hapi/lib/request.js:221:9)
  log   [06:59:37.655] [error][plugins][taskManager] [Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity
  log   [08:48:47.772] [error][plugins][taskManager] [Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity

[elasticmachine]

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

 statusCode: 503,
    payload: {
      statusCode: 503,
      error: 'Service Unavailable',
      message: 'Request timed out'
    },
    headers: {}
  },
  [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/esUnavailable'

From the provided logs, this is a connectivity issue with ES. I don't know the cause, but that would be a very valid reasons for the following errors.

Please note I don't have rollups, monitoring or fleet set up in my local

Were the plugins explicitly disabled? As long as they were not, having the collectors fetch the data is an expected behavior

[afharo]

I think this is a recurring issue in the collectors: whenever there's a network issue, they are very noisy. Since they don't/shouldn't really affect the user experience. I'd suggest we switch those errors to debug 😇

[afharo]

I've created #89839 to discuss if, instead of switching them to debug, we better set a specific config in those plugins' loggers to be muted when not running in dev mode. What do you think?

[LeeDr]

Why would we ever have network connectivity problems in dev mode with Elasticsearch and Kibana both running on the same machine? This makes me think there's a real problem we're sweeping under the rug.

If there's a connection problem, I'd be OK with doing a retry and not logging anything unless the retry fails. But at that point it should be an error and we should try to figure out what's causing it.

[rashmivkulkarni]

FYI : am seeing this on 7.13.0 BC3 - when I run Kibana and ES on the same host . ( Darwin)

Log messages are seen like this : log [13:16:40.084] [error][plugins][taskManager] [Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity

[afharo]

Why would we ever have network connectivity problems in dev mode with Elasticsearch and Kibana both running on the same machine?

One scenario I can think of: when the host goes to hibernation after a long period of no use (from the description, I had my kibana running in my local overnight).

If there's a connection problem, I'd be OK with doing a retry and not logging anything unless the retry fails.

That's a very good point. I recently created #97788 to avoid trying to collect any telemetry if the ES connection is down :)