Update docs with notes on deprecation logs throttling/limiting behavior

[cjcenizal]

Per a convo with @jakelandis, some (but not all) deprecation warnings are throttled or limited to being stored in the log to a single instance. As of 7.7, you can observe this behavior by enabling deprecation logging and executing this request multiple times:

PUT _template/test?include_type_name
{
  "index_patterns": [
    "my_index"
  ],
  "mappings": {
    "my_custom_type": {
      "properties": {
        "name": {
          "type": "geo_shape",
          "tree": "quadtree"
        }
      }
    }
  }
}

Both the include_type_name query param the tree geo-shape parameter have been deprecated in 7.0, however the deprecation message for the former is logged once while the latter is logged per request. This might be confusing for users unless documented somewhere. I think it would be discoverable in the docs linked to above.

{"type": "deprecation", "timestamp": "2020-01-28T13:33:37,156-08:00", "level": "WARN", "component": "o.e.d.r.a.a.i.RestPutIndexTemplateAction", "cluster.name": "elasticsearch", "node.name": "CJs-MacBook-Pro.local", "message": "[types removal] Specifying include_type_name in put index template requests is deprecated. The parameter will be removed in the next major version.", "cluster.uuid": "RHW6A9haRh2iIyhZ4R6gCA", "node.id": "Q8QttK8pRcaUaVwEyQRt_Q"  }
{"type": "deprecation", "timestamp": "2020-01-28T13:36:22,164-08:00", "level": "WARN", "component": "o.e.d.i.m.LegacyGeoShapeFieldMapper", "cluster.name": "elasticsearch", "node.name": "CJs-MacBook-Pro.local", "message": "Field parameter [tree] is deprecated and will be removed in a future version.", "cluster.uuid": "RHW6A9haRh2iIyhZ4R6gCA", "node.id": "Q8QttK8pRcaUaVwEyQRt_Q"  }
{"type": "deprecation", "timestamp": "2020-01-28T13:36:28,894-08:00", "level": "WARN", "component": "o.e.d.i.m.LegacyGeoShapeFieldMapper", "cluster.name": "elasticsearch", "node.name": "CJs-MacBook-Pro.local", "message": "Field parameter [tree] is deprecated and will be removed in a future version.", "cluster.uuid": "RHW6A9haRh2iIyhZ4R6gCA", "node.id": "Q8QttK8pRcaUaVwEyQRt_Q"  }
{"type": "deprecation", "timestamp": "2020-01-28T13:36:41,879-08:00", "level": "WARN", "component": "o.e.d.i.m.LegacyGeoShapeFieldMapper", "cluster.name": "elasticsearch", "node.name": "CJs-MacBook-Pro.local", "message": "Field parameter [tree] is deprecated and will be removed in a future version.", "cluster.uuid": "RHW6A9haRh2iIyhZ4R6gCA", "node.id": "Q8QttK8pRcaUaVwEyQRt_Q"  }

[jasontedor]

Note that we still include the warning header on every response, so that these logs would still be exposed to clients where they could be logged client side, and included in Console.

What do you suggest that we add to the docs here that would be helpful to users, without going TMI?

[cjcenizal]

@jasontedor I was thinking of something like this:

Note that some deprecations will only result in a single log message despite repeated requests while others will log a message for every request.

I think this will help users understand the behavior they're seeing in the log. To provide some context, the Kibana Upgrade Assistant advises users to turn on deprecation logging and check the logs to see if their applications use any deprecated functionality. When I was testing this out, I was surprised by the behavior I was seeing in the logs and couldn't find any explanation for it in the docs.

I can also imagine someone reading the logs and skipping to the latest ones to verify that they've fixed their application, not realizing that some deprecation messages don't repeat. This might require a different solution, since the change I'm suggesting wouldn't address this scenario.

[jasontedor]

Note that some deprecations will only result in a single log message despite repeated requests while others will log a message for every request.

I think this will help users understand the behavior they're seeing in the log.

And I think that gets to the core of my concern, I'm not convinced that it does help them. A user can not distinguish between a case where the log messages are de-duplicated from one where whatever application triggered the deprecated behavior has been fixed. The proposed docs only surface that we have a UX problem here, but I don't think they address it.

I can also imagine someone reading the logs and skipping to the latest ones to verify that they've fixed their application, not realizing that some deprecation messages don't repeat. This might require a different solution, since the change I'm suggesting wouldn't address this scenario.

Yeah, in another context (#46106) we've considered writing the deprecation logs to an index, to make it easier for the migration assistant (#46107) to expose the deprecation logs to a user. In that context, we have an opportunity to index all deprecation messages, so that this concern goes away.

[pgomulka]

using a different x-opaque-id for an updated application would be a possible work around.
deprecation logs are throttled per key+x-opaque-id. That means two requests with two different x-opaque-id header would log twice to deprecation logs.
#44587

[cjcenizal]

@jasontedor Storing deprecation messages in an index sounds like it would be a great solution to this problem. Do you want to close this issue if that's where we're headed?

[jasontedor]

Sure, thanks @cjcenizal.