[Filebeat][httpjson] Add cursor support

[andrewkroh]

The httpjson input should be enhanced to allow for persisting data returned in a response and then using that data in the next request.

Requirements

• The data must be retained in persistent storage (i.e. it must survive process restarts).
• Must be able to specify what data to retain (e.g. time of request, response header, response body field).
• Must be able to specify how the retained data is used in the next request.
  • As part of the request URL.
  • As a HTTP request header.
  • As part of an HTTP POST body.

For persistence the httpjson input should be update to implement the input cursor API that will be available soon in Filebeat.

There were some discussions around how to implement the two other requirements in a generic manner. The general consensus was to establish a data model for what should be persisted (response headers, fields from the body, pagination parameters) and have the configuration specify how to transform the response data into the data model. Then have a similar configuration to specify how to build the request from persisted data model. For example this might mean allowing the user to specify a bit of javascript to grab values from the response, transform them, and persist them. And similarly allowing some configuration to specify how the persisted data is used in the next request (set headers, set body parameters, etc).

Then once we have this more generic means of configuration how the input creates requests and does pagination then we can deprecate and retire some of the existing configuration options to make usage simpler and more clear.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)