Add Filebeat input httpjson

[alakahakai]

New filebeat input httpjson provides the following functions:

• Address issue #13545
• Take HTTP/HTTPS JSON input via configurable URL and API key and generate events
• Support configurable interval for repeated retrieval
• Support pagination using URL or additional fields.
• Sample configurations:

    ### Test POST with httpbin.org
     var.http_method: POST
     var.http_request_body: |-
       {
           "test": "abc",
           "test2": {
               "test21": 1234,
               "test22": "test22"
           }
       }
     var.url: "https://httpbin.org/post"

    ### Test Pagination with Elasticsearch
     var.http_method: POST
     var.json_objects_array: hits.hits
     var.pagination: |-
         {
             "enabled": true,
             "extra_body_content": {
                 "scroll": "5m"
             },
             "id_field": "_scroll_id",
             "req_field": "scroll_id",
             "url": "http://localhost:9200/_search/scroll"
         }
     var.url: http://localhost:9200/filebeat-test/_search?scroll=5m

New filebeat module misp will use the httpjson input to take threat intel information out of MISP via provided REST API, and generate threat intel attack patterns.
Other new modules can use the httpjson input as well for their REST inputs.

    var.api_key: "6ZZAsBQ4TrFxJk34IXYHPtkCn9F6CoGqBDl5FSeJ"
    var.json_objects_array: "response.Attribute"
    var.url: "http://192.168.86.31/attributes/restSearch"

[andrewkroh]

I've made some suggestions. Let me know if you have any questions or want to discuss more.

[andrewkroh]

Just a few minor changes requested. And I left a question for @urso.

[alakahakai]

jenkins, test this

[alakahakai]

Just a few minor changes requested. And I left a question for @urso.

Removed the external dependency on httpbin.org. It uses local httptest server now.

[andrewkroh]

This looks good to me. Just a question and a minor issue.

[alakahakai]

@andrewkroh Thanks for the approval. I am going to add a headers field to allow extra settings about HTTP headers.

[alakahakai]

jenkins, test this

[alakahakai]

jenkins, test this

[alakahakai]

jenkins, test this

[alakahakai]

jenkins, test this

[alakahakai]

jenkins, test this

[urso]

I think we should mark this input as experimental for now.

Question: Where is state tracked for the services this input is supposed to read from. For example Logging/Audit endpoints would require you to track some kind of ID or timestamp between queries (and restarts), so to correctly collect all data without duplicates. To some extent Pagination looks like it can be used for this use-cases, but it always seems to start from the beginning between filebeat restarts.

I'm also missing authentication settings like username/password.

Have you considered to add some support for cookies as well?

[alakahakai]

Thanks for your comments. Please see my answers inline below.

On Wed, Oct 2, 2019 at 8:15 AM Steffen Siering ***@***.***> wrote: I think we should mark this input as experimental for now. Question: Where is state tracked for the services this input is supposed to read from. For example Logging/Audit endpoints would require you to track some kind of ID or timestamp between queries (and restarts), so to correctly collect all data without duplicates. To some extent Pagination looks like it can be used for this use-cases, but it always seems to start from the beginning between filebeat restarts.

The httpjson input is not stateful across restarts, meaning that the pagination can track the IDs across continuous requests and perform proper pagination, but when filebeat is restarted, the input will start from the beginning. However, I am not sure that such a state should be stored across restarts, since many pagination scrolls only exist for a short period of time.

I'm also missing authentication settings like username/password.

User and password are not directly supported, but you can use the Header config option to add them, provided that the base64 hash needs to be computed separately before you can input it to the config.

Have you considered to add some support for cookies as well?

The user case that the input was designed for does not require cookies, so support for cookie is not planned at this point.

[urso]

The httpjson input is not stateful across restarts, meaning that the
pagination can track the IDs across continuous requests and perform proper
pagination, but when filebeat is restarted, the input will start from the
beginning. However, I am not sure that such a state should be stored across
restarts, since many pagination scrolls only exist for a short period of
time.

Adding this as input to filebeat I'd expect this input to follow logs through APIs. This is an infinite stream of events, potentially backed by some persistent storage. Without keeping state one would start from the beginning over and over again. For example Okta logs list API.

User and password are not directly supported, but you can use the Header
config option to add them, provided that the base64 hash needs to be
computed separately before you can input it to the config.

True, yet we have username and password settings for other modules/inputs and outputs. Also for consistency reasons we should not force users to create correct authentication headers.

The user case that the input was designed for does not require cookies, so
support for cookie is not planned at this point.

I see. I'm fine with that decision.

[alakahakai]

The httpjson input is not stateful across restarts, meaning that the
pagination can track the IDs across continuous requests and perform proper
pagination, but when filebeat is restarted, the input will start from the
beginning. However, I am not sure that such a state should be stored across
restarts, since many pagination scrolls only exist for a short period of
time.

Adding this as input to filebeat I'd expect this input to follow logs through APIs. This is an infinite stream of events, potentially backed by some persistent storage. Without keeping state one would start from the beginning over and over again. For example Okta logs list API.

I agree that this okta API will be a problem for this input as is, but I don't think keeping state will solve the problem. When the filebeat restarts, the scroll/state on the okta side can expire, and keeping state on the filebeat side will not help. Here the state needs to be kept and updated is the "since" parameter, hence requires understanding of the okta API. In this case, a dedicated input might be the way to go.

User and password are not directly supported, but you can use the Header
config option to add them, provided that the base64 hash needs to be
computed separately before you can input it to the config.

True, yet we have username and password settings for other modules/inputs and outputs. Also for consistency reasons we should not force users to create correct authentication headers.

Sure. The target application for this input initially is the MISP module, which uses API key. I think we can add user/password support quite easily.

The user case that the input was designed for does not require cookies, so
support for cookie is not planned at this point.

I see. I'm fine with that decision.