Microsoft Defender for Endpoint Responce.Split issue

[asiridiaz]

Hi Team,

In the Microsoft Defender for Endpoint integration we have found an issue where the empty valued events are getting failed at the ingest pipeline and getting thru to elastic with an error cannot access method/field [length] from a null def reference and making the index flooded with. when we investigate we found that in Elastic version 8.6.0 there has been a change in
response.split parameter where it keeps the parent document if the split target is empty. in order to drop the empty valued events we need to set the ignore_empty_value to true

where it did not required such in 8.5.x or less versions

Integration version: 2.8.0

Response from the API

{"@odata.context":"https://api.securitycenter.windows.com/api/$metadata#Alerts","value":[]}

example output:

{
  "_index": ".ds-logs-microsoft_defender_endpoint.log-xyz-2022.12.28-000002",
  "_id": "6M2c7YUB3rp0SPH_EmZe",
  "_version": 1,
  "_score": 0,
  "_source": {
    "cloud": {
      "provider": "azure"
    },
    "input": {
      "type": "httpjson"
    },
    "agent": {
      "name": "elastic-agent-xyz-1",
      "id": "574e6a81-7af4-471d-a8ac-a4f2ca6a87bb",
      "ephemeral_id": "573ac696-6d13-4938-a7e7-2df3357dffa7",
      "type": "filebeat",
      "version": "8.6.0"
    },
    "@timestamp": "2023-01-26T10:25:05.643Z",
    "ecs": {
      "version": "8.6.0"
    },
    "data_stream": {
      "namespace": "xyz",
      "type": "logs",
      "dataset": "microsoft_defender_endpoint.log"
    },
    "elastic_agent": {
      "id": "574e6a81-7af4-471d-a8ac-a4f2ca6a87bb",
      "version": "8.6.0",
      "snapshot": false
    },
    "json": {
      "@odata.context": "https://api.securitycenter.windows.com/api/$metadata#Alerts",
      "value": []
    },
    "event": {
      "agent_id_status": "verified",
      "ingested": "2023-01-26T10:25:06Z",
      "original": "{\"@odata.context\":\"https://api.securitycenter.windows.com/api/$metadata#Alerts\",\"value\":[]}",
      "provider": "defender_endpoint",
      "created": "2023-01-26T10:25:05.643Z",
      "kind": "alert",
      "timezone": "UTC",
      "category": [
        "host"
      ],
      "dataset": "microsoft_defender_endpoint.log"
    },
    "error": {
      "message": "cannot access method/field [length] from a null def reference"
    },
    "tags": [
      "microsoft-defender-endpoint",
      "forwarded"
    ]
  },
  "fields": {
    "elastic_agent.version": [
      "8.6.0"
    ],
    "event.category": [
      "host"
    ],
    "agent.type": [
      "filebeat"
    ],
    "event.module": [
      "microsoft_defender_endpoint"
    ],
    "agent.name": [
      "elastic-agent-xyz-1"
    ],
    "elastic_agent.snapshot": [
      false
    ],
    "event.agent_id_status": [
      "verified"
    ],
    "event.kind": [
      "alert"
    ],
    "event.timezone": [
      "UTC"
    ],
    "event.original": [
      "{\"@odata.context\":\"https://api.securitycenter.windows.com/api/$metadata#Alerts\",\"value\":[]}"
    ],
    "elastic_agent.id": [
      "574e6a81-7af4-471d-a8ac-a4f2ca6a87bb"
    ],
    "data_stream.namespace": [
      "xyz"
    ],
    "json.@odata.context": [
      "https://api.securitycenter.windows.com/api/$metadata#Alerts"
    ],
    "input.type": [
      "httpjson"
    ],
    "data_stream.type": [
      "logs"
    ],
    "tags": [
      "microsoft-defender-endpoint",
      "forwarded"
    ],
    "event.ingested": [
      "2023-01-26T10:25:06Z"
    ],
    "cloud.provider": [
      "azure"
    ],
    "event.provider": [
      "defender_endpoint"
    ],
    "@timestamp": [
      "2023-01-26T10:25:05.643Z"
    ],
    "agent.id": [
      "574e6a81-7af4-471d-a8ac-a4f2ca6a87bb"
    ],
    "ecs.version": [
      "8.6.0"
    ],
    "error.message": [
      "cannot access method/field [length] from a null def reference"
    ],
    "event.created": [
      "2023-01-26T10:25:05.643Z"
    ],
    "data_stream.dataset": [
      "microsoft_defender_endpoint.log"
    ],
    "agent.ephemeral_id": [
      "573ac696-6d13-4938-a7e7-2df3357dffa7"
    ],
    "agent.version": [
      "8.6.0"
    ],
    "event.dataset": [
      "microsoft_defender_endpoint.log"
    ]
  }
}

The fixing needs to be done on the below files in the integration

packages/microsoft_defender_endpoint/data_stream/log/agent/stream/httpjson.yml.hbs

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[cakarlen]

Also experiencing this bug on our cluster running same version of integration

[asiridiaz]

Hi, @efd6
Hope this issue also have been fixed with #5164

[efd6]

@asiridiaz Yes. Can you confirm?

[asiridiaz]

@efd6 yes, It's working fine now. Thanks, Dan,