Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cherry-pick #22371 to 7.x: Add support for platform logs in Filebeat Azure module #22749

Merged
merged 1 commit into from
Nov 30, 2020
Merged

Cherry-pick #22371 to 7.x: Add support for platform logs in Filebeat Azure module #22749

merged 1 commit into from
Nov 30, 2020

Conversation

narph
Copy link
Contributor

@narph narph commented Nov 25, 2020

Cherry-pick of PR #22371 to 7.x branch. Original message:

What does this PR do?

Add support for platform logs in Filebeat Azure module

Why is it important?

Platform logs provide detailed diagnostic and auditing information for Azure resources and the Azure platform they depend on.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Example platform logs:

{
   "ActivityId":"30ed877c-a36b-491a-bd4d-ddd847fe55b8",
   "Caller":"Portal",
   "Environment":"PROD",
   "EventName":"Retreive ConsumerGroup",
   "EventProperties":"{\"SubscriptionId\":\"...\",\"Namespace\":\"obstesteventhubs\",\"Via\":\"sb://obstesteventhubs.servicebus.windows.net/insights-logs-operationallogs/consumergroups?api-version=2017-04\u0026$skip=0\u0026$top=100\",\"TrackingId\":\"30ed877c-a36b-491a-bd4d-ddd847fe55b8_M2CH3_M2CH3_G3S2\"}",
   "EventTimeString":"11/3/2020 9:06:42 AM +00:00",
   "Region":"West Europe",
   "ScaleUnit":"PROD-AM3-AZ501",
   "Status":"Succeeded",
   "SubscriptionId":"....",
   "category":"OperationalLogs",
   "resourceId":"/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-TEST/PROVIDERS/MICROSOFT.EVENTHUB/NAMESPACES/OBSTESTEVENTHUBS"
}

{
   "Cloud":"AzureCloud",
   "Environment":"prod",
   "UnderlayClass":"hcp-underlay",
   "UnderlayName":"hcp-underlay-westeurope-cx-316",
   "attrs":"{\"annotation.io.kubernetes.container.hash\"=\u003e\"b74d7ef3\", \"annotation.io.kubernetes.container.ports\"=\u003e\"[{\\\"name\\\":\\\"https\\\",\\\"containerPort\\\":4444,\\\"protocol\\\":\\\"TCP\\\"}]\", \"annotation.io.kubernetes.container.preStopHandler\"=\u003e\"{\\\"exec\\\":{\\\"command\\\":[\\\"/bin/bash\\\",\\\"-c\\\",\\\"sleep 20\\\"]}}\", \"annotation.io.kubernetes.container.restartCount\"=\u003e\"0\", \"annotation.io.kubernetes.container.terminationMessagePath\"=\u003e\"/dev/termination-log\", \"annotation.io.kubernetes.container.terminationMessagePolicy\"=\u003e\"File\", \"annotation.io.kubernetes.pod.terminationGracePeriod\"=\u003e\"30\", \"io.kubernetes.container.logpath\"=\u003e\"/var/log/pods/5e4bf4baee195b00017cdbfa_kube-apiserver-666bd4b459-vgc5h_53331907-669d-458e-ab0e-6744e56164f9/kube-apiserver/0.log\", \"io.kubernetes.container.name\"=\u003e\"kube-apiserver\", \"io.kubernetes.docker.type\"=\u003e\"container\", \"io.kubernetes.pod.name\"=\u003e\"kube-apiserver-666bd4b459-vgc5h\", \"io.kubernetes.pod.namespace\"=\u003e\"5e4bf4baee195b00017cdbfa\", \"io.kubernetes.pod.uid\"=\u003e\"53331907-669d-458e-ab0e-6744e56164f9\", \"io.kubernetes.sandbox.id\"=\u003e\"09097fc97b3c0130bde19eec329b588a328430f3c221c1f8e1520933b16dde9f\"}",
   "category":"kube-apiserver",
   "ccpNamespace":"5e4bf4baee195b00017cdbfa",
   "operationName":"Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
   "properties":{
      "containerID":"ca7ca3b15f428368fabab4dff0c14879a838f8653f84312833d5024547a008f4",
      "log":"I1105 09:40:47.768168       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io\n",
      "pod":"kube-apiserver-666bd4b459-vgc5h",
      "stream":"stderr"
   },
   "resourceId":"/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-INFRASTRUCTURE/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/OBSKUBE",
   "time":"2020-11-05T09:40:47.0000000Z"
}

Ex outputs:

{
        "_index" : "filebeat-8.0.0-2020.11.05-000001",
        "_type" : "_doc",
        "_id" : "YQVzmHUBvB2moownK3Qu",
        "_score" : null,
        "_source" : {
          "agent" : {
            "name" : "DESKTOP-RFOOE09",
            "id" : "17cead09-57ad-4668-8a0e-b9025f8b0cb0",
            "type" : "filebeat",
            "ephemeral_id" : "a9a122d4-4b37-4c34-ac03-4a2f765a9ea1",
            "version" : "8.0.0"
          },
          "fileset" : {
            "name" : "platformlogs"
          },
          "azure-eventhub" : {
            "sequence_number" : 14,
            "consumer_group" : "$Default",
            "offset" : 4294975208,
            "eventhub" : "insights-logs-operationallogs",
            "enqueued_time" : "2020-11-05T10:27:43.439Z"
          },
          "tags" : [
            "forwarded"
          ],
          "cloud" : {
            "provider" : "azure",
            "region" : "West Europe"
          },
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2020-11-05T10:26:33.000Z",
          "ecs" : {
            "version" : "1.6.0"
          },
          "service" : {
            "type" : "azure"
          },
          "event" : {
            "ingested" : "2020-11-05T12:48:05.927630600Z",
            "kind" : "event",
            "module" : "azure",
            "action" : "Retreive Namespace",
            "dataset" : "azure.platformlogs",
            "outcome" : "succeeded"
          },
          "azure" : {
            "subscription_id" : "....",
            "platformlogs" : {
              "Status" : "Succeeded",
              "SubscriptionId" : "....",
              "Caller" : "Portal",
              "ActivityId" : "724eb3a4-6c1d-4a2f-8037-8d4db250154c",
              "EventTimeString" : "11/5/2020 10:26:33 AM +00:00",
              "Environment" : "PROD",
              "category" : "OperationalLogs",
              "event_category" : "Administrative",
              "ScaleUnit" : "PROD-AM3-AZ501",
              "properties" : {
                "SubscriptionId" : "...",
                "TrackingId" : "724eb3a4-6c1d-4a2f-8037-8d4db250154c_M6SN1_M6SN1_G10S1",
                "Namespace" : "obstesteventhubs",
                "Via" : "https://obstesteventhubs.servicebus.windows.net/$Resources/eventhubs?api-version=2017-04&$skip=0&$top=100"
              }
            },
            "resource" : {
              "provider" : "MICROSOFT.EVENTHUB/NAMESPACES",
              "name" : "OBSTESTEVENTHUBS",
              "id" : "/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-TEST/PROVIDERS/MICROSOFT.EVENTHUB/NAMESPACES/OBSTESTEVENTHUBS",
              "group" : "OBS-TEST"
            }
          }
        }

{
        "_index" : "filebeat-8.0.0-2020.11.05-000001",
        "_type" : "_doc",
        "_id" : "oQRumHUBvB2moownKezJ",
        "_score" : null,
        "_source" : {
          "agent" : {
            "name" : "DESKTOP-RFOOE09",
            "id" : "17cead09-57ad-4668-8a0e-b9025f8b0cb0",
            "ephemeral_id" : "a6339387-a2fe-4db0-9d13-ca3123f92366",
            "type" : "filebeat",
            "version" : "8.0.0"
          },
          "fileset" : {
            "name" : "platformlogs"
          },
          "message" : """I1105 12:41:49.339404       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
""",
          "azure-eventhub" : {
            "sequence_number" : 45,
            "consumer_group" : "$Default",
            "offset" : 100168,
            "eventhub" : "insights-logs-kube-apiserver",
            "enqueued_time" : "2020-11-05T12:42:14.453Z"
          },
          "tags" : [
            "forwarded"
          ],
          "cloud" : {
            "provider" : "azure"
          },
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2020-11-05T12:41:49.000Z",
          "ecs" : {
            "version" : "1.6.0"
          },
          "service" : {
            "type" : "azure"
          },
          "event" : {
            "ingested" : "2020-11-05T12:42:37.895235200Z",
            "kind" : "event",
            "module" : "azure",
            "action" : "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
            "dataset" : "azure.platformlogs"
          },
          "azure" : {
            "subscription_id" : "...",
            "platformlogs" : {
              "ccpNamespace" : "5e4bf4baee195b00017cdbfa",
              "operation_name" : "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
              "Cloud" : "AzureCloud",
              "Environment" : "prod",
              "UnderlayClass" : "hcp-underlay",
              "UnderlayName" : "hcp-underlay-westeurope-cx-316",
              "category" : "kube-apiserver",
              "event_category" : "Administrative",
              "properties" : {
                "pod" : "kube-apiserver-666bd4b459-vgc5h",
                "stream" : "stderr",
                "containerID" : "ca7ca3b15f428368fabab4dff0c14879a838f8653f84312833d5024547a008f4"
              },
              "attrs" : """{"annotation.io.kubernetes.container.hash"=>"b74d7ef3", "annotation.io.kubernetes.container.ports"=>"[{\"name\":\"https\",\"containerPort\":4444,\"protocol\":\"TCP\"}]", "annotation.io.kubernetes.container.preStopHandler"=>"{\"exec\":{\"command\":[\"/bin/bash\",\"-c\",\"sleep 20\"]}}", "annotation.io.kubernetes.container.restartCount"=>"0", "annotation.io.kubernetes.container.terminationMessagePath"=>"/dev/termination-log", "annotation.io.kubernetes.container.terminationMessagePolicy"=>"File", "annotation.io.kubernetes.pod.terminationGracePeriod"=>"30", "io.kubernetes.container.logpath"=>"/var/log/pods/5e4bf4baee195b00017cdbfa_kube-apiserver-666bd4b459-vgc5h_53331907-669d-458e-ab0e-6744e56164f9/kube-apiserver/0.log", "io.kubernetes.container.name"=>"kube-apiserver", "io.kubernetes.docker.type"=>"container", "io.kubernetes.pod.name"=>"kube-apiserver-666bd4b459-vgc5h", "io.kubernetes.pod.namespace"=>"5e4bf4baee195b00017cdbfa", "io.kubernetes.pod.uid"=>"53331907-669d-458e-ab0e-6744e56164f9", "io.kubernetes.sandbox.id"=>"09097fc97b3c0130bde19eec329b588a328430f3c221c1f8e1520933b16dde9f"}"""
            },
            "resource" : {
              "provider" : "MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS",
              "name" : "OBSKUBE",
              "id" : "/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-INFRASTRUCTURE/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/OBSKUBE",
              "group" : "OBS-INFRASTRUCTURE"
            }
          }
        },

@narph
Add support for platform logs in Filebeat Azure module (elastic#22371)
40ca271 
* mofidy doc

* platformlogs

* fix

* separate pr

* work on platform

* work

* platforms

* changelog

* fix file

* add tests

* add mapping

* test

* update mapping

* fix file name

* update file

* map field

* update files

* fix logs

* generate tests

(cherry picked from commit 2ae52c3)
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 25, 2020
@narph narph self-assigned this Nov 25, 2020
@narph narph added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Nov 25, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 25, 2020
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #22749 opened]

  • Start Time: 2020-11-25T10:15:08.690+0000

  • Duration: 44 min 50 sec

Test stats 🧪

Test Results
Failed 0
Passed 4492
Skipped 568
Total 5060

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 4492
Skipped 568
Total 5060

@narph narph merged commit d950462 into elastic:7.x Nov 30, 2020
@narph narph deleted the backport_22371_7.x branch November 30, 2020 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jsoriano jsoriano jsoriano approved these changes

Assignees

@narph narph

Labels
backport Team:Platforms Label for the Integrations - Platforms team [zube]: In Review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@narph @elasticmachine @jsoriano