Add input for Azure AD Entity Analytics #34305

taylor-swanson · 2023-01-18T17:19:19Z

What does this PR do?

Add new generic input for Entity Analytics. The input can be extended further through providers, which interface with an external identity provider, such as Azure Active Directory.
Add new Azure AD provider for Entity Analytics
Add docs

Why is it important?

Supports the greater Entity Analytics project by ingesting user and group identities from Azure Active Directory into Elasticsearch.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Run unit tests against x-pack/filebeat/input/entityanalytics
Testing the input outside of unit tests requires access to Azure Active Directory.

Related issues

Closes Build new input for ingesting user and group metadata from Azure AD #33866

Use cases

mergify · 2023-01-18T17:19:55Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @taylor-swanson? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2023-01-18T18:18:24Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-02-02T18:31:09.946+0000
Duration: 76 min 57 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7555
Skipped	746
Total	8301

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

- Add new generic input for Entity Analytics. The input can be extended further through providers, which interface with an external identity provider, such as Azure Active Directory. - Add new Azure AD provider for Entity Analytics - Add docs

elasticmachine · 2023-01-23T15:51:12Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

taylor-swanson · 2023-01-23T15:56:02Z

This should be ready for general review now. A couple of notes:

I'd like to keep this input "experimental" for now.
- I don't think we've completely narrowed down the data model for Entity Analytics yet, so things may change in the future.
- We plan on adding another identity provider in the future (Okta). The process of doing that may identify issues with the current design.
@jamiehynds or @SourinPaul, how do we feel about entity-analytics as the input name? Do we expect this project to have a different name in the future? While not impossible, changing the name of the input later can be a bit painful.

efd6

Initial review. From a maintenance perspective, it would be helpful to have more godoc in this.

x-pack/filebeat/input/entityanalytics/provider/provider.go

x-pack/filebeat/input/entityanalytics/internal/collections/set.go

x-pack/filebeat/input/entityanalytics/internal/collections/tree.go

x-pack/filebeat/input/entityanalytics/provider/azure/azure.go

x-pack/filebeat/input/entityanalytics/provider/azure/fetcher/mock/mock.go

x-pack/filebeat/input/entityanalytics/provider/azure/metrics.go

mergify · 2023-01-24T20:43:13Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b input-entity-analytics upstream/input-entity-analytics
git merge upstream/main
git push upstream input-entity-analytics

x-pack/filebeat/docs/inputs/input-entity-analytics.asciidoc

SourinPaul · 2023-01-25T03:20:10Z

@SourinPaul, how do we feel about entity-analytics

@taylor-swanson entity-analytics is suitable for this input. It covers the generic intent of our initiative, allowing you to reuse this input across additional vendor sources.

- Fixed issue where group relationship tree was being passed by value and not pointer to marshaling functions - Changed behavior of full sync so it will force a fresh sync from Azure AD rather than try to use existing state via the delta link token. It was observed in testing that the API sometimes doesn't report proper group membership information and never seems to come back into alignment. Forcing a fresh sync corrects this issue, and also aligns better with the concepts mentioned in the RFC.

efd6

Looking good, thank you for adding all the docs. Just a couple of queries.

x-pack/filebeat/input/entityanalytics/internal/collections/tree.go

x-pack/filebeat/input/entityanalytics/internal/collections/uuid_set_test.go

x-pack/filebeat/input/entityanalytics/internal/collections/uuid_tree.go

x-pack/filebeat/input/entityanalytics/internal/collections/set.go

x-pack/filebeat/input/entityanalytics/internal/collections/tree.go

mergify · 2023-01-27T20:06:29Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b input-entity-analytics upstream/input-entity-analytics
git merge upstream/main
git push upstream input-entity-analytics

efd6

LGTM after conflict is resolved.

taylor-swanson · 2023-01-30T14:12:20Z

Hey @belimawr or @rdner, looks like I will need approval from either one of you. Thanks!

rdner

Testing the input outside of unit tests requires access to Azure Active Directory.

I assume it was actually tested manually with AD, could you confirm that and describe the test steps and the example of the output?

Having just unit tests is not enough in this case.

x-pack/filebeat/input/entityanalytics/internal/kvstore/input_test.go

x-pack/filebeat/input/entityanalytics/internal/collections/uuid_set.go

x-pack/filebeat/input/entityanalytics/provider/azuread/authenticator/oauth2/oauth2.go

taylor-swanson · 2023-02-02T13:45:09Z

Testing the input outside of unit tests requires access to Azure Active Directory.

I assume it was actually tested manually with AD, could you confirm that and describe the test steps and the example of the output?

Having just unit tests is not enough in this case.

Yes, this was tested manually with Azure AD. For a list of test cases, I ran through the following scenarios (this was linked in the related issue):

Test Criteria

Verify users are synced.
Verify documents contain group membership info.
Update a user in Azure AD, verify the change is reflected in Elasticsearch as a new document.
Delete a user in Azure AD, verify the change is reflected in Elasticsearch as a new document that indicates a deleted status
Create a new user in Azure AD, verify the new user reflected in Elasticsearch
Verify that the data stream contains a "full sync marker" document with event.action: started when a new sync starts.
Verify that the data stream contains a "full sync marker" document with event.action: completed when a new sync completed.
Documentation exists that explains how the input works (what APIs it utilizes, how it persists information, etc)
Documentation exists that explains how to authenticate and authorize the input with least privileges.

It takes a while to get everything set up, and I just returned from traveling, so it may take me a bit to get sample documents.

- Switch from assert to require, interrupts tests in case of failure - Remove panics from test setup and replace with require.NoError - Use t.TempDir instead of os.MkdirTemp

taylor-swanson · 2023-02-02T15:30:56Z

Here are some sample documents:
Start write marker

{
    "@timestamp": "2023-02-02T15:18:08.691Z",
    "@metadata": {
        "beat": "filebeat",
        "type": "_doc",
        "version": "8.7.0"
    },
    "labels": {
        "identity_source": "azure-1"
    },
    "event": {
        "start": "2023-02-02T15:18:08.691Z",
        "action": "started"
    },
    "input": {
        "type": "entity-analytics"
    },
    "ecs": {
        "version": "8.0.0"
    },
    "host": {
        "hostname": "agent1",
        "architecture": "x86_64",
        "name": "agent1",
        "os": {
            "codename": "jammy",
            "type": "linux",
            "platform": "ubuntu",
            "version": "22.04.1 LTS (Jammy Jellyfish)",
            "family": "debian",
            "name": "Ubuntu",
            "kernel": "5.15.0-58-generic"
        },
        "id": "c7e8e9335ba042fabbbe850aa104d692",
        "containerized": false,
        "ip": [
            "10.0.2.15",
            "fe80::a00:27ff:fe88:e72a"
        ],
        "mac": [
            "08-00-27-88-E7-2A"
        ]
    },
    "agent": {
        "id": "28e36fc0-551b-4ac2-95f0-4ed66455756e",
        "name": "agent1",
        "type": "filebeat",
        "version": "8.7.0",
        "ephemeral_id": "c53ad281-54db-4263-87bf-b6da287bfd9c"
    }
}

User document:

{
    "@timestamp": "2023-02-02T15:18:08.693Z",
    "@metadata": {
        "beat": "filebeat",
        "type": "_doc",
        "version": "8.7.0"
    },
    "ecs": {
        "version": "8.0.0"
    },
    "host": {
        "mac": [
            "08-00-27-88-E7-2A"
        ],
        "name": "agent1",
        "hostname": "agent1",
        "architecture": "x86_64",
        "os": {
            "platform": "ubuntu",
            "version": "22.04.1 LTS (Jammy Jellyfish)",
            "family": "debian",
            "name": "Ubuntu",
            "kernel": "5.15.0-58-generic",
            "codename": "jammy",
            "type": "linux"
        },
        "id": "c7e8e9335ba042fabbbe850aa104d692",
        "containerized": false,
        "ip": [
            "10.0.2.15",
            "fe80::a00:27ff:fe88:e72a"
        ]
    },
    "agent": {
        "version": "8.7.0",
        "ephemeral_id": "c53ad281-54db-4263-87bf-b6da287bfd9c",
        "id": "28e36fc0-551b-4ac2-95f0-4ed66455756e",
        "name": "agent1",
        "type": "filebeat"
    },
    "azure_ad": {
        "surname": "User1",
        "userPrincipalName": "test.user1@azure2elasticsearch.onmicrosoft.com",
        "displayName": "Test User1",
        "givenName": "Test"
    },
    "labels": {
        "identity_source": "azure-1"
    },
    "user": {
        "id": "aeb2dc6a-797d-4e6d-8552-df43e4200f79",
        "group": [
            {
                "id": "a36ac877-d4e4-41d2-b2f8-5895c1ec3eb5",
                "name": "Test Group 1"
            },
            {
                "id": "3ef344f3-3cb9-45ba-b997-057b76b3c1f7",
                "name": "Test Group 2"
            },
            {
                "id": "6e47d59e-9e02-4c6b-bbb9-43ee57637619",
                "name": "Test Group 3"
            }
        ]
    },
    "event": {
        "action": "user-discovered"
    },
    "input": {
        "type": "entity-analytics"
    }
}

End write marker:

{
    "@timestamp": "2023-02-02T15:18:08.693Z",
    "@metadata": {
        "beat": "filebeat",
        "type": "_doc",
        "version": "8.7.0"
    },
    "labels": {
        "identity_source": "azure-1"
    },
    "event": {
        "action": "completed",
        "end": "2023-02-02T15:18:08.693Z"
    },
    "input": {
        "type": "entity-analytics"
    },
    "agent": {
        "version": "8.7.0",
        "ephemeral_id": "c53ad281-54db-4263-87bf-b6da287bfd9c",
        "id": "28e36fc0-551b-4ac2-95f0-4ed66455756e",
        "name": "agent1",
        "type": "filebeat"
    },
    "ecs": {
        "version": "8.0.0"
    },
    "host": {
        "containerized": false,
        "name": "agent1",
        "ip": [
            "10.0.2.15",
            "fe80::a00:27ff:fe88:e72a"
        ],
        "mac": [
            "08-00-27-88-E7-2A"
        ],
        "hostname": "agent1",
        "architecture": "x86_64",
        "os": {
            "type": "linux",
            "platform": "ubuntu",
            "version": "22.04.1 LTS (Jammy Jellyfish)",
            "family": "debian",
            "name": "Ubuntu",
            "kernel": "5.15.0-58-generic",
            "codename": "jammy"
        },
        "id": "c7e8e9335ba042fabbbe850aa104d692"
    }
}

Incremental update to user (removed group memberships):

{
    "@timestamp": "2023-02-02T15:27:17.543Z",
    "@metadata": {
        "beat": "filebeat",
        "type": "_doc",
        "version": "8.7.0"
    },
    "labels": {
        "identity_source": "azure-1"
    },
    "user": {
        "id": "aeb2dc6a-797d-4e6d-8552-df43e4200f79"
    },
    "event": {
        "action": "user-modified"
    },
    "input": {
        "type": "entity-analytics"
    },
    "ecs": {
        "version": "8.0.0"
    },
    "host": {
        "hostname": "agent1",
        "architecture": "x86_64",
        "os": {
            "family": "debian",
            "name": "Ubuntu",
            "kernel": "5.15.0-58-generic",
            "codename": "jammy",
            "type": "linux",
            "platform": "ubuntu",
            "version": "22.04.1 LTS (Jammy Jellyfish)"
        },
        "id": "c7e8e9335ba042fabbbe850aa104d692",
        "containerized": false,
        "ip": [
            "10.0.2.15",
            "fe80::a00:27ff:fe88:e72a"
        ],
        "mac": [
            "08-00-27-88-E7-2A"
        ],
        "name": "agent1"
    },
    "agent": {
        "id": "28e36fc0-551b-4ac2-95f0-4ed66455756e",
        "name": "agent1",
        "type": "filebeat",
        "version": "8.7.0",
        "ephemeral_id": "619d854f-1882-41b2-a189-9a914e068236"
    },
    "azure_ad": {
        "surname": "User1",
        "userPrincipalName": "test.user1@azure2elasticsearch.onmicrosoft.com",
        "displayName": "Test User1",
        "givenName": "Test"
    }
}

- Lone vertices aren't included in the group relationship tree, so they won't be included in the expansion from the group tree. Direct members are inserted into the transitiveMemberOf set first, then expansion occurs.

- Add new generic input for Entity Analytics. The input can be extended further through providers, which interface with an external identity provider, such as Azure Active Directory. - Add new Azure AD provider for Entity Analytics - Add docs for the new input/provider

taylor-swanson added Filebeat Filebeat Team:Security-External Integrations 8.7-candidate new input (filebeat) A new input for file beat labels Jan 18, 2023

taylor-swanson requested a review from a team January 18, 2023 17:19

taylor-swanson self-assigned this Jan 18, 2023

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 18, 2023

taylor-swanson added the backport-skip Skip notification from the automated backport with mergify label Jan 18, 2023

taylor-swanson force-pushed the input-entity-analytics branch 3 times, most recently from 7b962f4 to e03d43d Compare January 19, 2023 21:39

Add input for Azure AD Entity Analytics

e266ca4

- Add new generic input for Entity Analytics. The input can be extended further through providers, which interface with an external identity provider, such as Azure Active Directory. - Add new Azure AD provider for Entity Analytics - Add docs

taylor-swanson force-pushed the input-entity-analytics branch from e03d43d to e266ca4 Compare January 23, 2023 14:16

taylor-swanson marked this pull request as ready for review January 23, 2023 15:51

taylor-swanson requested a review from a team as a code owner January 23, 2023 15:51

taylor-swanson requested review from belimawr and rdner and removed request for a team January 23, 2023 15:51

efd6 reviewed Jan 24, 2023

View reviewed changes

andrewkroh reviewed Jan 24, 2023

View reviewed changes

x-pack/filebeat/docs/inputs/input-entity-analytics.asciidoc Outdated Show resolved Hide resolved

x-pack/filebeat/docs/inputs/input-entity-analytics.asciidoc Outdated Show resolved Hide resolved

x-pack/filebeat/docs/inputs/input-entity-analytics.asciidoc Outdated Show resolved Hide resolved

Review comments

bc8b595

taylor-swanson added 2 commits January 25, 2023 09:26

Rename provider/package and add docstrings

2cf0480

Add additional docs

a283998

taylor-swanson added 6 commits January 25, 2023 12:09

Merge branch 'main' into input-entity-analytics

eb6b2bc

make update

784c5b2

Fix edge case where edge has 'to' vertex that's not in the tree

0757128

Rewrote set/tree into non-generics version, use as value where possible

c764897

Merge branch 'main' into input-entity-analytics

e4d5ec8

Add missing license headers

1c29910

taylor-swanson mentioned this pull request Jan 26, 2023

filebeat.spec: Add new Entity Analytics input elastic/elastic-agent#2196

Merged

6 tasks

efd6 reviewed Jan 26, 2023

View reviewed changes

taylor-swanson added 2 commits January 27, 2023 08:14

Review comments

23cd434

Merge branch 'main' into input-entity-analytics

77d19e7

efd6 approved these changes Jan 27, 2023

View reviewed changes

Merge branch 'main' into input-entity-analytics

3026f74

rdner reviewed Jan 31, 2023

View reviewed changes

Clean up unit tests

a03e07a

- Switch from assert to require, interrupts tests in case of failure - Remove panics from test setup and replace with require.NoError - Use t.TempDir instead of os.MkdirTemp

taylor-swanson added 2 commits February 2, 2023 10:38

Merge branch 'main' into input-entity-analytics

2d180b9

ebeahan mentioned this pull request Feb 3, 2023

Azure AD User Entities Integration elastic/integrations#4732

Closed

rdner approved these changes Feb 6, 2023

View reviewed changes

taylor-swanson merged commit c8578a0 into elastic:main Feb 6, 2023

taylor-swanson deleted the input-entity-analytics branch February 6, 2023 14:21

jamiehynds mentioned this pull request Jul 19, 2023

[Epic] Create a Workday user provider for the entity_analytics Filebeat input elastic/integrations#7009

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add input for Azure AD Entity Analytics #34305

Add input for Azure AD Entity Analytics #34305

taylor-swanson commented Jan 18, 2023 •

edited

Loading

mergify bot commented Jan 18, 2023

elasticmachine commented Jan 18, 2023 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

elasticmachine commented Jan 23, 2023

taylor-swanson commented Jan 23, 2023

efd6 left a comment

mergify bot commented Jan 24, 2023

SourinPaul commented Jan 25, 2023

efd6 left a comment

mergify bot commented Jan 27, 2023

efd6 left a comment

taylor-swanson commented Jan 30, 2023

rdner left a comment

taylor-swanson commented Feb 2, 2023

taylor-swanson commented Feb 2, 2023

Add input for Azure AD Entity Analytics #34305

Add input for Azure AD Entity Analytics #34305

Conversation

taylor-swanson commented Jan 18, 2023 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Use cases

mergify bot commented Jan 18, 2023

elasticmachine commented Jan 18, 2023 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Jan 23, 2023

taylor-swanson commented Jan 23, 2023

efd6 left a comment

Choose a reason for hiding this comment

mergify bot commented Jan 24, 2023

SourinPaul commented Jan 25, 2023

efd6 left a comment

Choose a reason for hiding this comment

mergify bot commented Jan 27, 2023

efd6 left a comment

Choose a reason for hiding this comment

taylor-swanson commented Jan 30, 2023

rdner left a comment

Choose a reason for hiding this comment

taylor-swanson commented Feb 2, 2023

Test Criteria

taylor-swanson commented Feb 2, 2023

taylor-swanson commented Jan 18, 2023 •

edited

Loading

elasticmachine commented Jan 18, 2023 •

edited by jenkins-beats-ci bot

Loading