Build new input for ingesting user and group metadata from Azure AD

[taylor-swanson]

Follow up issue to fully implement the proof of concept developed in: https://github.com/elastic/security-team/issues/5088

This input will support an integration which will produce user entity data that is locally enriched with the user's groups from Azure Active Directory (Azure AD). Data is produced in accordance to RFC 2022-09-07-user-host-entity-ingestion.

Discussions have also occurred around whether this input should be added to filebeat or if a new beat should be created. We are leaning towards adding to filebeat, since the new V2 architecture for agent inputs intends for filebeat and other beats to be obsoleted.

Regardless of if a new beat is created or filebeat is used, we may also need to create a new persistence mechanism to store user and group metadata. The existing registry may work for small directories with few users and groups, but it will likely not scale into larger environments with thousands of users and groups.

Acceptance Criteria

• Collect user data from Azure AD API
• Collect group membership data from Azure AD API
• Generate one document per user that includes group membership data
• Periodically get a list of users that were modified or deleted and generate new documents
• Persist data to disk that allows the input to resume from previous state

Test Criteria

• Verify users are synced.
• Verify documents contain group membership info.
• Update a user in Azure AD, verify the change is reflected in Elasticsearch as a new document.
• Delete a user in Azure AD, verify the change is reflected in Elasticsearch as a new document that indicates a deleted status
• Create a new user in Azure AD, verify the new user reflected in Elasticsearch
• Verify that the data stream contains a "full sync marker" document with event.action: started when a new sync starts.
• Verify that the data stream contains a "full sync marker" document with event.action: completed when a new sync completed.
• Documentation exists that explains how the input works (what APIs it utilizes, how it persists information, etc)
• Documentation exists that explains how to authenticate and authorize the input with least privileges.

References

• GET /users
• GET /users/delta
• GET /users/{id | userPrincipalName}/transitiveMemberOf
• User Entity Integrations RFC

Related

• PoC repo: https://github.com/elastic/entitystream
• PoC: https://github.com/elastic/security-team/issues/5088
• Parent: https://github.com/elastic/security-team/issues/4163#issuecomment-1165709940
• Elastic-Agent change: filebeat.spec: Add new Entity Analytics input elastic-agent#2196

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)