[receiver/mongodbatlas] Support collection of MongoDBAtlas Access Logs #21406
Conversation
| nowTimestamp := pcommon.NewTimestampFromTime(now) | ||
| logs := transformAccessLogs(nowTimestamp, fullLogs, project, cluster, alr.logger) | ||
|
|
||
| if logs.LogRecordCount() > 0 { |
There was a problem hiding this comment.
I can imagine this being a bottleneck for logs. Say we are getting the max number of log entries of 20000 and we keep adding logs to the fullLogs. We only ever send the logs down at once so we could theoretically stream logs down the telemetry pipeline. Could we try consuming logs after each request when safe to not overload processors with large batches of logs?
|
Since this got an approval - don't merge yet, figured out a timing issue in testing where logs are sometimes not available in the API for upwards of 1 minute, need to factor this delay into the start/end calculations to avoid data loss. |
There was a problem hiding this comment.
Just requesting changes cause I have some minor concerns/design decisions I want to get clarification on (they may be necessary, just wanted to discuss them first).
Also was curious if users would ever want to run logs without project access_logs? It doesn't seem like that's a use case of the current implementation or its not a documented case in the testdata/README.md
Description: Add the ability to collect MongoDB Atlas Access Logs
Link to tracking Issue: #21182
Testing: Unit test coverage & integration test based on a recently retrieved payload.
Documentation: Added documentation of configuration to the Readme.