[AWS S3] Introduce start timestamp and ignore older timespan to AWS S3 based integrations #12645
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kavindu-Dodan
force-pushed
the
feat/s3-start-and-ignore-older-timestamps
branch
3 times, most recently
from
888b269 to
2540178
Compare
🚀 Benchmarks reportTo see the full report comment with |
Kavindu-Dodan
added
the
Team:obs-ds-hosted-services
Kavindu-Dodan
changed the title
gizas
reviewed
Feb 10, 2025
packages/cloudflare_logpush/data_stream/audit/agent/stream/aws-s3.yml.hbs
Show resolved
Hide resolved
gizas
reviewed
Feb 10, 2025
packages/cloudflare_logpush/data_stream/casb/agent/stream/aws-s3.yml.hbs
Show resolved
Hide resolved
gizas
approved these changes
Feb 10, 2025
|
@Kavindu-Dodan , could you please consider backporting the beats changes so that we can bump up the package versions only to cc @zmoog |
zmoog
reviewed
Feb 11, 2025
Kavindu-Dodan
force-pushed
the
feat/s3-start-and-ignore-older-timestamps
branch
from
2540178 to
58cdb90
Compare
Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # packages/amazon_security_lake/changelog.yml # packages/amazon_security_lake/manifest.yml # packages/aws/changelog.yml # packages/canva/changelog.yml # packages/canva/manifest.yml # packages/cloudflare_logpush/changelog.yml # packages/cloudflare_logpush/manifest.yml # packages/f5_bigip/changelog.yml # packages/imperva_cloud_waf/changelog.yml # packages/imperva_cloud_waf/manifest.yml # packages/servicenow/changelog.yml # packages/servicenow/manifest.yml # packages/sublime_security/changelog.yml # packages/symantec_endpoint_security/changelog.yml # packages/symantec_endpoint_security/manifest.yml
Kavindu-Dodan
force-pushed
the
feat/s3-start-and-ignore-older-timestamps
branch
from
58cdb90 to
4f09456
Compare
Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>
💚 Build Succeeded
History
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Introduce
Ignore Older TimespanandStart Timestampproperties to integrations backed by AWS S3 input,Configuring these properties allows S3 input to efficiently manage its internal registry. For example, setting
Ignore Older Timespanto2hmakes the S3 input registry only track entries within the last 2 hours. Once entries are beyond the timespan, input can remove them from the registry, thus reducing memory consumption.Checklist
changelog.ymlfile.Related issues
Screenshots
Configuration rendered (Title matching existing format)