[aws] [s3] Introduce ignore_older & start_timestamp for S3 input allowing better registry cleanups

[Kavindu-Dodan]

Proposed commit message

Introduce ignore_older and start_timestamp properties to AWS S3 input. This is a follow-up for #41694.

The configurations introduced here act as input object filters. If the object fails to match derived filters, the entries will be cleaned up from the registry, reducing filebeat memory consumption.

Introduced configurations are,

• ignore_older : Accepts a time duration in which entries are accepted for processing
• start_timestamp: A timestamp from which objects are accepted for processing

For both inputs, the object's last modified timestamp is taken into comparison. See Use cases section for further explanation

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None as defaults are disabled. However, when configurations introduced here are used, the following can have an impact on the user,

• Whenstart_timestamp is defined, then objects with the last modified timestamps prior to the timestamp are ignored from processing (documented ¹)
• When ignore_older is defined, then objects that do not fall within the look-back period when processing starts (polling run) are ignored (documented ¹)
• When both start_timestamp & ignore_older are defined, the initial run will process all entries up to start_timestamp. The subsequent runs will not include entries that do not fall within ignore_older even if processing failed for an object. (documented ¹)

How to test this PR locally

• Build filebeat from the changest included in the PR
• Source S3 bucket with objects (you may use this tool ² to create entries)
• Try configuring filebeat with alternative values for ignore_older & start_timestamp to see how data ingestion change with their values. See Use cases section for further explanation

Related issues

• Depends on [aws] feat: aws-s3 input registry cleanup for untracked s3 objects #41694
• Fixes aws-s3 input's bucket polling accumulates state in the registry #39116
• Fixes [AWS-S3] Add a timestamp filter for s3 polling mode #41232

Use cases

Consider below diagrams where there're 3 objects Object A, Object B and Object C with their last modified timestamps of t1, t2 and t3.

And consider how filebeat processes and tracks registry entries based on the following scenarios

Default behavior

If none of the configurations are used, then filebeat will process and the internal registry will track all objects continuously unless they are removed from the bucket.

Use start_timestamp

If start_timestamp is used, objects newer than the timestamp are accepted for processing. The registry will grow unless objects are removed from the bucket by other means (ex:- lifecycle policy).

Use ignore_older

If ignore_older is defined, input will process objects within the provided duration, calculated from the current time. The registry will track objects within the current timeframe and others will get cleaned up eventually by subsequent runs.

Use both ignore_older & start_timestamp

If both properties are defined,

• The initial run will include entries within the start_timestamp (ignoring ignore_older duration).
• Subsequent runs will only consider entries within the ignore_older duration.

Footnotes

1. https://github.com/elastic/beats/pull/41817/files#diff-422765b7341c5bbf6de7af38927e34e00a5073b188585a7af3c4fee1175b64a6 ↩ ↩² ↩³

2. https://github.com/Kavindu-Dodan/data-gen ↩

[leehinman]

LGTM, but I'd like @faec to take a look.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[Kavindu-Dodan]

After considering with the team and with the background from this conversation when upgrading the Integration ¹, I have decided to backport the change to 8.16.x and 8.17.x release tracks.

Footnotes

1. https://github.com/elastic/integrations/pull/12645#discussion_r1950711236 ↩