Add backup to bucket functionality

[jniebuhr]

• Enhancement

What does this PR do?

Adds a functionality to backup processed files to another or the same bucket with an optional prefix. If enabled it will delete files from the source bucket. This functionality is influenced by the same feature in the logstash s3 plugin.

Why is it important?

This is important to signal that specific files are processed in S3 buckets and perform actions such as archiving / deleting those. It will also help not process files again when state has been lost.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Changelog files seem to contain a lot of old stuff, how do those work?

Author's Checklist

How to test this PR locally

Related issues

• Closes Backup to bucket feature in filebeat aws-s3 input #30696

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @jniebuhr? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-12-22T11:00:41.855+0000

• Duration: 132 min 26 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7718
Skipped	513
Total	8231

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[aspacca]

hello @jniebuhr , thanks for the contribution

the feature you added is very interesting indeed :)

it needs some refinement anyway: the main problem is that having the backup and delete strategy in the s3 processor can lead to some issue when the input is set up with sqs+s3

a single sqs message could reference multiple s3 objects: if one of this objects processing returns an error the message will go back to the queue, but in the meantime the other objects could be already deleted. once we process the sqs message again now the previous backed up and deleted s3 objects will return an error, and the message will go back to the queue again. and so on untile the max number of retries is reached.

I think the best we be to have a finalise processor where we reference the s3 objects in batch, as required by the content of the sqs message or the s3 listing, and proceed with running the backup and deletion only if when the listing or the sqs message are fully processed without errors.

it requires a little more complexity but it will have the stability and reliance we need.

also I'd prefer to force a different bucket as backup destination: while with the correct setting of sqs notification and prefix path listing using the same bucket is not a problem, it requires prior knowledge a proper setup. doing differently will produce and endless loop where an s3 object is processed, backed up, then the backup will be processed, backup ed on its own, and the backup of the backup processed etc etc

if possible I would avoid user to be put in the situation to fire themselves in the foot like this :)

[jniebuhr]

@aspacca sorry, added it

[aspacca]

@jniebuhr all good

please: add an entry in CHANGELOG.next.asciidoc, thanks

[jniebuhr]

@jniebuhr all good

please: add an entry in CHANGELOG.next.asciidoc, thanks

done :)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/backup-to-bucket upstream/feature/backup-to-bucket
git merge upstream/main
git push upstream feature/backup-to-bucket

[aspacca]

/test

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/backup-to-bucket upstream/feature/backup-to-bucket
git merge upstream/main
git push upstream feature/backup-to-bucket

[aspacca]

@jniebuhr , can you resolve the conflicts?

thanks

[jniebuhr]

Hi @aspacca, all resolved

[aspacca]

@jniebuhr thanks

I will run integration tests locally, since they are currently skipped in CI, and if everything is good I will merge

[kaiyan-sheng]

Do we need to add s3:PutObject into the AWS Permissions section of this documentation at line 469?

[aspacca]

good catch! also s3:DeleteObject

[aspacca]

Hi @aspacca, all resolved

hello @jniebuhr , sorry, I've missed the notification about conflicts resolved.

I'm going to add the documentation about the permission required by the back feature and took the liberty to rename delete to delete_after_backup

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
19 Code Smells

No Coverage information
9.8% Duplication