[Filebeat] Add support for events triggered by S3 uploads from the AWS console in S3 input

[darupedk]

• Add support for the ObjectCreated:CompleteMultipartUpload event in S3 input

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[kaiyan-sheng]

jenkins, test this please

[kaiyan-sheng]

@darupedk Thanks for adding this 👍 We are working on adding elb fileset in Filebeat aws module and ran into this same problem.

Looks like this PR might need a rebase to make CI happy. Could you do that please?

[darupedk]

@darupedk Thanks for adding this 👍 We are working on adding elb fileset in Filebeat aws module and ran into this same problem.

Looks like this PR might need a rebase to make CI happy. Could you do that please?

There we go.

In case we want full support for create events, we need to support the complete list of events:

• s3:ObjectCreated:Put
• s3:ObjectCreated:Post
• s3:ObjectCreated:Copy
• s3:ObjectCreated:CompleteMultipartUpload

I haven't inspected the Post and Copy events (JSON) which is why I haven't added them. However, we could do strings.HasPrefix('s3:ObjectCreated:') and check if the Post and Copy events are based on the same JSON structure. I wouldn't mind checking up on that.

[darupedk]

@darupedk Thanks for adding this 👍 We are working on adding elb fileset in Filebeat aws module and ran into this same problem.
Looks like this PR might need a rebase to make CI happy. Could you do that please?

There we go.

In case we want full support for create events, we need to support the complete list of events:

• s3:ObjectCreated:Put
• s3:ObjectCreated:Post
• s3:ObjectCreated:Copy
• s3:ObjectCreated:CompleteMultipartUpload

I haven't inspected the Post and Copy events (JSON) which is why I haven't added them. However, we could do strings.HasPrefix('s3:ObjectCreated:') and check if the Post and Copy events are based on the same JSON structure. I wouldn't mind checking up on that.

I have just inspected them and they do follow the same structure. I recommend changing the code to use strings.HasPrefix(record.EventName, "ObjectCreated:").

[kaiyan-sheng]

jenkins, test this please

[darupedk]

TestHandleMessage is failing due to the following invalid S3 event:

		{
			"sqs message with event source aws:s3 and event name ObjectCreated:Delete",
			sqs.Message{
				Body: awssdk.String("{\"Records\":[{\"eventSource\":\"aws:s3\",\"awsRegion\":\"ap-southeast-1\",\"eventTime\":\"2019-06-21T16:16:54.629Z\",\"eventName\":\"ObjectCreated:Delete\",\"s3\":{\"configurationId\":\"object-created-event\",\"bucket\":{\"name\":\"test-s3-ks-2\",\"arn\":\"arn:aws:s3:::test-s3-ks-2\"},\"object\":{\"key\":\"server-access-logging2019-06-21-16-16-54-E68E4316CEB285AA\"}}}]}"),
			},
			[]s3Info{},
		},

We can go ahead and validate the events a bit more in the code but AWS would never send a message like this. We could instead replace the test message with ObjectRemoved:Delete which is a valid S3 event though not one that should be handled by the plugin.

[kaiyan-sheng]

TestHandleMessage is failing due to the following invalid S3 event:

		{
			"sqs message with event source aws:s3 and event name ObjectCreated:Delete",
			sqs.Message{
				Body: awssdk.String("{\"Records\":[{\"eventSource\":\"aws:s3\",\"awsRegion\":\"ap-southeast-1\",\"eventTime\":\"2019-06-21T16:16:54.629Z\",\"eventName\":\"ObjectCreated:Delete\",\"s3\":{\"configurationId\":\"object-created-event\",\"bucket\":{\"name\":\"test-s3-ks-2\",\"arn\":\"arn:aws:s3:::test-s3-ks-2\"},\"object\":{\"key\":\"server-access-logging2019-06-21-16-16-54-E68E4316CEB285AA\"}}}]}"),
			},
			[]s3Info{},
		},

We can go ahead and validate the events a bit more in the code but AWS would never send a message like this. We could instead replace the test message with ObjectRemoved:Delete which is a valid S3 event though not one that should be handled by the plugin.

Changing to ObjectRemoved:Delete sounds good. Sorry I made this up just for pure testing purpose originally.

[jsoriano]

ok to test

[jsoriano]

LGTM, we will need to backport this

[kaiyan-sheng]

LGTM, we will need to backport this

Thanks! I will merge it now and backport it.