-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add input for Cloudwatch logs via Kinesis #13317
Add input for Cloudwatch logs via Kinesis #13317
Conversation
x-pack/functionbeat/provider/aws/aws/transformer/transformer.go
Outdated
Show resolved
Hide resolved
return nil, err | ||
} | ||
kinesisData = outBuf.Bytes() | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that is what I expected also. (That's why the development took longer than expected.) But when I tested it with a Kinesis stream receiving events from a Log stream, I got gzipped data. I tried to find documentation and all I got is this:
You can use CloudWatch Logs subscription feature to stream data from CloudWatch Logs to Kinesis Data Firehose. All log events from CloudWatch Logs are already compressed in gzip format, so you should keep Firehose’s compression configuration as uncompressed to avoid double-compression.
But this is about Data Firehose and I used Data Streams, so I guess it applies to all of Kinesis services.
I will spend a bit more time about finding out more about compression in Kinesis and Cloudwatch.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As a compromise, I added a new option base64_encoded
so we support both my experience with Kinesis and the documentation.
Failing tests are unrelated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changes LGTM, we will need to add a followup docs PR after, but lets get that merge for FF.
dfab075
to
10fc0af
Compare
Failing tests are unrelated. |
@kvch @dedemorton when going through the documentation, it was a bit difficult to find what inputs Functionbeat has. Perhaps I'm missing something, but I only see it in the |
@acchen97 Yah now that we have more than a couple of inputs, it's probably worth breaking them out into separate sections like we do for Filebeat. So we might have:
Is that what you had in mind? (We can also change Configure functions to Configure inputs, but I'm not sure if we want to do that.) |
@dedemorton yep, that's what I was thinking (enhanced list below). I think the current
|
Does this solve for #12442? If I understand the code and read the discussion correctly, I think it should solve for it such that multiple records from the array of events from cloudwatch indeed are transformed into individual documents in ES. |
I just did a quick test with |
We will work on updating the documentation. The type value for this new feature is actually |
New experimental input is added to Functionbeat to read logs from Cloudwatch coming through Kinesis.
I have tested the input manually.
Configuration
The configuration is similar to the existing Kinesis function. It has two additional options,
base64_encoded
andcompressed
. If these options are set first base64 decode takes place, then gzip decompression. If your data is just gzipped, only setcompressed
option.TODO