Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this change?
When cluster mode was enabled in #8929 the log count from the ec2 -> kinesis stream dropped by ~half. On investigation the logs from the child processes were not being written to the log file by log4js.
I tried two approaches to fix this:
Write separate files for each process. This worked except
amazon-kinesis-agent
would sporadically duplicate log entries in ELK. Althoughamazon-kinesis-agent
accepts a file pattern I suspect this is for log rotation and it does not support mulitple active log files as per: https://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html#config-startAlthough this approach is cleaner I abandoned it due to the
amazon-kinesis-agent
issue.Write to one log for all processes. log4js states that it will handle this transparently and it works as expected when tested on CODE.
I've also added the (existing) field
thread_name
to the log object to aid querying in ELK.Why?
Log from all processes
After