Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix cluster mode logs #9037

Merged
merged 8 commits into from
Oct 6, 2023
Merged

Fix cluster mode logs #9037

merged 8 commits into from
Oct 6, 2023

Conversation

arelra
Copy link
Member

@arelra arelra commented Oct 5, 2023

What does this change?

When cluster mode was enabled in #8929 the log count from the ec2 -> kinesis stream dropped by ~half. On investigation the logs from the child processes were not being written to the log file by log4js.

I tried two approaches to fix this:

  1. Write separate files for each process. This worked except amazon-kinesis-agent would sporadically duplicate log entries in ELK. Although amazon-kinesis-agent accepts a file pattern I suspect this is for log rotation and it does not support mulitple active log files as per: https://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html#config-start

    Although this approach is cleaner I abandoned it due to the amazon-kinesis-agent issue.

  2. Write to one log for all processes. log4js states that it will handle this transparently and it works as expected when tested on CODE.

I've also added the (existing) field thread_name to the log object to aid querying in ELK.

Why?

Log from all processes

After

Screenshot 2023-10-05 at 15 47 28

@arelra arelra requested a review from a team as a code owner October 5, 2023 15:06
@github-actions
Copy link

github-actions bot commented Oct 5, 2023

Size Change: 0 B 🆕

Total Size: 0 B

compressed-size-action

@arelra arelra added the run_chromatic Runs chromatic when label is applied label Oct 5, 2023
@arelra arelra merged commit 4dbe62c into main Oct 6, 2023
@arelra arelra deleted the ravi/consolidate-logs branch October 6, 2023 10:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dotcom-rendering run_chromatic Runs chromatic when label is applied
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants