Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sort logs by timestamp if present #72

Merged
merged 1 commit into from
Oct 5, 2023
Merged

Sort logs by timestamp if present #72

merged 1 commit into from
Oct 5, 2023

Conversation

tlmii
Copy link
Member

@tlmii tlmii commented Oct 5, 2023

Currently logs from stdout and stderr can be displayed out of order when the streams (containers) or files (projects and executables) for each are processed and fed into the log viewer.

This PR partially addresses this by doing a couple of things:

  1. If a timestamp is present as the first thing on a line, that timestamp is recorded and used to insert the line after any prior (per the timestamp) lines that were already added. The line is also treated as the first line (of potentially many) in a single log entry.
  2. If a line instead starts with trce/dbug/info/warn/fail/crit, it is also treated as the first line (of potentially many) in a single log entry. But for inserting it into the log viewer, it is simply appended to the end since there is no timestamp
  3. If a line does not meet either of the above conditions, then it is considered a child line relative to the most recent first line that was written to the logs, if any. If there is a prior "first line" then it will be inserted after the last child line (if any) of that log entry. If there isn't, then it is simply appended to the end.

The clearest way to see this in action right now is with the catalog (postgres) container in the repo right now. It has both stdout/stderr logs and they are currently written out of order. With this PR, they're written in order.

If timestamps or log level indicators are present, sorting/grouping will take place. But if they're not present, all log entries will just be written in the order they're received.

Project and executable logs do not currently have timestamps (and are not guaranteed to the way container logs are) but it is simple enough to add.

Note that for the moment the only timestamp format we support is RFC3339 (including RFC3339Nano used by docker logs). We can add more support in the future as needed.

@tlmii
Copy link
Member Author

tlmii commented Oct 5, 2023

Project and executable logs do not currently have timestamps (and are not guaranteed to the way container logs are) but it is simple enough to add.

Adding timestamps to the projects in a supported format is as easy as:

builder.Logging.AddSimpleConsole(configure =>
{
    configure.TimestampFormat = "yyyy-MM-dd'T'HH:mm:ss.ffffff'Z' ";
    configure.UseUtcTimestamp = true;
});

I debated adding that to all of the projects in the in-repo sample and to the templates, but left them alone for now. Once this PR goes in, I'll create an issue to track potentially adding them.

if (parentId) {
// If we have a parent id, then we know we're on a non-timestamped line that is part
// of a multi-line log entry. We need to find the prior line from that entry
prior = container.querySelector(`div[data-log-id="${parentId}"][data-line-index="${lineIndex - 1}"]`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would we have lineIndex for previous line already inserted for sure always?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If not, a lot of my other assumptions are broken too 😂 But yes, it should always be there already. The stdout and stderr logs can come in asynchronously and out of order from each other, but within one or the other (e.g. within stdout) they should always be coming in in order.

@tlmii tlmii merged commit 30bd084 into dotnet:main Oct 5, 2023
@tlmii tlmii deleted the dev/sorted-logs branch October 5, 2023 18:24
@github-actions github-actions bot locked and limited conversation to collaborators May 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants