Fine tune performance

[RolandPheasant]

1. Scrolling can be sluggish on files which have a very large number of lines. So far I have seen scrolling stutter on a file with 800k lines which was opened on a server share. In theory this can be solved by adjusting the scrolling sample rate, but in practice it is hard to predict when back pressures are formed. The idea solution would be for a task to run and if a queue has formed disregard all but the end of the queued job.
2. Some times there is a minor latency when opening a file.
3. Investigate whether the read algorithms can be improved

[gmta]

+1 on the scrolling. It seems that the UI is updated every 500ms or so, while other tools update instantaneously.

Also, when heavy file update activity occurs, the application seems to hog 100% of one CPU core.

[RolandPheasant]

I have just been experimenting with indexing all the lines and have been able to achieve blisteringly fast scrolling. When the file is indexed, stream.seek method is used to load the current page.

The largest file I have tested has 1.3 million lines and it scrolls very well. Of course the indexing adds to memory pressures but the experiment has shown me I am on the right track.

[gmta]

You could reduce memory pressure by keeping a sparse index, e.g. indexing every 100th line instead of every line and using seeks for all lines in between. Doesn't have to be a constant, you can vary it by distance for example (index every line within 1000 lines of the current position, and decrease the resolution the further you go).

Alternatively, some tail tools (such as BareTail) give you the option to just keep the last X lines of the file in memory, discarding everything that falls off.

[RolandPheasant]

@oniony and I were also discussing some more sophisticated indexing options and there is some overlap with your suggestion.

You can be assured that fine tuning performance is my current priority
Thanks

[RolandPheasant]

@gmta Its fast now check out this Fast Scrolling

For super large files probably more work required but for now I am going to put my feet up

[gmta]

@RolandPheasant I tried the latest release but it isn't able to read my log files correctly. Lines are garbled and distorted, sometimes beyond recognition. They are fairly standard WildFly log files.

So I'm not really able to test your changes. However, it does seem like the UI updates every 500ms making it a somewhat slow experience compared to other tail tools like Baretail.

[RolandPheasant]

I just spotted that . Seems to be the way I have calculated the line lengths when indexing the file. I am on the case. Is it possible to attach as smallish extract here so I can test the indexing on your file type.

Edit: I will fix the encoding / indexing then I will address the scroll issue you have - my suspicion is the scroll issue is my sample rate is actually too fast which leads to a queue of updates hitting the screen at once.

[RolandPheasant]

@gmta I think I have worked out the issue for both problems. Should be able to get a fix out in a day or 2.

[RolandPheasant]

@gmta the files should decode correctly now.

[gmta]

Decoding seems to be fixed, great work! Tailblazer seems very CPU bound at the moment, it barely responds when other applications are taxing my CPU at the same time, even for small log files.

[RolandPheasant]

Thanks for that. I suspect the underlying problem is network latency coupled with too fast a poll interval. I am looking in work at an active 1.3 M line log file (100 MB in size) over a network share and my cpu is doing almost nothing

So to address the issue I will:

a) slow down the poll rate
b) rather than polling on a schedule, poll and allow for inactivity before polling again

I suspect in your instance the time it takes to check for changes and subsequently load the changes is longer than the polling interval leading to an ever growing queue of updates. I will add a load of logging soon.

[RolandPheasant]

@gmta I have raise issue #30 to deal this this one and will be opening issues specifically for different performance problems. I will let you know when that is done

[gmta]

Alright, but just to make sure: my log files are locally accessed, not over a network share.

[RolandPheasant]

Blimey! I will add a load of more logging to help get to the bottom of this.

[RolandPheasant]

Again thanks for the input

[RolandPheasant]

@gmta I have done lots of work on performance and found a few issues clearly slowing things down. The latest release (see v0.2.2) should be a lot quicker. I would appreciate if you take a look.

At the weekend I will be doing a release which according to my tests should be highly performing even for gigabyte sized files.

If these don't fix the performance for you than perhaps it could be a graphics issue

[gmta]

@RolandPheasant Performance has improved! I'll be testing some more tomorrow.

[RolandPheasant]

👍 Thanks for checking. Someone in work just showed me a folder full of 20 Gigabyte sized log files so handling files that size has become my new challenge.

[RolandPheasant]

@gmta version 0.3 should be much much better. Please try and let me know how it goes.

[gmta]

I've been testing version 0.5 and there are two major issues in the way it handles tailing my log files:

• CPU usage is maxing out on a moderately updated log file (<1 MB, sane line lengths)
• Sometimes the UI seems to "hang" on a new line, becoming jittery and resuming normal operation after about half a second. This also happens when scrolling through files without updates happening.

[RolandPheasant]

The only time I have seen performance suffer is when there are very long lines in the log file. Is this the case for you? Is so then I am aware of the issue.

Is it possible for you to upload a sample file here so I can open it on my machine, or is the content sensitive?

[RolandPheasant]

@gmta My suspicion was correct. I generated a file with lines which are greater than 10kB each and Tail Blazer ground to a halt

Now I know what the issue is I will work out a solution