Server utilisation

[Timothyoverton]

Hi Team,

We have a dedicated VM for Seq in production. Windows server, 32Gb ram, lots of cpu. Running version: Seq-2024.3.11510.

Ops team have had numerous reports on query performance as of late. And looking at resource usage, ram usage of the Seq service is barely above 3Gb. By going through the Seq issues, I can see a drastic change has happened, and now Seq doesn't cache events in memory and it appears as though the load is now being put onto the file system when querying. Is that correct? Is it possible to enable the old functionality, this server is dedicated for Seq? We now have a sizable VM running for no reason, what would your best practice recommendations be?

I've looked at these issues:
#1822
#1869
#2038

Our config:
our process.workingSetLimitBytes is not set
our cache.systemRamTarget setting doesn't exist

Indexes:
Total index storage 10 GB
Indexing time 1.1 hours last 24 hours

Statistics (All Time):
Buffered 433 MB
Unindexed 0 bytes
Indexed 75 GB
Queued for reindexing 543 Gb
Indexes 10 Gb

[nblumhardt]

Hi @Timothyoverton, thanks for your message!

The memory shown as "unused" will normally be holding file pages for the Windows's virtual memory manager, so the Seq event data cached in RAM should generally be the same or higher than on previous versions. Patterns of caching are different, however, so the effectiveness of the cache might change based on what kinds of queries are being run.

I think the best way to dig in further would be to look at the specific queries that are timing out, especially the time ranges they're running over and the signals that are applied. Shooting some examples through here, or via support@, would be great.

It might also be useful to look at Data > Storage, and check out how your total data volume is changing over time. If you're ingesting and retaining more now than in the past, you might need to look for bottlenecks in the system configuration, e.g. the throughput and IOPS of the attached disks. If any operation needs to touch a significant slice of that half-terabyte, I/O speed will usually be the limiting factor.

Hope this helps, and please keep us posted!
Nick

[Timothyoverton]

Thanks Nick,

I'm not sure what changed, the queries are still the same. We are going to try a premium disk and improve the IOPS capability of the VM. The issue seems to be the disk speed.

I'll let you know,
Tim