Slow stop when using a lot of memory

[DanAvni]

I have a Seq instance configured to use 140GB of memory. When trying to stop the service it can take well over 5 minutes as the memory is slowly being released until the service stops and the process terminates. This makes upgrading Seq very slow as the installer sometimes times out waiting for Seq to stop. Please make Seq stop faster.

On my case the installer showed this after about 5 min of waiting

[KodrAus]

Hi @DanAvni, which version of Seq are you referring to here? 2023 or earlier?

[DanAvni]

2023.1

[KodrAus]

Thanks, it sounds like Windows might be taking its time unmapping open files. We’ve been talking about changing the behaviour around mapping that might have an impact on this.

[nblumhardt]

There's also the possibility of apps taking a long time to stop due to queued events to process. Checking out the logs from around the time of shutdown would show this.

@DanAvni to test out @KodrAus 's theory, you can update to 2023.2 and run:

seq config set -k process.aggressiveReleaseMemory -v True
seq service restart

If that sorts the issue out (on the next shutdown, not the restart that applies the setting :-)) please let us know, it'd be great data to have.

Otherwise, if it appears to cause problems of any kind, you can revert it with:

seq config clear -k process.aggressiveReleaseMemory
seq service restart

2023.2 is otherwise very very similar internally to 2023.1, so you shouldn't expect any compatibility issues or behavior changes due to the upgrade.

Thanks as usual for the feedback/info, much appreciated.

[DanAvni]

I configured the aggressiveReleaseMemory and on my VM with 140GB Seq was now only using about 4GB as opposed to 140GB a few days ago and at about the same server time. When trying to stop the service it stopped in a few seconds.

[nblumhardt]

That sounds very promising - thanks for the follow-up, Dani.

Unless you notice any unexpected side-effects, it should be fine to leave that setting in place. Its effect will be for Seq to unmap storage files immediately after performing queries. Windows Server should normally keep these hot in the page cache anyway, but it's possible under some circumstances it might lead to more I/O being required.

[KodrAus]

Hi @DanAvni 👋 Just following up here to see how things are going since you set aggresiveReleaseMemory. Does your system seem to be performing at least equivalently to how it was before?

[DanAvni]

Hi @KodrAus , So far it looks ok and I did not get any performance complaints

[KodrAus]

Thanks for the update @DanAvni. We’ve been planning in making that he default behavior in a future release so that’s a really helpful datapoint.

[nblumhardt]

Seq 2023.3 will make "aggressive release memory" the default on all operating systems.