Export not working

[luifire]

Hi all
I want to export a query to a newline-delimited JSON.
I do this for a past day, there are no new data coming in nor is data being deleted (retention is way longer).
When I start the function it start the export of filtered logs only a few minutes are downloaded, but it should be an entire day.
When I start the same function again, it stops at another time. So sometimes it is 6 minutes, 13 minutes, 2 minutes, 20 minutes, ...
The amount of data clearly does not change.

The export as CSV I get in the files
"Error"
"Runtime limit exceeded: exceeded working set limit of 3435920179 bytes (observed 7520768944 bytes)."

[liammclennan]

Hi @luifire,

I haven't found that error message. Which version of Seq are you using?

[luifire]

The error message is written within the CSV file.
The SEQ version is: 2024.1.11028

I couldn't find anything related to this issue in the full release milestones and IT only wants to update, if they are sure the issue is fixed.

[nblumhardt]

Hi @luifire, thanks for the additional info!

This error is approximately an "out of memory" error that Seq raises when computing the query result takes more memory than deemed safe for a single request.

It might be possible to rewrite the query into a more efficient form, but otherwise, running the query over shorter time windows is your best bet. If you have a supported Seq subscription and need some help with this, feel free to send the query text through to support@datalust.co along with a description of the data, and we'll take a look.

2024.1 is now a few versions out-of-date so upgrading is worthwhile when you're able to.

Hope this helps!
Nick

[luifire]

Hi Nick
Thanks for your response.

I split the query in smaller portions (which was quite tedious) and the resulting data have size 454 MB, which is probably safe enough?
This time the CSV says:
"The result would exceed the internal working limit of 100000 rows."
I get that you want to be rather safe the exception, but considering that this is only 450MB, I don't see why there is this limitation?

I don't think one can optimize much on the query, it is of type ("X" or "Y" or "Z") and not "A" and I need every line retrieved.

I will talk to IT to see if we could update.

Thanks again
David