Cannot generate csv files larger than ~1 GB

[BeritJanssen]

What went wrong?

When trying to download a larger dataset from the People & Parliament production server, I noticed that it's not possible to generate .csv files larger than ca. 1 GB. The downloads overview will show "error" after a while, and the celery logs show a SIGKILL (9). Probably caused by too much memory usage, though I couldn't see memory & CPU usage going up beyond 30% though.

What did you expect to happen?

I thought I could give researchers download rights beyond 1million documents, and they would be able to harvest large amounts of data in one go.

Screenshot

No response

Where did you find the bug?

• https://ianalyzer.hum.uu.nl
• https://peopleandparliament.hum.uu.nl
• https://peace.sites.uu.nl
• a server hosted elsewhere (i.e. not by the research software lab)
• a local server

Version

No response

Steps to reproduce

Attempt a download of >1million documents from parliament-netherlands with a user who has rights to do so.

[lukavdplas]

If a download of this size were generated without issue, the user would (probably) be unable to download it due to an Apache timeout. See #1474 - our decision for now was that we don't support downloads of this size.

That said, this describes an error in generating the file, which is a separate issue.

[BeritJanssen]

It is probably a memory issue - we would need to generate large files in batches, as well as serve them in batches (see #1474).