[Roadmap] Improve Data Export Functionality

[SchrodingersGat]

The data export functionality (i.e. exporting datasets to a file such as .csv or .xlsx) is currently very inefficient, to the point where it causes session timeout when exporting more than a few hundred records.

django-import-export

We use django-import-export to export records - however this has not been optimized and makes a LOT of database hits. In particular, it does not do any of the clever query annotations which we have implemented for the API. So, we have the ability to extract lots of records quickly (via queryset annotation) but django-import-export does not make use of this.

We can refactor the export process to use our optimized API serializers, and use this data for file export.

This has the huge advantage that the exported file data would then exactly match the data exported via the API - currently there is a discrepancy between the two export formats

Background Worker

We can also potentially offload the export process to the background worker thread. This would allow for longer running processes that do not get prematurely killed by short connection timeouts from the frontend worker...

related issues

• Timeout when downloading large parts list. #6376
• [BUG] 400 Bad Request when working with a larger BOM #2568
• Export of parts through admin fails with 502 Bad Gateway #5571
• Data import API #5125

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[matmair]

@SchrodingersGat is this in relation to #5125?

[SchrodingersGat]

@matmair yes, thanks, I have updated the related issues list now.