[FR] Import tool refactor

[matmair]

Is your feature request the result of a bug?
#2341
#3098
#3097
#6321

Problem
We have several ways to do imports - we should refactor imports out.

Suggested solution
Two principles:

• API first
• backend-run first

Backend:

• Switch to data-wizard
• Refactor all import code into a seperate app
• Clean up / limit exposed APIs from data-wizard
• Create backend worker for django-q
• Create frontend that consumes the new APIs to build up an import flow

Refactor code base

• Refactor BOM importer [BUG] 0.7.0 BOM Import fails to match parts by Part ID #3097
• Refactor PO importer [BUG] 0.7.0 PO Import tools issues #3098
• Refactor SO importer
• Refactor Part importer

Unit tests:

• E2E Tests for import flow
• E2E for all refactored importers based on static import files that are in git

Describe alternatives you've considered
We should fix this, not sure what release to target though.

Do you want to develop this?
Yes - this will be something for the whole team I think.

---

Outstanding Tasks

• Support "default values" when importing dataset
• Allow user to navigate back to previous data steps
• Add unit test for cleanup backup task
• Ensure user has correct permissions before creating a new import session
• Expose import / export functions to plugins
• Allow plugins to define custom export fields / processes
• Allow bulk data export action to django admin interface
• Replace existing "BOM Import" tool with new importer
• Replace existing "BOM export" tool with new exporter
• Replace existing "part import" tool with new importer
• Replace existing "purchase order import" tool with new importer
• Remove django-import-export framework entirely
• Allow bulk "update" via import (override current records, not create new ones)
• Remove all custom import UI template code (html templates, legacy)
• Check that all existing "django-import-export" models are covered by new framework
• Show "original row data" for each row when importing data
• Allow user to create new related model "in place"
• Add "context specific" importers for particular tables, rather than just in the admin center
• Improve efficiency of data import process (currently takes ~1 minute for 1,000 records)
• Impove user interface for data import - in particular, react hook efficiency
• Enable per-column specification of "default" values in the admin portal
• Documentation for new data import process

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[SchrodingersGat]

This one feels like a high priority once we've got the current backlog of devops stuff out of the way

[lukasbrchl]

Hi @SchrodingersGat, would it please be possible to have at least the basic functionality of the import tool fixed for 0.8.0? I mean just import the parts from CSV and create orders from them as we had before. I know there is a lot of work for this, but this functionality is really important fur us. Thanks!

[SchrodingersGat]

Hi @lukasbrchl nothing planned at this stage but maybe your company would be willing to sponsor some development time on this? Please reach out to me if this would be an option for you.

[matmair]

@SchrodingersGat this will be important for #3901 / #2789 - our current functionality is not set up to work through the API so it can not be easily decoupled from the integrated frontend

[SchrodingersGat]

Yep, the import tool definitely needs to work via the API :)

[matmair]

Would be easier once #4304 is done as there is already a worker for celery.

[matmair]

https://www.reddit.com/r/InvenTree/comments/1410bvf/bulk_part_import/