Fix cascade delete of submission step

[sergei-maertens]

Fixes #2135

This is a bandaid with some nasty code, we definitely need to discuss this properly what it should look like for 2.0. (see #2305)

The gist is:

• the SubmissionStep.form_step FK is now nullable
• if the formstep would have cascade deleted the submission step, then we serialize the form step + definition (using django's serializers that power fixture dumping) and record it on the SubmissionStep instance
• When the FK is empty, we then load the historical data from the record history (essentially load the fixture in memory)

⚠️ This will not survive the test of time - recorded fixtures cannot anticipate database schema changes, so it won't be guaranteed forwards compatible.

⚠️ This abuses Django's deletion collector API (which is private afaik) to schedule the updates on delete in a reliable manner. We have to use frozenset because of internal implementation details and I don't suspect this will be fixed in Django for this use case.

[SilviaAmAm]

If I understand correctly, loading the form step from the history on the submission step is only needed if the form is restored from an older version while a user is filling in the form?
For all other purposes, if the submission step has no form_step FK it is not a problem?

[SilviaAmAm]

Does this also happen in import export (I thought it was only deleting the form step or restoring a form from an older version)?
During the export all the form resources are turned to JSON (i don't think anything is being deleted?) and during the import, the form should be new, so there is no submission step data that could be associated with it?

[sergei-maertens]

Does this also happen in import export (I thought it was only deleting the form step or restoring a form from an older version)? During the export all the form resources are turned to JSON (i don't think anything is being deleted?) and during the import, the form should be new, so there is no submission step data that could be associated with it?

I'll have to double check that for exactly which mechanism is in play, but it might be that you're importing over an existing form and updating it that way? The original issue reported it being a problem with imports, though I was able to reproduce with a simplified case of deleting the form step.

[sergei-maertens]

so, when restoring old versions of a form, this flow is hit - there is an existing form instance, which makes you hit this codepath (via https://github.com/open-formulieren/open-forms/blob/master/src/openforms/forms/models/form.py#L434):

https://github.com/open-formulieren/open-forms/blob/stable/1.1.x/src/openforms/forms/utils.py#L126

I'll add a testcase that essentially operates on Form.restore_old_version to ensure this is also covered by the patch.

[sergei-maertens]

If I understand correctly, loading the form step from the history on the submission step is only needed if the form is restored from an older version while a user is filling in the form? For all other purposes, if the submission step has no form_step FK it is not a problem?

Correct!

edit: no, this applies basically as soon as there is a submission, but you'll typically notice it when looking at completed submissions

[codecov-commenter]

Codecov Report

Base: 92.45% // Head: 92.45% // Decreases project coverage by -0.00% ⚠️

Coverage data is based on head (7d1eb9f) compared to base (e160e7e).
Patch coverage: 91.78% of modified lines in pull request are covered.

Additional details and impacted files

@@               Coverage Diff                @@
##           stable/1.1.x    #2302      +/-   ##
================================================
- Coverage         92.45%   92.45%   -0.01%     
================================================
  Files               531      531              
  Lines             16415    16483      +68     
  Branches           1471     1491      +20     
================================================
+ Hits              15177    15239      +62     
- Misses              978      981       +3     
- Partials            260      263       +3     

Impacted Files	Coverage Δ
...rc/openforms/submissions/models/submission_step.py	93.25% <90.00%> (-6.75%)	⬇️
src/openforms/submissions/models/submission.py	96.07% <100.00%> (+0.14%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.