Recursion error in periodic data removal

[LaurensBurger]

Product versie / Product version

All

Sentry 321138

RecursionError: maximum recursion depth exceeded while calling a Python object
  File "openforms/data_removal/tasks.py", line 31, in delete_submissions
    successful_submissions_to_delete.delete()
  File "django/db/models/query.py", line 745, in delete
    collector.collect(del_query)
  File "django/db/models/deletion.py", line 295, in collect
    if sub_objs:
  File "django/db/models/query.py", line 284, in __bool__
    self._fetch_all()
  File "django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "django/db/models/query.py", line 69, in __iter__
    obj = model_cls.from_db(db, init_list, row[model_fields_start:model_fields_end])
  File "django/db/models/base.py", line 515, in from_db
    new = cls(*values)
  File "openforms/submissions/models/submission_step.py", line 134, in __init__
    if not self.form_step_id:
  File "django/db/models/query_utils.py", line 144, in __get__
    instance.refresh_from_db(fields=[field_name])
  File "django/db/models/base.py", line 650, in refresh_from_db
    db_instance = db_instance_qs.get()
  File "django/db/models/query.py", line 431, in get
    num = len(clone)
  File "django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "django/db/models/query.py", line 69, in __iter__
    obj = model_cls.from_db(db, init_list, row[model_fields_start:model_fields_end])
  File "django/db/models/base.py", line 515, in from_db
    new = cls(*values)
  File "openforms/submissions/models/submission_step.py", line 135, in __init__
    self.form_step = self._load_form_step_from_history()
  File "openforms/submissions/models/submission_step.py", line 141, in _load_form_step_from_history
    history = deepcopy(self.form_step_history)
  File "django/db/models/query_utils.py", line 144, in __get__
    instance.refresh_from_db(fields=[field_name])
  File "django/db/models/base.py", line 650, in refresh_from_db
    db_instance = db_instance_qs.get()
  File "django/db/models/query.py", line 431, in get
    num = len(clone)
  File "django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "django/db/models/query.py", line 69, in __iter__
    obj = model_cls.from_db(db, init_list, row[model_fields_start:model_fields_end])
  File "django/db/models/base.py", line 515, in from_db
    new = cls(*values)
  File "openforms/submissions/models/submission_step.py", line 134, in __init__
    if not self.form_step_id:
  File "django/db/models/query_utils.py", line 144, in __get__
    instance.refresh_from_db(fields=[field_name])
  File "django/db/models/base.py", line 650, in refresh_from_db
    db_instance = db_instance_qs.get()
  File "django/db/models/query.py", line 431, in get
    num = len(clone)
  File "django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "django/db/models/query.py", line 69, in __iter__
    obj = model_cls.from_db(db, init_list, row[model_fields_start:model_fields_end])
  File "django/db/models/base.py", line 515, in from_db
    new = cls(*values)
  File "openforms/submissions/models/submission_step.py", line 135, in __init__
    self.form_step = self._load_form_step_from_history()
  File "openforms/submissions/models/submission_step.py", line 141, in _load_form_step_from_history
[...]

Omschrijf het probleem / Describe the bug

Error can be found in Sentry. Originates from delete_submission and can be found in all deployed instances

[sergei-maertens]

Very much related to #2305 and #2135

[sergei-maertens]

@joeribekker I'd like to make a case again for snapshotting the entire state/configuration of the form inside the submission so we can get rid of this less-than-optimal historical step configuration workaround, it's unpredictable

[joeribekker]

@joeribekker I'd like to make a case again for snapshotting the entire state/configuration of the form inside the submission so we can get rid of this less-than-optimal historical step configuration workaround, it's unpredictable

I agree this can be one of the big topics, but seems less urgent than others.

[LaurensBurger]

I'm afraid this bug is causing submissions to not be deleted at all when it occurs, iv'e checked multiple instances and the rules set in the general config don't seem to apply anymore for all instances where this bug occurs.

[joeribekker]

Alright, prio re-evaluated. I need someone to explain to me how this works and @sergei-maertens how much work the state-save means.

[joeribekker]

Refinement: Timebox this on 1 day to see if we can have a bandaid fix for this that at least doesn't break the entire task so nothing gets cleaned up. If not possible, we go for the full solution (inform us in the next refinement): Keep the state of forms properly in the submission.

[sergei-maertens]

I've found a bandaid fix :)