error on PUT to /api/v2/forms/{uuid_or_slug}/variables issues (investigating) #4824
Comments
LaurensBurger
added
bug
triage
joeribekker
removed
the
triage
|
@LaurensBurger is the form with ID 608 the right one? I see many more fields in that one and no
Also if I save the form I get a different error than the one you got: |
|
@stevenbal sorry I've messed a bit with it in the meantime - i'll bring it back to the original state. (should be back to the state of showing 2 dummy fields) The error is just a way to trigger the state, looking at Sentry there are many ways to trigger it - i used the "softRequiredErrors" to trigger the state because i have hard time reproducing the most common issue in my example above. (also the way i trigger it seems to also break navigating the form. But you can still observe the effect if you pauze the form after filling in all fields and looking at the submission, dummy and dummy 1 should be missing) |
|
This might affect submissions as well as building the form? Hard to estimate the impact. We'll make a database dump to reproduce the exact issue since locally this didnt seem to happen. |
|
I am now able to reproduce this locally with the database dump @LaurensBurger for my understanding, the issue here is that when the form is in this broken state, you're not able to go to the next step, right? If I "fix" the form by removing the EDIT: nvm, I changed the form a bit and you're right, it seems that when the form has any errors in the variables tab when saving, componentvariables are not created and it seems the submissionstep is also not created. I'll look into it (might have something to do with the |
|
I've been able to reproduce this reliably:
Because the component was newly added, no variable existed for it yet and because of the validation errors on the variables endpoint, no new variable is created either. The formdefinition is already saved though, so it does appear in the form. I'm not sure what would be the correct way to fix this, I don't think it'd be possible/desired to roll back the form definition with the added component because the variable tab fails. We could automatically put the form into maintenance mode until the errors are resolved (thinking out loud here)? I guess we could at least add a check to the email digest that detects forms for which there are missing formvariables, to make sure that it is clear that these need attention |
|
Discussed with Sergei: ideally we would fix this by having a single API endpoint for the entire form + formsteps + variables etc., but this is a big refactor that would be an epic. For now we will go for a band aid solution by scheduling the task that ensures creation of FormVariables for components when the FormSteps are saved, this way the variables will be created regardless of errors when making requests to the variables endpoint Also created a separate issue for the softRequiredErrors raising errors: #4884 |
stevenbal
added
the
needs-backport
stevenbal
added a commit
that referenced
this issue
Dec 9, 2024
this ensures that the Form cannot be left in a broken state with regards to FormVariables, because every component will always have a FormVariable this is a bandaid fix, because ideally we would have a single endpoint and the FormStep wouldn't be updated if there are errors with variables
stevenbal
added a commit
that referenced
this issue
Dec 9, 2024
9 tasks
stevenbal
added a commit
that referenced
this issue
Dec 10, 2024
this ensures that the Form cannot be left in a broken state with regards to FormVariables, because every component will always have a FormVariable this is a bandaid fix, because ideally we would have a single endpoint and the FormStep wouldn't be updated if there are errors with variables
stevenbal
added a commit
that referenced
this issue
Dec 10, 2024
stevenbal
added a commit
that referenced
this issue
Dec 10, 2024
this ensures that the Form cannot be left in a broken state with regards to FormVariables, because every component will always have a FormVariable this is a bandaid fix, because ideally we would have a single endpoint and the FormStep wouldn't be updated if there are errors with variables
stevenbal
added a commit
that referenced
this issue
Dec 10, 2024
stevenbal
added a commit
that referenced
this issue
Dec 10, 2024
sergei-maertens
added a commit
that referenced
this issue
Dec 12, 2024
…mvariable-for-component 🐛 [#4824] Schedule task to create FormVariables after saving FormStep
sergei-maertens
pushed a commit
that referenced
this issue
Dec 12, 2024
this ensures that the Form cannot be left in a broken state with regards to FormVariables, because every component will always have a FormVariable this is a bandaid fix, because ideally we would have a single endpoint and the FormStep wouldn't be updated if there are errors with variables Backport-of: #4893
sergei-maertens
pushed a commit
that referenced
this issue
Dec 12, 2024
this ensures that the Form cannot be left in a broken state with regards to FormVariables, because every component will always have a FormVariable this is a bandaid fix, because ideally we would have a single endpoint and the FormStep wouldn't be updated if there are errors with variables Backport-of: #4893
sergei-maertens
added a commit
that referenced
this issue
Jan 30, 2025
Before this patch, we ran some reconciliation when a form is edited to ensure that existing SubmissionValueVariable instances don't have 'null' form_variable foreign keys because the FormVariable set of a form was updated after editing (a) step(s) - the idea here was to be eventually consistent. This turns out not to be necessary (if we can trust our test suite) because the load_submission_value_variables_state method on the Submission operates on the variable keys rather than the FK relation and is able to properly resolve everything. This is also the interface that developers should use when accessing the submission values and it appears to be done properly in the registration backends, otherwise tests would likely fail. This re-coupling was extended in #4900, after it was noticed in #4824 that the re-coupling didn't happen for other forms that use the same re-usable form definition. At the time, we didn't understand how this seemingly didn't cause issues or at least didn't result in issues being reported to us, but we can now conclude that it just wasn't a problem in the first place because the proper interfaces/service layer are/were being used and everything is done/reconciled in-memory when comparing/populating submission variables with form variables. A further cleanup step will then also be to remove this FK field from the submission value variable model, as it is not necessary.
sergei-maertens
added a commit
that referenced
this issue
Jan 31, 2025
Before this patch, we ran some reconciliation when a form is edited to ensure that existing SubmissionValueVariable instances don't have 'null' form_variable foreign keys because the FormVariable set of a form was updated after editing (a) step(s) - the idea here was to be eventually consistent. This turns out not to be necessary (if we can trust our test suite) because the load_submission_value_variables_state method on the Submission operates on the variable keys rather than the FK relation and is able to properly resolve everything. This is also the interface that developers should use when accessing the submission values and it appears to be done properly in the registration backends, otherwise tests would likely fail. This re-coupling was extended in #4900, after it was noticed in #4824 that the re-coupling didn't happen for other forms that use the same re-usable form definition. At the time, we didn't understand how this seemingly didn't cause issues or at least didn't result in issues being reported to us, but we can now conclude that it just wasn't a problem in the first place because the proper interfaces/service layer are/were being used and everything is done/reconciled in-memory when comparing/populating submission variables with form variables. A further cleanup step will then also be to remove this FK field from the submission value variable model, as it is not necessary.
sergei-maertens
added a commit
that referenced
this issue
Jan 31, 2025
Before this patch, we ran some reconciliation when a form is edited to ensure that existing SubmissionValueVariable instances don't have 'null' form_variable foreign keys because the FormVariable set of a form was updated after editing (a) step(s) - the idea here was to be eventually consistent. This turns out not to be necessary (if we can trust our test suite) because the load_submission_value_variables_state method on the Submission operates on the variable keys rather than the FK relation and is able to properly resolve everything. This is also the interface that developers should use when accessing the submission values and it appears to be done properly in the registration backends, otherwise tests would likely fail. This re-coupling was extended in #4900, after it was noticed in #4824 that the re-coupling didn't happen for other forms that use the same re-usable form definition. At the time, we didn't understand how this seemingly didn't cause issues or at least didn't result in issues being reported to us, but we can now conclude that it just wasn't a problem in the first place because the proper interfaces/service layer are/were being used and everything is done/reconciled in-memory when comparing/populating submission variables with form variables. A further cleanup step will then also be to remove this FK field from the submission value variable model, as it is not necessary.
sergei-maertens
added a commit
that referenced
this issue
Jan 31, 2025
**Removed the submission value variable recoupling** Before this patch, we ran some reconciliation when a form is edited to ensure that existing SubmissionValueVariable instances don't have 'null' form_variable foreign keys because the FormVariable set of a form was updated after editing (a) step(s) - the idea here was to be eventually consistent. This turns out not to be necessary (if we can trust our test suite) because the load_submission_value_variables_state method on the Submission operates on the variable keys rather than the FK relation and is able to properly resolve everything. This is also the interface that developers should use when accessing the submission values and it appears to be done properly in the registration backends, otherwise tests would likely fail. This re-coupling was extended in #4900, after it was noticed in #4824 that the re-coupling didn't happen for other forms that use the same re-usable form definition. At the time, we didn't understand how this seemingly didn't cause issues or at least didn't result in issues being reported to us, but we can now conclude that it just wasn't a problem in the first place because the proper interfaces/service layer are/were being used and everything is done/reconciled in-memory when comparing/populating submission variables with form variables. A further cleanup step will then also be to remove this FK field from the submission value variable model, as it is not necessary. **Run form variable sync in same transaction** Offloading this to celery results in more, separate tasks each competing for database locks and can lead to integrity errors. Changing the business logic of form variable creation/update to make sure a FormDefinition write results in a sync action gives a single trigger for these kind of changes, and it's responsible for updating all the forms that are affected by it. Since multiple FormDefinition/FormStep writes happen in parallel because of parallel client requests, we can only update or create variables and can't delete variables that are not present in our current form definition, as they may belong to another request. This shouldn't immediately cause problems, as old, unused variables don't have an 'execution path'. We piggy back on the variables bulk update endpoint/transaction to clean these up, as that call is made after all the form step persistence succeeded so we know that everything is resolved by then. The left-over variables problem only exists when updating a form step to use a different form definition. It's not relevant when a form definition is updated through the admin or a form step is added to a form. **Add test for expected new behaviour of bulk variable update endpoint** The bulk update endpoint may no longer manage the variables for form steps, as that's taken care of by the form step endpoint now. But, user defined variables must still be managed. **Use the form variables bulk update for consistency** Instead of triggering celery tasks, perform the variable state synchronization for a form in the bulk update endpoint. This endpoint now still validates the state of all component/user defined variables, but no longer drops all the variables and re-creates them, instead it only touches user defined variables and leaves the form step variables alone. Since this is called after the steps have been sorted out, a complete view of left-over variables from earlier form definitions is available and those can be cleaned up safely now. Backport-of: #5064
sergei-maertens
added a commit
that referenced
this issue
Jan 31, 2025
**Removed the submission value variable recoupling** Before this patch, we ran some reconciliation when a form is edited to ensure that existing SubmissionValueVariable instances don't have 'null' form_variable foreign keys because the FormVariable set of a form was updated after editing (a) step(s) - the idea here was to be eventually consistent. This turns out not to be necessary (if we can trust our test suite) because the load_submission_value_variables_state method on the Submission operates on the variable keys rather than the FK relation and is able to properly resolve everything. This is also the interface that developers should use when accessing the submission values and it appears to be done properly in the registration backends, otherwise tests would likely fail. This re-coupling was extended in #4900, after it was noticed in #4824 that the re-coupling didn't happen for other forms that use the same re-usable form definition. At the time, we didn't understand how this seemingly didn't cause issues or at least didn't result in issues being reported to us, but we can now conclude that it just wasn't a problem in the first place because the proper interfaces/service layer are/were being used and everything is done/reconciled in-memory when comparing/populating submission variables with form variables. A further cleanup step will then also be to remove this FK field from the submission value variable model, as it is not necessary. **Run form variable sync in same transaction** Offloading this to celery results in more, separate tasks each competing for database locks and can lead to integrity errors. Changing the business logic of form variable creation/update to make sure a FormDefinition write results in a sync action gives a single trigger for these kind of changes, and it's responsible for updating all the forms that are affected by it. Since multiple FormDefinition/FormStep writes happen in parallel because of parallel client requests, we can only update or create variables and can't delete variables that are not present in our current form definition, as they may belong to another request. This shouldn't immediately cause problems, as old, unused variables don't have an 'execution path'. We piggy back on the variables bulk update endpoint/transaction to clean these up, as that call is made after all the form step persistence succeeded so we know that everything is resolved by then. The left-over variables problem only exists when updating a form step to use a different form definition. It's not relevant when a form definition is updated through the admin or a form step is added to a form. **Add test for expected new behaviour of bulk variable update endpoint** The bulk update endpoint may no longer manage the variables for form steps, as that's taken care of by the form step endpoint now. But, user defined variables must still be managed. **Use the form variables bulk update for consistency** Instead of triggering celery tasks, perform the variable state synchronization for a form in the bulk update endpoint. This endpoint now still validates the state of all component/user defined variables, but no longer drops all the variables and re-creates them, instead it only touches user defined variables and leaves the form step variables alone. Since this is called after the steps have been sorted out, a complete view of left-over variables from earlier form definitions is available and those can be cleaned up safely now. Backport-of: #5064
sergei-maertens
added a commit
that referenced
this issue
Jan 31, 2025
**Removed the submission value variable recoupling** Before this patch, we ran some reconciliation when a form is edited to ensure that existing SubmissionValueVariable instances don't have 'null' form_variable foreign keys because the FormVariable set of a form was updated after editing (a) step(s) - the idea here was to be eventually consistent. This turns out not to be necessary (if we can trust our test suite) because the load_submission_value_variables_state method on the Submission operates on the variable keys rather than the FK relation and is able to properly resolve everything. This is also the interface that developers should use when accessing the submission values and it appears to be done properly in the registration backends, otherwise tests would likely fail. This re-coupling was extended in #4900, after it was noticed in #4824 that the re-coupling didn't happen for other forms that use the same re-usable form definition. At the time, we didn't understand how this seemingly didn't cause issues or at least didn't result in issues being reported to us, but we can now conclude that it just wasn't a problem in the first place because the proper interfaces/service layer are/were being used and everything is done/reconciled in-memory when comparing/populating submission variables with form variables. A further cleanup step will then also be to remove this FK field from the submission value variable model, as it is not necessary. **Run form variable sync in same transaction** Offloading this to celery results in more, separate tasks each competing for database locks and can lead to integrity errors. Changing the business logic of form variable creation/update to make sure a FormDefinition write results in a sync action gives a single trigger for these kind of changes, and it's responsible for updating all the forms that are affected by it. Since multiple FormDefinition/FormStep writes happen in parallel because of parallel client requests, we can only update or create variables and can't delete variables that are not present in our current form definition, as they may belong to another request. This shouldn't immediately cause problems, as old, unused variables don't have an 'execution path'. We piggy back on the variables bulk update endpoint/transaction to clean these up, as that call is made after all the form step persistence succeeded so we know that everything is resolved by then. The left-over variables problem only exists when updating a form step to use a different form definition. It's not relevant when a form definition is updated through the admin or a form step is added to a form. **Add test for expected new behaviour of bulk variable update endpoint** The bulk update endpoint may no longer manage the variables for form steps, as that's taken care of by the form step endpoint now. But, user defined variables must still be managed. **Use the form variables bulk update for consistency** Instead of triggering celery tasks, perform the variable state synchronization for a form in the bulk update endpoint. This endpoint now still validates the state of all component/user defined variables, but no longer drops all the variables and re-creates them, instead it only touches user defined variables and leaves the form step variables alone. Since this is called after the steps have been sorted out, a complete view of left-over variables from earlier form definitions is available and those can be cleaned up safely now. Backport-of: #5064
sergei-maertens
added a commit
that referenced
this issue
Jan 31, 2025
**Removed the submission value variable recoupling** Before this patch, we ran some reconciliation when a form is edited to ensure that existing SubmissionValueVariable instances don't have 'null' form_variable foreign keys because the FormVariable set of a form was updated after editing (a) step(s) - the idea here was to be eventually consistent. This turns out not to be necessary (if we can trust our test suite) because the load_submission_value_variables_state method on the Submission operates on the variable keys rather than the FK relation and is able to properly resolve everything. This is also the interface that developers should use when accessing the submission values and it appears to be done properly in the registration backends, otherwise tests would likely fail. This re-coupling was extended in #4900, after it was noticed in #4824 that the re-coupling didn't happen for other forms that use the same re-usable form definition. At the time, we didn't understand how this seemingly didn't cause issues or at least didn't result in issues being reported to us, but we can now conclude that it just wasn't a problem in the first place because the proper interfaces/service layer are/were being used and everything is done/reconciled in-memory when comparing/populating submission variables with form variables. A further cleanup step will then also be to remove this FK field from the submission value variable model, as it is not necessary. **Run form variable sync in same transaction** Offloading this to celery results in more, separate tasks each competing for database locks and can lead to integrity errors. Changing the business logic of form variable creation/update to make sure a FormDefinition write results in a sync action gives a single trigger for these kind of changes, and it's responsible for updating all the forms that are affected by it. Since multiple FormDefinition/FormStep writes happen in parallel because of parallel client requests, we can only update or create variables and can't delete variables that are not present in our current form definition, as they may belong to another request. This shouldn't immediately cause problems, as old, unused variables don't have an 'execution path'. We piggy back on the variables bulk update endpoint/transaction to clean these up, as that call is made after all the form step persistence succeeded so we know that everything is resolved by then. The left-over variables problem only exists when updating a form step to use a different form definition. It's not relevant when a form definition is updated through the admin or a form step is added to a form. **Add test for expected new behaviour of bulk variable update endpoint** The bulk update endpoint may no longer manage the variables for form steps, as that's taken care of by the form step endpoint now. But, user defined variables must still be managed. **Use the form variables bulk update for consistency** Instead of triggering celery tasks, perform the variable state synchronization for a form in the bulk update endpoint. This endpoint now still validates the state of all component/user defined variables, but no longer drops all the variables and re-creates them, instead it only touches user defined variables and leaves the form step variables alone. Since this is called after the steps have been sorted out, a complete view of left-over variables from earlier form definitions is available and those can be cleaned up safely now. Backport-of: #5064
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Product versie / Product version
2.7.x and higher (reproduced on our test env)
Customer reference
No response
Omschrijf het probleem / Describe the bug
Seeing cases where errors while saving a form ("variables") lead to a form where no input will be stored in de submission for the new variables.
This can bee seen in the form itself as navigating back to these steps will simply show empty fields that previously held data.
These specific variables are also missing in the variables tab of the form.
These variables are "present" in the form builder, they also show in the front-end of the form but they are missing in the submission, and consequently trigger a validation error if they are required.
This is either showing up more frequent now because of BE validation, or 2.7 introduced a bug where saving of variables has changed?
suspected ValidationErrors look something like this
[{}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {'key': [ErrorDetail(string='Ongeldige componentvariabele: er bestaat geen component met de opgegeven sleutel in de formulierdefinitie.', code='invalid')]}, {'key': [ErrorDetail(string='Ongeldige componentvariabele: er bestaat geen component met de opgegeven sleutel in de formulierdefinitie.', code='invalid')]}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}]This would suggest this form now contains 2 "ghost" variables that unless refreshed by the from builder and saved again, will be missing from every submission
maykin-test form 608 is currently in this state - it contains 2 fields "dummyveld" and "dummyveld1" these were added with a "bad" component that triggered a error.
Both fields show up in the form designer and front-end, but are not present when creating a submission.
Stappen om te reproduceren / Steps to reproduce
No response
Verwacht gedrag / Expected behavior
No response
Screen resolution
None
Device
None
OS
None
Browser
No response
The text was updated successfully, but these errors were encountered: