Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a registration backend that sends the form/submission variables as JSON to some API endpoint #4908

Closed
12 tasks done
sergei-maertens opened this issue Dec 12, 2024 · 4 comments · Fixed by #4956
Closed
12 tasks done

Comments

@sergei-maertens
Copy link
Member

sergei-maertens commented Dec 12, 2024

There is a desire for a "simple" registration backend/plugin which just dumps the submission/form variables with their values as a "flat" JSON structure and sends this data to an API endpoint for further processing. The goal is to keep this super simple and light weight, and let the post-processing be done on the receiving end (with an enterprise service bus or other technologies).

Requirements

Example

An example form with the following configuration:

  • Authentication via DigiD, capturing a BSN
  • First form step with text fields firstName and lastName
  • Second form step with a user upload attachment

Should lead to a JSON object being sent with the shape:

{
    "values":
    {
        "auth_bsn": "123456782",
        "firstName": "Donnie",
        "lastName": "Darko",
        "attachment": "<base64 encoded data>"
    },
    "schema":
    {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "type": "object",
        "properties":
        {
            "auth_bsn":
            {
                "type": "string",
                "pattern": "^\\d{9}$"
            },
            "firstName":
            {
                "type": "string"
            },
            "lastName":
            {
                "type": "string"
            },
            "attachment":
            {
                "type": "string",
                "contentEncoding": "base64"
            }
        },
        "required":
        [
            "auth_bsn",
            "firstName",
            "lastName"
        ],
        "additionalProperties": false
    }
}

Tasks

  • Define new registration plugin - it's probably wise to gate it behind a if settings.DEBUG since I don't expect this to be done for 3.0
  • Implement plugin configuration options:
    • Service to use
    • Endpoint to send data to (relative to service API root)
    • Form variables (including static variables!) to include
  • Add docker-compose mock service (flask app?) to simulate a receiving server
  • Implement sending the data to the configured service in the defined format, taking into account the configuration options
  • Add configuration checks to configuration overview
  • Ensure that we can generate an appropriate json-schema for each form component type
  • Add tests

Pointers

Existing code that can be useful:

  • service fetch
  • email registration plugin
  • objects API registration configuration/options
@sergei-maertens sergei-maertens added this to the Release 3.1.0 milestone Dec 12, 2024
@sergei-maertens sergei-maertens added waiting for approval An estimate was made but the stakeholder still needs to approve it. owner: venlo labels Dec 12, 2024
@viktorvanwijk viktorvanwijk moved this from Todo to In Progress in Development Dec 16, 2024
viktorvanwijk added a commit that referenced this issue Dec 18, 2024
viktorvanwijk added a commit that referenced this issue Dec 18, 2024
Plugin name did not make sense before
viktorvanwijk added a commit that referenced this issue Dec 18, 2024
API endpoint is relative, name should reflect that
viktorvanwijk added a commit that referenced this issue Dec 18, 2024
@viktorvanwijk
Copy link
Contributor

viktorvanwijk commented Dec 18, 2024

  • Optionally, a JSON Schema definition of the submitted data is provided in the schema key

Does 'optionally' mean that the configuration options should also include a checkbox (for example) with which you can select this?

@sergei-maertens
Copy link
Member Author

  • Optionally, a JSON Schema definition of the submitted data is provided in the schema key

Does 'optionally' mean that the configuration options should also include a checkbox (for example) with which you can select this?

No - it's a wish to include this, but having the actual data is the minimum requirement for this ticket. Including the JSON Schema is desired, but not crucial and should likely be the final task of this ticket to implement :)

viktorvanwijk added a commit that referenced this issue Dec 19, 2024
viktorvanwijk added a commit that referenced this issue Dec 19, 2024
Plugin name did not make sense before
viktorvanwijk added a commit that referenced this issue Dec 19, 2024
API endpoint is relative, name should reflect that
viktorvanwijk added a commit that referenced this issue Dec 19, 2024
viktorvanwijk added a commit that referenced this issue Dec 19, 2024
It should match the corresponding property of the serializer (which is 'service' and not 'service_select')
viktorvanwijk added a commit that referenced this issue Dec 19, 2024
Not relevant anymore or duplicates
viktorvanwijk added a commit that referenced this issue Dec 19, 2024
Wrong place and doesn't look great
viktorvanwijk added a commit that referenced this issue Dec 19, 2024
…n plugin

Need to include all form variables listed in the options to a values dictionary
viktorvanwijk added a commit that referenced this issue Dec 19, 2024
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
We check if the path contains '..' to prevent possible path traversal attacks. For example: '..', 'foo/..', '../foo', and 'foo/../bar' are all not allowed
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
Now supports multi selection, so we can use it here
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
In the JSON dump docker app, explicitly load the received data before sending again. This prevents the data from being interpreted as a string instead of a JSON object.
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
…n plugin

Need to include all form variables listed in the options to a values dictionary
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
The content of the attachment is now added to the result
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
Required for simulating a receiving service
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
* Add required properties
* Filter service queryset on ORC, as this is the only relevant type here
* Add minimum list length for the form variables: the configuration options could be saved without specifying any form variables to include. This does not make much sense. Added a minimum length of 1 for the form variables list in the serializer
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
…lugin

Just as an example of what will be implemented in #4980
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
* Rename plugin from JSON to JSONDump
* Add required and noManageChildProps to ServiceSelect and FormVariableSelect because they are required and need to prevent some Field props being passed down to the children
* Remove max length of FormioVariableKeyField: it is based on a text field which has no length constraints
* Remove id field from RelativeAPIEndpoint TextInput: is managed automatically
* Remove Required from JSONDumpOptions: required is the default for TypedDict
* Add default value for relative_api_endpoint: guarantees that the key is always present, which gives a strict type definition
* Remove content type headers: is set automatically when using the json kwargs
* Replace 'Included/Not Included' text with icons: makes it easier for the user to see whether it is included.
* Revise JSONDumpVariableConfigurationEditor: use useFormikContext instead of useField, and clean up the on-change event
* Add initial form data: formData will be an empty object if we add a new object, which means the default values for the configuration fields are missing
* Add missing form and static variable properties in JSONDump story
* Refactor building the values object of the to be submitted data: tt is neater to build up what needs to be processed first, before doing the processing
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
* The requests.Response object is not JSON serializable, so need to add just the json response (assuming it is json for now) of it .
* Convert the data to be sent to a json format. Needed because date/datetimes are not JSON serializable
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
There was a bug with the previous attachment processing, where attachments would get overwritten if multiple were uploaded. Now, an object will be returned with the 'file_name' and 'content' properties in case of a single file component (or 'None' if no file was uploaded). And in case of a multiple files component, it will be a list of these file objects (or an empty list if no file was uploaded). This keeps the data type consistent, and we can give guarantees that an object will always have the file_name and content keys.
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
We check if the path contains '..' to prevent possible path traversal attacks. For example: '..', 'foo/..', '../foo', and 'foo/../bar' are all not allowed
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
Now supports multi selection, so we can use it here
viktorvanwijk added a commit that referenced this issue Jan 21, 2025
In the JSON dump docker app, explicitly load the received data before sending again. This prevents the data from being interpreted as a string instead of a JSON object.
@github-project-automation github-project-automation bot moved this from Implemented to Done in Development Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants