♻️ [#1068] -- Refactor formio integration

docs/developers/backend/core/formio.rst

@@ -0,0 +1,134 @@
+.. _developers_backend_core_formio:
+
+=========================
+Core: form.io integration
+=========================
+
+Open Forms uses `form.io`_ under the hood to build and render forms, and then adds its
+own layers on top of that, such as:
+
+* implementing multi-step forms where every step is a form.io definition
+* evaluating backend logic using data from earlier steps
+* dynamically adapting form.io definitions as needed
+
+This means that we process the form.io datastructures in the backend, using Python. All
+the code for this is organized in the ``openforms.formio`` package.
+
+.. versionchanged:: 2.1.0
+
+    ``openforms.custom_field_types`` was refactored into the ``openforms.formio`` package,
+    and all of the separate registries (formatters, normalizers...) were merged into a
+    single compoment registry.
+
+Supported features
+==================
+
+.. currentmodule:: openforms.formio.service
+
+Formatting values for display
+-----------------------------
+
+Value formatting is done for displaying form submission data summaries, rendering
+confirmation PDFs and emails... It is aware if it's in a HTML context or not. It is
+heavily used in the :ref:`renderers <developers_backend_core_submission_renderer>`.
+
+Whenever a component plugin is registered, the
+:attr:`openforms.formio.registry.BasePlugin.formatter` class attribute **must** be
+specified.
+
+.. autofunction:: format_value
+    :noindex:
+
+
+Normalizing input data
+----------------------
+
+Data for a component can be sourced from external systems that employ different
+formatting rules compared to what form.io expects. Normalizing this data helps to be
+able to make proper comparisons at different stages in the submission life-cycle.
+
+You can opt-in to this by configuring :attr:`openforms.formio.registry.BasePlugin.normalizer`.
+
+.. autofunction:: normalize_value_for_component
+    :noindex:
+
+
+Dynamically modifying component configuration
+---------------------------------------------
+
+Certain component types require on-the-fly configuration rewriting, such as applying
+global configuration options that may change independently from when the form is
+actually being designed.
+
+Dynamic rewriting is enabled by implementing
+:meth:`openforms.formio.registry.BasePlugin.mutate_config_dynamically`. It receives the
+current :class:`openforms.submissions.models.Submission` instance and a mapping of all
+the variable names and values at the time.
+
+.. autofunction:: get_dynamic_configuration
+    :noindex:
+
+For an example of a custom field type, see :class:`openforms.formio.components.custom.Date`.
+
+Finally, the resulting resolved component definitions are evaluated with the template
+engine where variable values are evaluated for compoment labels, descriptions... and
+configuration based on the HTTP request is performed (see
+:func:`openforms.formmio.service.rewrite_formio_components_for_request`).
+
+Reference
+=========
+
+Public API - ``openforms.formio.service``
+-----------------------------------------
+
+.. automodule:: openforms.formio.service
+    :members:
+
+.. autoclass:: openforms.formio.registry.BasePlugin
+    :members:
+    :exclude-members: verbose_name
+
+Extending
+---------
+
+Using our :ref:`usual extension pattern <developers_extending>` you can register your
+own types.
+
+Extensions should inherit from :class:`openforms.formio.registry.BasePlugin` or
+implement the same protocol(s) and be registered with their form.io type:
+
+.. code-block:: python
+
+    from openforms.formio.formatters.formio import DefaultFormatter
+    from openforms.formio.registry import BasePlugin
+
+    @register("myCustomType")
+    class MyComponent(BasePlugin):
+        formatter = DefaultFormatter
+
+
+You can find some examples in ``openforms.formio.components.custom``.
+
+
+Private API
+-----------
+
+Module: ``openforms.formio.dynamic_config``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: openforms.formio.dynamic_config
+    :members:
+
+Module: ``openforms.formio.formatters``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: openforms.formio.formatters
+    :members:
+
+Module: ``openforms.formio.rendering``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: openforms.formio.rendering
+    :members:
+
+.. _form.io: https://www.form.io/

docs/developers/backend/core/index.rst

@@ -1,44 +1,27 @@
 .. _developers_backend_core_index:
 
 ==================
-Core functionality
+Core: introduction
 ==================
 
-Reference documentation for the Open Forms core.
+As stated in the :ref:`developers_architecture`:
 
-General principles
-==================
-
-**Keep Django apps contained**
-
-Django apps should focus on a single responsibility with minimal dependencies on the
-"outside world".
-
-**Explicitly expose public API**
-
-Usually we only consider the ``service.py`` module of a Django app to be public API.
-Please think twice before introducing breaking changes in those modules.
-
-**Refrain from importing private API**
-
-Importing things from modules in core functionality is generally frowned upon (
-``service.py`` is the exception here). Modules should be able to freely alter their
-implementation details, including their data model!
-
-**Document useful, generic functionality**
+    Core functionality is considered functionality that does not or very loosely tie in
+    to particular modules. It is functionality that has meaning on its own without
+    dependencies, but is enriched by modules.
 
-Documentation makes it easier to find out what exists and avoid re-implementing the
-same thing twice.
+The core really implements the "engine" of Open Forms and hides all the implementation
+details. It should be fairly stable, but also continually allow for new feature
+additions, which is a challenging task!
 
+The following Django apps are considered core functionality:
 
-.. toctree::
-   :caption: Contents
-   :maxdepth: 2
+* :mod:`openforms.accounts`: (staff) user account management
+* :mod:`openforms.config`: application-wide configuration and defaults
+* :mod:`openforms.forms`: designing and building of forms
+* :mod:`openforms.formio`: integration with the `form.io`_ frontend library
+* :mod:`openforms.submissions`: persisting and handling of submitted form data
+* :mod:`openforms.variables`: persisting (intermediate) data into variables for further
+  operations
 
-   submissions
-   submission-renderer
-   utils
-   tokens
-   testing-tools
-   variables
-   templating
+.. _form.io: https://www.form.io/

docs/developers/backend/core/submission-renderer.rst

@@ -2,9 +2,9 @@
 
 .. currentmodule:: openforms.submissions.rendering
 
-====================
-Submission rendering
-====================
+==========================
+Core: submission rendering
+==========================
 
 Submission rendering is the concept of outputting the submitted data for a given form
 in a particular format. A couple of examples:

docs/developers/backend/core/submissions.rst

@@ -1,8 +1,8 @@
 .. _developers_backend_core_submissions:
 
-===========
-Submissions
-===========
+===========================
+Core: submission processing
+===========================
 
 Once a form is submitted by the user, a number of actions take place to process
 this submission. The complete process is shown below.

docs/developers/backend/core/testing-tools.rst

@@ -1,17 +1,27 @@
 .. _developers_backend_core_testing_tools:
 
-Testing tools
-=============
+============
+Test helpers
+============
+
+HTML assertions
+===============
 
 .. automodule:: openforms.utils.tests.html_assert
    :members:
 
 .. automodule:: openforms.utils.tests.webtest_base
    :members:
 
+Migrations
+==========
+
 .. automodule:: openforms.utils.tests.test_migrations
    :members:
 
+Formio assertions
+=================
+
 .. automodule:: openforms.formio.tests.assertions
    :members:
    :undoc-members:

docs/developers/backend/core/variables.rst

@@ -1,12 +1,13 @@
 .. _developers_backend_core_variables:
 
-=========
-Variables
-=========
+===============
+Core: variables
+===============
 
 There are two models for variables, :class:`openforms.forms.models.FormVariable` and
-:class:`openforms.submissions.models.SubmissionValueVariable`. ``FormVariable`` are related to
-a form while ``SubmissionValueVariable`` are related to a submission.
+:class:`openforms.submissions.models.SubmissionValueVariable`. ``FormVariable`` objects
+are related to a form while ``SubmissionValueVariable`` objects are related to a
+submission (and a form variable).
 
 Form Variables
 ==============

docs/developers/backend/index.rst

@@ -10,15 +10,76 @@ Django has a concept of "apps" within a project which logically contain some
 functionality. The codebase layout reflects this - but the documentation is structured
 following the outlined :ref:`developers_architecture`.
 
+General principles
+==================
+
+On top of all this, we apply some general principles to keep/make the codebase
+maintainable.
+
+**Keep Django apps contained**
+
+Django apps should focus on a single responsibility with minimal dependencies on the
+"outside world".
+
+**Explicitly expose public API**
+
+Usually we only consider the ``service.py`` module of a Django app to be public API.
+Please think twice before introducing breaking changes in those modules.
+
+**Refrain from importing private API**
+
+Importing things from modules in core functionality is generally frowned upon (
+``service.py`` is the exception here). Modules should be able to freely alter their
+implementation details, including their data model!
+
+**Document useful, generic functionality**
+
+Documentation makes it easier to find out what exists and avoid re-implementing the
+same thing twice.
+
+Core
+====
+
 .. toctree::
-   :caption: Contents
-   :maxdepth: 2
-
-   core/index
-   modules/index
-   dev-rendering
-   profiling
-   upgrade-checks
-   file-uploads
+    :maxdepth: 1
+
+    core/index
+    core/formio
+    core/submissions
+    core/submission-renderer
+    core/variables
+    file-uploads
+
+Modules
+=======
+
+.. toctree::
+    :maxdepth: 1
+
+    modules/index
+    modules/dmn
+
+General purpose functionality
+=============================
+
+.. toctree::
+    :maxdepth: 1
+
+    core/utils
+    core/tokens
+    core/testing-tools
+    core/templating
+
+    upgrade-checks
+
+Development and debug tooling
+=============================
+
+.. toctree::
+    :maxdepth: 1
+
+    dev-rendering
+    profiling
+    file-uploads
 
 .. _Django: https://www.djangoproject.com/

docs/developers/backend/modules/index.rst

@@ -1,18 +1,12 @@
 .. _developers_backend_modules_index:
 
-=======
-Modules
-=======
+=====================
+Modules: introduction
+=====================
 
 Open Forms groups functionalities in *modules*. Within a module there are often one
 or more plugins available for a vendor-specific implementation of the high-level
 functionality. Modules can usually be :ref:`extended <developers_extending>` with custom
 plugins.
 
 .. todo:: This documentation is incomplete
-
-.. toctree::
-   :caption: Contents
-   :maxdepth: 2
-
-   dmn