RFC 42: Data transfer between Wagtail installations

[gasman]

Rendered

[kaedroho]

Any reason to not require an identifier for each source? (similar to how we identify database and search backends)

WAGTAIL_IMPORT_SOURCE_INSTANCES = {
    'staging': {
        'label': "Staging",   # Could default to the identifier.title()
    }
}

[kaedroho]

We tend to omit the underscore between WAGTAIL and the module name, For example. WAGTAILSEARCH_BACKENDS.

Also maybe _SOURCE_INSTANCES could be replaced with _SOURCES?

[kaedroho]

Should we add a unique_together constraint on content_type and local_id?

[cyface]

I think in the spirit of the use case that I have for this, the pull approach should be fine, but the push approach is very much preferred. :)

In cases where the reason for doing data transfer is security related, it may not be possible for direct login to 'publish' instances vs. 'author' instances. UX-wise, the push approach also prevents the author from having to log in to multiple instances, and figure out which instance they are on.

Perhaps a menu item in the author instance that jumps you over to publish and initiates the import for that page?

[chosak]

This would be a great feature to have and should definitely be discussed as an RFC here.

In cases where the reason for doing data transfer is security related, it may not be possible for direct login to 'publish' instances vs. 'author' instances.

+1 @cyface. A reasonable and common pattern is to have the production environment in a network purposefully isolated from staging and development environments. You might not want to expose those environments to the production one because that opens the possibility of external requests making it down to an internal network.

In those cases it would be preferable to be able to do some kind of file-based export, where users on the "source" Wagtail can generate a transfer artifact which can be imported into the "destination" Wagtail. This would be akin to a super dumpdata which dumped only the desired models, along with a super loaddata which figured out how to smartly ingest them.

This file-based export could potentially even live along side a direct connection-based import/export, sharing the same logic underneath. This might depend a bit on how the logic works: if an import is "make a bunch of API calls, then manipulate and ingest the data", that seems like it'd work just as well with an interim file-based step. If instead the import logic were iterative (meaning, a query is made to the API, then other queries are made depending on how the destination is set up), this wouldn't fit so well with the proposed approach.

[gasman]

@chosak I think it is an iterative process, and one that can potentially vary depending on what objects exist at the destination. For any item that has been explicitly chosen to be imported, there's a tree of related items to be imported along with it, potentially of any depth (for example, a page may reference a snippet which references an image).

For every such relation in that tree, there's a business rule (i.e. one of the bullet points given under https://github.com/gasman/weps/blob/81060c34c190482e262be6b7595ac4d213276704/text/042-data-transfer-between-installations.md#configuration) which determines whether the related object will be imported or skipped. Some of these business rules have a 'get-or-create' behaviour, which means that they may or may not end up importing the related object (and possibly pulling in even more related objects associated with it) depending on whether it exists already. This means that the dataset exported from the source site has to be larger in the 'create' scenario than the 'get' scenario.

As a concrete example: suppose we're importing a blog section, and the blog posts have categories, identified by a slug. These categories are snippet/modeladmin models, and should be created at the destination whenever we encounter one that doesn't exist. Now, suppose that a category object isn't just a plain slug + label, but includes some 'rich' fields, such as an image or a rich-text description. The import logic has to consider the object references in these fields if, and only if, the category doesn't exist already at the destination - which is something we can't know if we're preparing the export dataset up-front.

This isn't insurmountable - the exporter logic could take a "pessimistic" view and always output the extra records whenever there's a chance of needing them - but it's added complexity that can be left until later.

[kaedroho]

Discussed in Wagtail core meeting. Everyone in favour of approach as it stands as it could be extended to support a push workflow and transfer with a file later on.

We also discussed implementing a mechanism in Wagtail core to discover all objects that a page references. This would scrape foreign keys, chooser blocks and rich text features to find all linked pages, images, documents and snippets. Everyone was in favour of this change too.

[zerolab]

Work done in https://github.com/wagtail/wagtail-transfer/

[kaedroho]

Merged since this is implemented now and the RFC generally reflects what made it in to Wagtail transfer

[SidharajYadav]

I want to start issues