User migration (foundation)

[AndyScherzinger]

How to use GitHub

• Please use the 👍 reaction to show that you are interested into the same feature.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Is your feature request related to a problem? Please describe.
Enable users or admins to migrate a user and their data from one server/instance to another

Describe the solution you'd like
A way to export and import a user - as the user themself via webUI as well as being an admin via occ

This issue only covers the the step that are needed to layout its foundation and is limited to the user's files:

---

Base

• OCC command to create an export zip with all files of a user
• OCC command to import an export zip with all files of a user
• Make sure that the export file is flexible to have more or less data for different apps included or not
• Make sure that the export file contains the versions or the Nextcloud server and the apps to make sure that it can only be imported as compatible servers.

Extra packages (ordered by priority)

• Show what data/apps are exported and what data is lost.
• Make it available via an OCS call for the global scale balancer
• Make it available via the Web UI for normal users.
• Implement import/export for the user config.
• Implement import/export for other apps. (Activities, Calendar, Deck, …) Migration orchestration user_migration#27
• Implement import/export of shares and convert them to federated shared

---

• Export user data UI - Accessible via the users settings to request an export
• Export user's files - Export the user's data storing it in the users storage space and making the location available in the export UI (as well as showing it in the regular file listing).
  • Run the export as a background job. The background job should be schedule-able / queue-able to it doesn't hurt the system and can be run in low-load situations like at night and/or is limited in parallel exports triggered by multiple users at once.
• Export rate-limit - make sure it can't be requested again until it's been processed and the export has been deleted. Also, there should be only one export that can run at a time
• Exclude export file from user quota - The export data doesn't count against the quota
• Export limit and retention for users - There can be only one. Export should expire after a week. A new export can only be started after the export is expired
• Export user data occ command - allow an admin to export the files of a specified user
• Import user data UI - Accessible via the users settings added to the export UI
• Import user data occ command - allow an admin to import the user-data-export of a specified user
• Manual export deletion - Add option to users export/import UI to delete the existing export, also when loading the import/export UI check if the export still exists (could have been manually deleted via the files view or clients

Remarks:

• No export of sharing data, no export of files shared with the user that exports data

[Pytal]

@AndyScherzinger would this include any file metadata or would this simply be a clone of a user's directory structure and files?

[jancborchardt]

@AndyScherzinger would this include any file metadata or would this simply be a clone of a user's directory structure and files?

Good point – as per @AndyScherzinger’s remark on the bottom we don’t / can’t migrate any sharing data, but there is additional metadata. Here’s a list

• Versions: these are a hidden folder so should be easy to migrate?
• Favorite state: would be nice to have, doable?
• Tags: would be nice to have, doable?
• Comments: on the fence about that – maybe people use it to comment on their own files, and also comments from others might be useful. But if we can’t do it it’s fine for now as long as we put a notice.
• Projects connections: will be reworked to automatic mechanism anyway Projects rework to automatic "Related resources" #28320
• Activity: passive data so not really needed?
• Anything missing?

[Pytal]

@jancborchardt list is complete as far as I can tell 👌

Versions, favorite state, tags, and maybe comments good for MVP then?

[jancborchardt]

Versions, favorite state, tags, and maybe comments good for MVP then?

Sounds great, yep! :) (And good point on bringing it up cause indeed even for an MVP that would be useful cause otherwise people might as well just Select all → Download. :D)

[come-nc]

What about user metadata? (account name, personal information, email, settings, password…)

[come-nc]

Should this go in core/ or in an application?

Export the user's data storing it in the users storage space and making the location available in the export UI (as well as showing it in the regular file listing).

Does that mean it should go inside <user>/files?
I did two tests, putting the copy in <user>/files/export or <user>/export, in the first case we will need special handling to avoid copying the export folder inside itself (recursion problem), in the second case it works right away and does not count in the quota, but it means we’ll need special treatment to make it appear in the UI and in the dav endpoint if we want to.

[jancborchardt]

What about user metadata? (account name, personal information, email, settings, password…)

Personal info, emails and settings would be very nice indeed. It would be a pain to set all of that again, and that’s always one of the issues when setting up a new laptop if that’s not automatic.

Account name not sure if it makes sense, we could put it in there, but maybe people will have a different name on the other instance, in which case we should just ignore it?

And passwords might be better not to export and handle? Is of course convenient, but then we would need to encrypt?

cc @AndyScherzinger what do you think?

[come-nc]

Account name not sure if it makes sense, we could put it in there, but maybe people will have a different name on the other instance, in which case we should just ignore it?

I suppose export might be used for backup in some cases, where username is important. Having the username in the export does not mean it has to be imported as is, but it may be used as a default value in the import UI or something.

And passwords might be better not to export and handle? Is of course convenient, but then we would need to encrypt?

Yeah I would not be shocked if passwords are not exported at all. I was just mentionnig them to be complete.

[AndyScherzinger]

Well, if the use case is migration (which it is for this issue) then a use is moved from one server to another. So the password would need to stay the same. But we could of course leave that out in the first phase and add the password part at a later stage.

[Spartachetto]

In my humble opinion, if I have access to two user accounts located in two different servers I could want to migrate all the content of one account to the other.
This means that I do not need to have the same username and password. It is enough that I do know the two usernames and passwords.

So to migrate MrWhite@cloud.nextcloud.com to MrBrown@mynewcloud.com I must have the two passwords to start the process; that should be enough.

[come-nc]

So with nextcloud/user_migration#5 current state I have occ export and import commands able to export and import a user basic information.

Here is the output for an export, delete, import of the same user alice from a test setup:

Copying files…
Exporting user information in user.json…
Exporting account information in account.json…
Exporting settings in settings.json…
Exporting versions in versions.json…
Export saved in /tmp/oc_tmp_lOy0B8-.zip
Moved the export to /tmp/alice_2022-01-27_13-44-46.zip
The specified user was deleted
Importing from /tmp/alice_2022-01-27_13-44-46.zip…
Importing user information from user.json…
Importing account information from account.json…?
Importing settings from settings.json…
Importing files…
Successfully imported alice from /tmp/alice_2022-01-27_13-44-46.zip

Here are the differences in the output of commands user:info and user:settings between before/after migration:

Before Migration                                              | After Migration
docker exec /nextcloud_nextcloud_1 occ user:setting alice       docker exec /nextcloud_nextcloud_1 occ user:setting alice
  - avatar:                                                       - avatar:
    - generated: true                                               - generated: true
  - core:                                                     <
    - lang: fr                                                <
    - templateDirectory:                                      <
    - timezone: Europe/Paris                                  <
  - login:                                                    <
    - lastLogin: 1643290908                                   <
  - login_token:                                              <
    - Dhz3KSxGS7h5yHH/prwmHL5d+ThtpjY7: 1643290909            <
  - settings:                                                     - settings:
    - display_name: alice                                           - display_name: alice
docker exec /nextcloud_nextcloud_1 occ user:info alice          docker exec /nextcloud_nextcloud_1 occ user:info alice
  - user_id: alice                                                - user_id: alice
  - display_name: alice                                           - display_name: alice
  - email:                                                        - email: 
  - cloud_id: alice@nextcloud.local                               - cloud_id: alice@nextcloud.local
  - enabled: true                                                 - enabled: true
  - groups:                                                       - groups:
  - quota: none                                                   - quota: none
  - storage:                                                      - storage:
    - free: 275763142656                                      |     - free: 275742556160
    - used: 20575003                                                - used: 20575003
    - total: 275783717659                                     |     - total: 275763131163
    - relative: 0.01                                                - relative: 0.01
    - quota: -3                                                     - quota: -3
  - last_seen: 2022-01-27T13:41:48+00:00                      |   - last_seen: 1970-01-01T00:00:00+00:00
  - user_directory: /var/www/html/data/alice                      - user_directory: /var/www/html/data/alice
  - backend: Database                                             - backend: Database
docker exec /nextcloud_nextcloud_1 occ user:lastseen alice      docker exec /nextcloud_nextcloud_1 occ user:lastseen alice
alice`s last login: 27.01.2022 13:41                          | User alice has never logged in, yet.

So core settings are lost, which I will be able to fix thanks to #30739 , last login is reseted (which seems expected?), and for some reason storage is different which might indicate a problem in files migration.
The password is also not exported for now.

Regarding flexibility nextcloud/user_migration#11 is adding the base for implementing exporters/importers in each application to handle their data. Files migration should be refactored to use this system as well.