[Bug]: Multikey Encryption Breaks When Upgrading from NC26 to NC27

[derschiw]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Hi there

We are running a multikey encrypted NC instance with some hundreds of users. Recently we upgraded from NC26 to NC27. From then on multiple users reported, that they coud not open files anymore (console logging "Cannot decrypt this file, probably this is a shared file"). After doing quiet some research, we were able to track down the problem to this cause:

In the following issue openssl_seal and rc4 were removed (https://github.com/nextcloud/server/pull/37243/files). In this PR the so called fileKey used for RC4 was removed after re-encrypting a file (

server/apps/encryption/lib/Crypto/Encryption.php

Line 395 in 461f6d4

$this->keyManager->deleteAllFileKeys($path);

). Additionally a legacy format was introduced to still beeing able to open old files. However we found out that under certain contidions (resharing or move operations), the fileKey gets deleted without re-encrypting the data, which leads to data loss.

In particular, we could observe that the files_encryption/key/files/[PATH]/OC_DEFAULT_MODULE directory of affected files did contain new [USERNAME].shareKey files and nofileKey anymore. We were then able to replace the content of this directory with the old share keys and the fileKey (that we restored from a backup). By doing this, we could open the file again (in the webbrowser), which leads to the conclusion that the file must have lost its keys without beeing reencrypted.

We were though not able to precisely find the precise code that causes the missing reencryption. We believe that the end or update method in apps/encryption/lib/Crypto/Encryption.php get called without re-encrypting the file.

Help is very much appreciated, as this currently breaks all of our users data and we can't do something against that problem besides informing our customers. Thanks you very much for your support!

Steps to reproduce

Install NC26 with User Key Encryption

1. Install NC 26
2. Enable the default encryption module
3. occ encryption:list-modules to check that the app is enabled
4. occ encryption:disable-master-key and accept the warning
5. occ encryption:enable
6. occ encryption:encrypt-all
7. Log out and log in again.
8. Go to personal settings -> security -> update password in security tab with the password from the CLI.

Create a Folder and Share

1. Create a second user
2. Create a directory, share the directory.
3. Put some files into the directory (not .md - preferrably .jpg, .pdf or .zip). Now in the files_encryption directory are two [USERNAME].shareKey and a fileKey.

Update

1. Update to NC27
2. Create a new directory (for example foo).
3. Move the directory into another directory (click on the three dots -> move or copy -> move to foo).
4. Open the file. It will fail with a warning in the console ("Cannot decrypt...").

Expected behavior

The files should be re-encrypted before removing the fileKeys.

Installation method

None

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

None

Web server

Nginx

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (26 to 27)

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.wolke7.wtf"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "27.1.9.1",
        "overwrite.cli.url": "https:\/\/cloud.wolke7.wtf",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\HEIC",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\MP3"
        ],
        "preview_max_scale_factor": 2,
        "preview_max_x": 128,
        "preview_max_y": 128,
        "lost_password_link": "disabled",
        "allow_user_to_change_display_name": false,
        "skeletondirectory": "\/srv\/nextcloud-assets\/skeleton",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpsecure": "ssl",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "activity_expire_days": 60,
        "log_type": "file",
        "loglevel": 3,
        "logfile": "\/dev\/stdout",
        "maintenance": false,
        "theme": "",
        "app_install_overwrite": [
            "end_to_end_encryption"
        ],
        "simpleSignUpLink.shown": false,
        "mail_sendmailmode": "smtp",
        "encryption.legacy_format_support": true,
        "encryption.key_storage_migrated": false,
        "default_locale": "de_CH",
        "default_phone_region": "CH",
        "updater.server.url": "https:\/\/updates.nextcloud.com\/customers\/[KEY]",
        "auth.webauthn.enabled": false
    }
}

List of activated Apps

Enabled:
  - activity: 2.19.0
  - bruteforcesettings: 2.7.0
  - calendar: 4.7.1
  - cloud_federation_api: 1.10.0
  - contacts: 5.5.3
  - dav: 1.27.0
  - encryption: 2.15.0
  - external: 5.2.1
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.10.0
  - notifications: 2.15.0
  - oauth2: 1.15.2
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - support: 1.10.1
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_totp: 9.0.0
  - twofactor_webauthn: 1.4.0
  - updatenotification: 1.17.0
  - viewer: 2.1.0
  - workflowengine: 2.9.0
Disabled:
  - admin_audit: 1.17.0
  - circles: 27.0.1 (installed 22.1.1)
  - comments: 1.17.0 (installed 1.10.0)
  - contactsinteraction: 1.8.0 (installed 1.3.0)
  - dashboard: 7.7.0 (installed 7.0.0)
  - end_to_end_encryption: 1.13.1 (installed 1.13.1)
  - files_external: 1.19.0
  - survey_client: 1.15.0 (installed 1.4.0)
  - suspicious_login: 5.0.0
  - systemtags: 1.17.0 (installed 1.10.0)
  - user_ldap: 1.17.0
  - user_status: 1.7.0 (installed 1.0.1)
  - weather_status: 1.7.0 (installed 1.0.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

[kesselb]

fyi @AndyScherzinger @sorbaugh @come-nc

[kesselb]

I cannot tell if related but noticed that end_to_end_encryption is listed in app_install_overwrite.

E2E and SSE are not compatible and cannot be used at the same time afaik, because they both use the encrypted column in oc_filecache.

[derschiw]

@kesselb when starting the cloud we tried E2EE but then disabled the app (years ago) without having any problems so far. So i guess this is not related.

[AndyScherzinger]

Also Looping in @artonge

[artonge]

I failed to reproduce.
@derschiw I tried to move a file out of the shared folder, both from the emitter and the recipient.
When doing it from the recipient account, the fileKey is deleted, but the file can still be opened. Anything I am missing?

[derschiw]

Sorry for the late response.
Did you try it with an markdown / image file? We believe that some files are either in the cache or unencrypted thumbnails. But the timestamps in the key files and encrypted files differ. Also you will see that the download will fail.

However we just reproduced it with .mp4 and .pdf files locally in a fresh install. It breaks as expected.

Sorry we didn't realize that before... I updated the Steps to reproduce.

[m-vz]

I was just able to reproduce the error on a fresh instance with the updated steps @derschiw posted.

Moved all default files into a folder on NC26 and shared that. Upgrade to NC27. After moving the folder without editing the files, none of them can be opened anymore.

This seems like a serious problem to me...

[1H0]

I was also able to reproduce this issue with the steps provided by @derschiw.

I also moved all the preexisting files in a folder on NC26 and shared it with a user. I then upgraded to NC27 and added another user to that same folder, which then made files in it inacessible to all participants.

[derschiw]

Are there any news on that? Were you able to reproduce it? This bug still keeps breaking all files from our customers and we can't do something against it. So, help would be very much appreciated!

[artonge]

I was able to reproduce by adding a recipient after the update. @come-nc:

When adding a new recipient, we trigger

server/apps/encryption/lib/Crypto/Encryption.php

Lines 412 to 455 in 5bc8329

	public function update($path, $uid, array $accessList) {
	if (empty($accessList)) {
	if (isset(self::$rememberVersion[$path])) {
	$this->keyManager->setVersion($path, self::$rememberVersion[$path], new View());
	unset(self::$rememberVersion[$path]);
	}
	return false;
	}
	$fileKey = $this->keyManager->getFileKey($path, $uid, null);
	if (!empty($fileKey)) {
	$publicKeys = [];
	if ($this->useMasterPassword === true) {
	$publicKeys[$this->keyManager->getMasterKeyId()] = $this->keyManager->getPublicMasterKey();
	} else {
	foreach ($accessList['users'] as $user) {
	try {
	$publicKeys[$user] = $this->keyManager->getPublicKey($user);
	} catch (PublicKeyMissingException $e) {
	$this->logger->warning('Could not encrypt file for ' . $user . ': ' . $e->getMessage());
	}
	}
	}
	$publicKeys = $this->keyManager->addSystemKeys($accessList, $publicKeys, $this->getOwner($path));
	$shareKeys = $this->crypt->multiKeyEncrypt($fileKey, $publicKeys);
	$this->keyManager->deleteAllFileKeys($path);
	foreach ($shareKeys as $uid => $keyFile) {
	$this->keyManager->setShareKey($path, $uid, $keyFile);
	}
	} else {
	$this->logger->debug('no file key found, we assume that the file "{file}" is not encrypted',
	['file' => $path, 'app' => 'encryption']);
	return false;
	}
	return true;
	}

Which:

1. Generate $shareKeys for each recipient
2. deleteAllFileKeys, including fileKey
3. setShareKey for each recipient, but not fileKey

The issue is with n°2

server/lib/private/Encryption/Keys/Storage.php

Lines 187 to 191 in 5bc8329

	public function deleteAllFileKeys($path) {
	$keyDir = $this->getFileKeyDir('', $path);
	return !$this->view->file_exists($keyDir) || $this->view->deleteAll($keyDir);
	}

Which will remove the fileKey which is not added again by n°3.

Draft level solutions ideas:

1. Temporary save fileKey in update, and rewrite it after deleteAllFileKeys`.
2. Or update deleteAllFileKeys to not delete fileKey. But that would be unexpected and might lead to other issues.
3. Use detect if using legacy file key and use multiKeyEncryptLegacy instead of multiKeyEncrypt?
4. Or something else? What were we doing previously?

What do you think?

[come-nc]

I think it is on purpose that the fileKey is removed, because it should be embedded in the generated shareKeys once legacy encryption is not used anymore.
Maybe the problem here is that the useLegacyFileKey is not set to false in the header when this update happens?

I remember this was complicated because rewriting header means rewriting the file.