Merge remote-tracking branch 'origin/develop' into clokep/thread-bund…

…led-relations

CHANGES.md

@@ -1,3 +1,13 @@
+Synapse 1.47.0rc3 (2021-11-16)
+==============================
+
+Bugfixes
+--------
+
+- Fix a bug introduced in 1.47.0rc1 which caused worker processes to not halt startup in the presence of outstanding database migrations. ([\#11346](https://github.com/matrix-org/synapse/issues/11346))
+- Fix a bug introduced in 1.47.0rc1 which prevented the 'remove deleted devices from `device_inbox` column' background process from running when updating from a recent Synapse version. ([\#11303](https://github.com/matrix-org/synapse/issues/11303), [\#11353](https://github.com/matrix-org/synapse/issues/11353))
+
+
 Synapse 1.47.0rc2 (2021-11-10)
 ==============================
 
@@ -8678,14 +8688,14 @@ General:
 
 Federation:
 
--   Add key distribution mechanisms for fetching public keys of unavailable remote home servers. See [Retrieving Server Keys](https://github.com/matrix-org/matrix-doc/blob/6f2698/specification/30_server_server_api.rst#retrieving-server-keys) in the spec.
+-   Add key distribution mechanisms for fetching public keys of unavailable remote homeservers. See [Retrieving Server Keys](https://github.com/matrix-org/matrix-doc/blob/6f2698/specification/30_server_server_api.rst#retrieving-server-keys) in the spec.
 
 Configuration:
 
 -   Add support for multiple config files.
 -   Add support for dictionaries in config files.
 -   Remove support for specifying config options on the command line, except for:
-    -   `--daemonize` - Daemonize the home server.
+    -   `--daemonize` - Daemonize the homeserver.
     -   `--manhole` - Turn on the twisted telnet manhole service on the given port.
     -   `--database-path` - The path to a sqlite database to use.
     -   `--verbose` - The verbosity level.
@@ -8890,7 +8900,7 @@ This version adds support for using a TURN server. See docs/turn-howto.rst on ho
 Homeserver:
 
 -   Add support for redaction of messages.
--   Fix bug where inviting a user on a remote home server could take up to 20-30s.
+-   Fix bug where inviting a user on a remote homeserver could take up to 20-30s.
 -   Implement a get current room state API.
 -   Add support specifying and retrieving turn server configuration.
 
@@ -8980,7 +8990,7 @@ Changes in synapse 0.2.3 (2014-09-12)
 
 Homeserver:
 
--   Fix bug where we stopped sending events to remote home servers if a user from that home server left, even if there were some still in the room.
+-   Fix bug where we stopped sending events to remote homeservers if a user from that homeserver left, even if there were some still in the room.
 -   Fix bugs in the state conflict resolution where it was incorrectly rejecting events.
 
 Webclient:

changelog.d/11230.bugfix

@@ -0,0 +1,2 @@
+Fix a long-standing bug wherein display names or avatar URLs containing null bytes cause an internal server error
+when stored in the DB.

changelog.d/11280.misc

@@ -0,0 +1 @@
+Drop unused db tables `room_stats_historical` and `user_stats_historical`.

changelog.d/11316.misc

@@ -0,0 +1 @@
+Add type hints to storage classes.

changelog.d/11318.feature

@@ -0,0 +1 @@
+Add support for the `/_matrix/client/v3` APIs from Matrix v1.1.

changelog.d/11320.doc

@@ -0,0 +1 @@
+Changed the word 'Home server' as one word 'homeserver' in documentation.

changelog.d/11327.misc

@@ -0,0 +1 @@
+Test that room alias deletion works as intended.

changelog.d/11328.misc

@@ -0,0 +1 @@
+Add type hints to `synapse.util`.

changelog.d/11330.misc

@@ -0,0 +1 @@
+Improve type annotations in Synapse's test suite.

changelog.d/11332.misc

@@ -0,0 +1 @@
+Add type hints to storage classes.

changelog.d/11335.feature

@@ -0,0 +1 @@
+Support the stable version of [MSC2778](https://github.com/matrix-org/matrix-doc/pull/2778): the `m.login.application_service` login type. Contributed by @tulir.

changelog.d/11339.misc

@@ -0,0 +1 @@
+Add type hints to storage classes.

changelog.d/11340.bugfix

@@ -0,0 +1 @@
+Fix a bug, introduced in Synapse 1.46.0, which caused the `check_3pid_auth` and `on_logged_out` callbacks in legacy password authentication provider modules to not be registered. Modules using the generic module API were not affected.

changelog.d/11342.misc

@@ -0,0 +1 @@
+Add type hints to storage classes.

changelog.d/11347.feature

@@ -0,0 +1 @@
+Add admin API to un-shadow-ban a user.

changelog.d/11355.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in 1.41.0 where space hierarchy responses would be incorrectly reused if multiple users were to make the same request at the same time.

changelog.d/11359.misc

@@ -0,0 +1 @@
+Require all files in synapse/ and tests/ to pass mypy unless specifically excluded.

debian/changelog

@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.47.0~rc3) stable; urgency=medium
+
+  * New synapse release 1.47.0~rc3.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 16 Nov 2021 14:32:47 +0000
+
 matrix-synapse-py3 (1.47.0~rc2) stable; urgency=medium
 
   [ Dan Callahan ]

docs/admin_api/user_admin_api.md

@@ -948,7 +948,7 @@ The following fields are returned in the JSON response body:
 See also the
 [Client-Server API Spec on pushers](https://matrix.org/docs/spec/client_server/latest#get-matrix-client-r0-pushers).
 
-## Shadow-banning users
+## Controlling whether a user is shadow-banned
 
 Shadow-banning is a useful tool for moderating malicious or egregiously abusive users.
 A shadow-banned users receives successful responses to their client-server API requests,
@@ -961,16 +961,22 @@ or broken behaviour for the client. A shadow-banned user will not receive any
 notification and it is generally more appropriate to ban or kick abusive users.
 A shadow-banned user will be unable to contact anyone on the server.
 
-The API is:
+To shadow-ban a user the API is:
 
 ```
 POST /_synapse/admin/v1/users/<user_id>/shadow_ban
 ```
 
+To un-shadow-ban a user the API is:
+
+```
+DELETE /_synapse/admin/v1/users/<user_id>/shadow_ban
+```
+
 To use it, you will need to authenticate by providing an `access_token` for a
 server admin: [Admin API](../usage/administration/admin_api)
 
-An empty JSON dict is returned.
+An empty JSON dict is returned in both cases.
 
 **Parameters**
 

docs/ancient_architecture_notes.md

@@ -7,7 +7,7 @@
 
 ## Server to Server Stack
 
-To use the server to server stack, home servers should only need to
+To use the server to server stack, homeservers should only need to
 interact with the Messaging layer.
 
 The server to server side of things is designed into 4 distinct layers:
@@ -23,7 +23,7 @@ Server with a domain specific API.
 
 1. **Messaging Layer**
 
-    This is what the rest of the Home Server hits to send messages, join rooms,
+    This is what the rest of the homeserver hits to send messages, join rooms,
     etc. It also allows you to register callbacks for when it get's notified by
     lower levels that e.g. a new message has been received.
 
@@ -45,7 +45,7 @@ Server with a domain specific API.
 
     For incoming PDUs, it has to check the PDUs it references to see
     if we have missed any. If we have go and ask someone (another
-    home server) for it.
+    homeserver) for it.
 
 3. **Transaction Layer**
 

docs/turn-howto.md

@@ -1,12 +1,12 @@
 # Overview
 
-This document explains how to enable VoIP relaying on your Home Server with
+This document explains how to enable VoIP relaying on your homeserver with
 TURN.
 
-The synapse Matrix Home Server supports integration with TURN server via the
+The synapse Matrix homeserver supports integration with TURN server via the
 [TURN server REST API](<https://tools.ietf.org/html/draft-uberti-behave-turn-rest-00>). This
-allows the Home Server to generate credentials that are valid for use on the
-TURN server through the use of a secret shared between the Home Server and the
+allows the homeserver to generate credentials that are valid for use on the
+TURN server through the use of a secret shared between the homeserver and the
 TURN server.
 
 The following sections describe how to install [coturn](<https://github.com/coturn/coturn>) (which implements the TURN REST API) and integrate it with synapse.
@@ -165,18 +165,18 @@ This will install and start a systemd service called `coturn`.
 
 ## Synapse setup
 
-Your home server configuration file needs the following extra keys:
+Your homeserver configuration file needs the following extra keys:
 
 1.  "`turn_uris`": This needs to be a yaml list of public-facing URIs
     for your TURN server to be given out to your clients. Add separate
     entries for each transport your TURN server supports.
 2.  "`turn_shared_secret`": This is the secret shared between your
-    Home server and your TURN server, so you should set it to the same
+    homeserver and your TURN server, so you should set it to the same
     string you used in turnserver.conf.
 3.  "`turn_user_lifetime`": This is the amount of time credentials
-    generated by your Home Server are valid for (in milliseconds).
+    generated by your homeserver are valid for (in milliseconds).
     Shorter times offer less potential for abuse at the expense of
-    increased traffic between web clients and your home server to
+    increased traffic between web clients and your homeserver to
     refresh credentials. The TURN REST API specification recommends
     one day (86400000).
 4.  "`turn_allow_guests`": Whether to allow guest users to use the

mypy.ini

@@ -28,7 +28,6 @@ exclude = (?x)
    |synapse/storage/databases/main/account_data.py
    |synapse/storage/databases/main/cache.py
    |synapse/storage/databases/main/devices.py
-   |synapse/storage/databases/main/directory.py
    |synapse/storage/databases/main/e2e_room_keys.py
    |synapse/storage/databases/main/end_to_end_keys.py
    |synapse/storage/databases/main/event_federation.py
@@ -39,15 +38,13 @@ exclude = (?x)
    |synapse/storage/databases/main/metrics.py
    |synapse/storage/databases/main/monthly_active_users.py
    |synapse/storage/databases/main/presence.py
-   |synapse/storage/databases/main/profile.py
    |synapse/storage/databases/main/purge_events.py
    |synapse/storage/databases/main/push_rule.py
    |synapse/storage/databases/main/receipts.py
    |synapse/storage/databases/main/room.py
    |synapse/storage/databases/main/roommember.py
    |synapse/storage/databases/main/search.py
    |synapse/storage/databases/main/state.py
-   |synapse/storage/databases/main/state_deltas.py
    |synapse/storage/databases/main/stats.py
    |synapse/storage/databases/main/transactions.py
    |synapse/storage/databases/main/user_directory.py
@@ -175,100 +172,46 @@ disallow_untyped_defs = True
 [mypy-synapse.state.*]
 disallow_untyped_defs = True
 
-[mypy-synapse.storage]
+[mypy-synapse.storage.databases.main.client_ips]
 disallow_untyped_defs = True
 
-[mypy-synapse.streams.*]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.batching_queue]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.caches.cached_call]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.caches.dictionary_cache]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.caches.lrucache]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.caches.response_cache]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.caches.stream_change_cache]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.caches.ttl_cache]
+[mypy-synapse.storage.databases.main.directory]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.daemonize]
+[mypy-synapse.storage.databases.main.room_batch]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.file_consumer]
+[mypy-synapse.storage.databases.main.profile]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.frozenutils]
+[mypy-synapse.storage.databases.main.state_deltas]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.hash]
+[mypy-synapse.storage.databases.main.user_erasure_store]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.httpresourcetree]
+[mypy-synapse.storage.util.*]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.iterutils]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.linked_list]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.logcontext]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.logformatter]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.macaroons]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.manhole]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.module_loader]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.msisdn]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.patch_inline_callbacks]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.ratelimitutils]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.retryutils]
-disallow_untyped_defs = True
-
-[mypy-synapse.util.rlimit]
+[mypy-synapse.streams.*]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.stringutils]
+[mypy-synapse.util.*]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.templates]
-disallow_untyped_defs = True
+[mypy-synapse.util.caches.treecache]
+disallow_untyped_defs = False
 
-[mypy-synapse.util.threepids]
+[mypy-tests.handlers.test_user_directory]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.wheel_timer]
+[mypy-tests.storage.test_profile]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.versionstring]
+[mypy-tests.storage.test_user_directory]
 disallow_untyped_defs = True
 
-[mypy-tests]
+[mypy-tests.rest.client.test_directory]
 disallow_untyped_defs = True
 
 ;; Dependencies without annotations

synapse/__init__.py

@@ -47,7 +47,7 @@
 except ImportError:
     pass
 
-__version__ = "1.47.0rc2"
+__version__ = "1.47.0rc3"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/app/homeserver.py

@@ -193,6 +193,7 @@ def _configure_named_resource(
                 {
                     "/_matrix/client/api/v1": client_resource,
                     "/_matrix/client/r0": client_resource,
+                    "/_matrix/client/v3": client_resource,
                     "/_matrix/client/unstable": client_resource,
                     "/_matrix/client/v2_alpha": client_resource,
                     "/_matrix/client/versions": client_resource,