Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Updates to the Room DAG concepts development document #12179

Merged
merged 2 commits into from
Mar 10, 2022
Merged

Updates to the Room DAG concepts development document #12179

merged 2 commits into from
Mar 10, 2022

Conversation

richvdh
Copy link
Member

@richvdh richvdh commented Mar 8, 2022

Some stuff that came up while we were talking about #12173.

@richvdh richvdh requested a review from a team as a code owner March 8, 2022 13:08
clokep
clokep reviewed Mar 9, 2022
View reviewed changes
docs/development/room-dag-concepts.md
Comment on lines +45 to +46
(though it's entirely possible that we *might* have them for some other
reason). Other things that make outliers different from regular events:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not saying we need to include them all here, but could you give a couple of examples for my own understanding? I imagine it is something like we have one, but we don't know the connection between the outlier and the previous events for some reason (maybe we previously left a room and were re-invited or something weird)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a couple of examples:

  • When you first join a room, you pull in the current state events in that room as outliers, which will normally include the first few events (m.room.create, m.room.member, m.room.power_levels, etc). Obviously, the m.room.create is the prev_event of the first m.room.member. So we have both in the database.
  • Suppose we received a chunk of timeline, and then went offline for a while. The first thing that happens to happen when we go offline is a state event (call it S), and then there's a load of other activity. Later, we come back online. We don't fill the entire gap in the DAG, but we do end up pulling in any state that changed - including S - as outliers. So S is an outlier, but it just so happens that we have its prev_events in the database.

I can stick this in the body of the doc if it would be helpful.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I don't know if we really need the examples, as you mentioned in chat it might be taken as a list of the only ways it happens.

clokep
clokep reviewed Mar 9, 2022
View reviewed changes
docs/development/room-dag-concepts.md

## Outliers
* invites received over federation, before we join the room
Copy link
Member

@clokep clokep Mar 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have local users Alice and Bob; remote user Charlie. Charlie and Bob share a room. Charlie invites Alice to join them.

In this case, do we still end up with an outlier / out-of-band membership event? The homeserver will have the full auth chain, etc. from Bob so I don't think so?

(I think the below cases are similar.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question!

I think the invite still follows the same codepath, even though we share a room. So yeah, it probably still ends up stored as an out-of-band-membership event. I don't really know, without trying it out.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the answer! I suspect it isn't important to figure out (it should do the right thing?), but figured I'd ask!

@clokep
Copy link
Member

clokep commented Mar 9, 2022

Thanks for writing this up! Definitely nice to have documentation on this! 🎉

@richvdh richvdh merged commit 52a947d into develop Mar 10, 2022
@richvdh richvdh deleted the rav/update_dag_concepts branch March 10, 2022 15:18
@MadLittleMods MadLittleMods added the A-Docs things relating to the documentation label Mar 26, 2022
MadLittleMods
MadLittleMods reviewed Mar 26, 2022
View reviewed changes
docs/development/room-dag-concepts.md
Comment on lines +39 to +42
Outliers typically arise when we fetch the auth chain or state for a given
event. When that happens, we just grab the events in the state/auth chain,
without calculating the state at those events, or backfilling their
`prev_events`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence makes less practical sense to me than the previous one:

For example, when we fetch the event auth chain or state for a given event, we
mark all of those claimed auth events as outliers because we haven't done the
state calculation ourself.

Perhaps:

Suggested change
Outliers typically arise when we fetch the auth chain or state for a given
event. When that happens, we just grab the events in the state/auth chain,
without calculating the state at those events, or backfilling their
`prev_events`.
Outliers typically arise when we fetch the auth chain or state for a given
event. When that happens, we mark all of those claimed auth events as
outliers because we haven't done the state calculation ourself, or backfilled
their `prev_events`.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand what you're trying to get at with "we haven't done the state calculation ourself". There are other times where we don't calculate the state at an event and yet those events aren't outliers (eg: the join event when joining a room over federation).

Also: the fact that we haven't backfilled prev_events doesn't, in itself, make it an outlier.

Copy link
Contributor

@MadLittleMods MadLittleMods Mar 28, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the big differentiator phrase to me is "mark all of those claimed auth events as outliers". Without something like that, it's not clear to me what we do after "we just grab the events in the state/auth chain".

The other phrasing is just notes on what Synapse is doing. By "ourself", I mean the local homeserver doing the work that we trust as a user on that server. Vs other servers who already calculated the auth_events on the outlier since it has auth_events and is available over federation but we don't know if that's absolutely correct.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, I don't disagree that the wording could be clarified, but I'm still struggling to make improvements without saying things that are actually wrong. For example,

mark all of those claimed auth events as outliers

isn't really true: we only do so for (claimed) auth events that we didn't already have.

How about just adding a sentence:

Suggested change
Outliers typically arise when we fetch the auth chain or state for a given
event. When that happens, we just grab the events in the state/auth chain,
without calculating the state at those events, or backfilling their
`prev_events`.
Outliers typically arise when we fetch the auth chain or state for a given
event. When that happens, we just grab the events in the state/auth chain,
without calculating the state at those events, or backfilling their
`prev_events`. Since we don't have the state at any events fetched in that
way, we mark them as outliers.

Copy link
Contributor

@MadLittleMods MadLittleMods Mar 29, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's better 👍

For some reason "claimed" clarifies the point that they can't be trusted for me which would be nice to include.

mark all of those claimed auth events as outliers

isn't really true: we only do so for (claimed) auth events that we didn't already have.

I feel like your suggestion also doesn't clarify this point but I feel like it doesn't matter whether this is clarified anyway. It's a separate fact that we don't re-outlier a persisted normal event. And the paragraph below slightly touches a bit on this point in a different way.

I'm not coming up without something better than the following but your suggestion is also a good improvement.

Suggested change
Outliers typically arise when we fetch the auth chain or state for a given
event. When that happens, we just grab the events in the state/auth chain,
without calculating the state at those events, or backfilling their
`prev_events`.
Outliers typically arise when we fetch the auth chain or state for a given
event. When that happens, we mark all of those claimed auth events that we
don't already have as outliers because we haven't done the state calculation
ourself, or backfilled their `prev_events`.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in #12345 with your suggestion 👍

MadLittleMods added a commit that referenced this pull request Mar 31, 2022
@MadLittleMods
Clarify that we mark as outliers because we don't have any state for …
dc7edbd 
…them

As discussed at #12179 (comment)
@MadLittleMods MadLittleMods mentioned this pull request Mar 31, 2022
Merged
4 tasks
MadLittleMods added a commit that referenced this pull request Apr 6, 2022
@MadLittleMods
Clarify that we mark as outliers because we don't have any state for …
3500626 
…them (#12345)

As discussed at #12179 (comment)
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Apr 17, 2022
@Midar
Update chat/matrix-synapse to 1.56.0
3237623 
Synapse 1.56.0 (2022-04-05)
===========================

Synapse will now refuse to start up if open registration is enabled, in order to help mitigate
abuse across the federation. If you would like
to provide registration to anyone, consider adding [email](https://github.com/matrix-org/synapse/blob/8a519f8abc6de772167c2cca101d22ee2052fafc/docs/sample_config.yaml#L1285),
[recaptcha](https://matrix-org.github.io/synapse/v1.56/CAPTCHA_SETUP.html)
or [token-based](https://matrix-org.github.io/synapse/v1.56/usage/administration/admin_api/registration_tokens.html) verification
in order to prevent automated registration from bad actors.
This check can be disabled by setting the `enable_registration_without_verification` option in your
homeserver configuration file to `true`. More details are available in the
[upgrade notes](https://matrix-org.github.io/synapse/v1.56/upgrade.html#open-registration-without-verification-is-now-disabled-by-default).

Synapse will additionally now refuse to start when using PostgreSQL with a non-`C` values for `COLLATE` and `CTYPE`, unless
the config flag `allow_unsafe_locale`, found in the database section of the configuration file, is set to `true`. See the
[upgrade notes](https://matrix-org.github.io/synapse/v1.56/upgrade#change-in-behaviour-for-postgresql-databases-with-unsafe-locale)
for details.

Internal Changes
----------------

- Bump the version of `black` for compatibility with the latest `click` release. ([\#12320](matrix-org/synapse#12320))


Synapse 1.56.0rc1 (2022-03-29)
==============================

Features
--------

- Allow modules to store already existing 3PID associations. ([\#12195](matrix-org/synapse#12195))
- Allow registering server administrators using the module API. Contributed by Famedly. ([\#12250](matrix-org/synapse#12250))


Bugfixes
--------

- Fix a long-standing bug which caused the `/_matrix/federation/v1/state` and `/_matrix/federation/v1/state_ids` endpoints to return incorrect or invalid data when called for an event which we have stored as an "outlier". ([\#12087](matrix-org/synapse#12087))
- Fix a long-standing bug where events from ignored users would still be considered for relations. ([\#12227](matrix-org/synapse#12227), [\#12232](matrix-org/synapse#12232), [\#12285](matrix-org/synapse#12285))
- Fix a bug introduced in Synapse 1.53.0 where an unnecessary query could be performed when fetching bundled aggregations for threads. ([\#12228](matrix-org/synapse#12228))
- Fix a bug introduced in Synapse 1.52.0 where admins could not deactivate and GDPR-erase a user if Synapse was configured with limits on avatars. ([\#12261](matrix-org/synapse#12261))


Improved Documentation
----------------------

- Fix the link to the module documentation in the legacy spam checker warning message. ([\#12231](matrix-org/synapse#12231))
- Remove incorrect prefixes in the worker documentation for some endpoints. ([\#12243](matrix-org/synapse#12243))
- Correct `check_username_for_spam` annotations and docs. ([\#12246](matrix-org/synapse#12246))
- Correct Authentik OpenID typo, and add notes on troubleshooting. Contributed by @IronTooch. ([\#12275](matrix-org/synapse#12275))
- HAProxy reverse proxy guide update to stop sending IPv4-mapped address to homeserver. Contributed by @villepeh. ([\#12279](matrix-org/synapse#12279))


Internal Changes
----------------

- Rename `shared_rooms` to `mutual_rooms` ([MSC2666](matrix-org/matrix-spec-proposals#2666)), as per proposal changes. ([\#12036](matrix-org/synapse#12036))
- Remove check on `update_user_directory` for shared rooms handler ([MSC2666](matrix-org/matrix-spec-proposals#2666)), and update/expand documentation. ([\#12038](matrix-org/synapse#12038))
- Refactor `create_new_client_event` to use a new parameter, `state_event_ids`, which accurately describes the usage with [MSC2716](matrix-org/matrix-spec-proposals#2716) instead of abusing `auth_event_ids`. ([\#12083](matrix-org/synapse#12083), [\#12304](matrix-org/synapse#12304))
- Refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config flag `enable_registration_without_verification` is set to `true`. ([\#12091](matrix-org/synapse#12091), [\#12322](matrix-org/synapse#12322))
- Add tests for database transaction callbacks. ([\#12198](matrix-org/synapse#12198))
- Handle cancellation in `DatabasePool.runInteraction`. ([\#12199](matrix-org/synapse#12199))
- Add missing type hints for cache storage. ([\#12216](matrix-org/synapse#12216))
- Add missing type hints for storage. ([\#12248](matrix-org/synapse#12248), [\#12255](matrix-org/synapse#12255))
- Add type hints to tests files. ([\#12224](matrix-org/synapse#12224), [\#12240](matrix-org/synapse#12240), [\#12256](matrix-org/synapse#12256))
- Use type stubs for `psycopg2`. ([\#12269](matrix-org/synapse#12269))
- Improve type annotations for `execute_values`. ([\#12311](matrix-org/synapse#12311))
- Clean-up logic around rebasing URLs for URL image previews. ([\#12219](matrix-org/synapse#12219))
- Use the `ignored_users` table in additional places instead of re-parsing the account data. ([\#12225](matrix-org/synapse#12225))
- Refactor the relations endpoints to add a `RelationsHandler`. ([\#12237](matrix-org/synapse#12237))
- Generate announcement links in the release script. ([\#12242](matrix-org/synapse#12242))
- Improve error message when dependencies check finds a broken installation. ([\#12244](matrix-org/synapse#12244))
- Compress metrics HTTP resource when enabled. Contributed by Nick @ Beeper. ([\#12258](matrix-org/synapse#12258))
- Refuse to start if the PostgreSQL database has a non-`C` locale, unless the config flag `allow_unsafe_db_locale` is set to true. ([\#12262](matrix-org/synapse#12262), [\#12288](matrix-org/synapse#12288))
- Optionally include account validity expiration information to experimental [MSC3720](matrix-org/matrix-spec-proposals#3720) account status responses. ([\#12266](matrix-org/synapse#12266))
- Add a new cache `_get_membership_from_event_id` to speed up push rule calculations in large rooms. ([\#12272](matrix-org/synapse#12272))
- Re-enable Complement concurrency in CI. ([\#12283](matrix-org/synapse#12283))
- Remove unused test utilities. ([\#12291](matrix-org/synapse#12291))
- Enhance logging for inbound federation events. ([\#12301](matrix-org/synapse#12301))
- Fix compatibility with the recently-released Jinja 3.1. ([\#12313](matrix-org/synapse#12313))
- Avoid trying to calculate the state at outlier events. ([\#12314](matrix-org/synapse#12314))


Synapse 1.55.2 (2022-03-24)
===========================

This patch version reverts the earlier fixes from Synapse 1.55.1, which could cause problems in certain deployments, and instead adds a cap to the version of Jinja to be installed. Again, this is to fix an incompatibility with version 3.1.0 of the [Jinja](https://pypi.org/project/Jinja2/) library, and again, deployments of Synapse using the `matrixdotorg/synapse` Docker image or Debian packages from packages.matrix.org are not affected.

Internal Changes
----------------

- Pin Jinja to <3.1.0, as Synapse fails to start with Jinja 3.1.0. ([\#12297](matrix-org/synapse#12297))
- Revert changes from 1.55.1 as they caused problems with older versions of Jinja ([\#12296](matrix-org/synapse#12296))


Synapse 1.55.1 (2022-03-24)
===========================

This is a patch release that fixes an incompatibility with version 3.1.0 of the [Jinja](https://pypi.org/project/Jinja2/) library, released on March 24th, 2022. Deployments of Synapse using the `matrixdotorg/synapse` Docker image or Debian packages from packages.matrix.org are not affected.

Internal Changes
----------------

- Remove uses of the long-deprecated `jinja2.Markup` which would prevent Synapse from starting with Jinja 3.1.0 or above installed. ([\#12289](matrix-org/synapse#12289))


Synapse 1.55.0 (2022-03-22)
===========================

This release removes a workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. **This breaks compatibility with Mjolnir 1.3.1 and earlier. ([\#11700](matrix-org/synapse#11700; Mjolnir users should upgrade Mjolnir before upgrading Synapse to this version.

This release also moves the location of the `synctl` script; see the [upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#synctl-script-has-been-moved) for more details.


Internal Changes
----------------

- Tweak copy for default Single Sign-On account details template to better adhere to mobile app store guidelines. ([\#12265](matrix-org/synapse#12265), [\#12260](matrix-org/synapse#12260))


Synapse 1.55.0rc1 (2022-03-15)
==============================

Features
--------

- Add third-party rules callbacks `check_can_shutdown_room` and `check_can_deactivate_user`. ([\#12028](matrix-org/synapse#12028))
- Improve performance of logging in for large accounts. ([\#12132](matrix-org/synapse#12132))
- Add experimental env var `SYNAPSE_ASYNC_IO_REACTOR` that causes Synapse to use the asyncio reactor for Twisted. ([\#12135](matrix-org/synapse#12135))
- Support the stable identifiers from [MSC3440](matrix-org/matrix-spec-proposals#3440): threads. ([\#12151](matrix-org/synapse#12151))
- Add a new Jinja2 template filter to extract the local part of an email address. ([\#12212](matrix-org/synapse#12212))


Bugfixes
--------

- Use the proper serialization format for bundled thread aggregations. The bug has existed since Synapse v1.48.0. ([\#12090](matrix-org/synapse#12090))
- Fix a long-standing bug when redacting events with relations. ([\#12113](matrix-org/synapse#12113), [\#12121](matrix-org/synapse#12121), [\#12130](matrix-org/synapse#12130), [\#12189](matrix-org/synapse#12189))
- Fix a bug introduced in Synapse 1.7.2 whereby background updates are never run with the default background batch size. ([\#12157](matrix-org/synapse#12157))
- Fix a bug where non-standard information was returned from the `/hierarchy` API. Introduced in Synapse v1.41.0. ([\#12175](matrix-org/synapse#12175))
- Fix a bug introduced in Synapse 1.54.0 that broke background updates on sqlite homeservers while search was disabled. ([\#12215](matrix-org/synapse#12215))
- Fix a long-standing bug when a `filter` argument with `event_fields` which did not include the `unsigned` field could result in a 500 error on `/sync`. ([\#12234](matrix-org/synapse#12234))


Improved Documentation
----------------------

- Fix complexity checking config example in [Resource Constrained Devices](https://matrix-org.github.io/synapse/v1.54/other/running_synapse_on_single_board_computers.html) docs page. ([\#11998](matrix-org/synapse#11998))
- Improve documentation for demo scripts. ([\#12143](matrix-org/synapse#12143))
- Updates to the Room DAG concepts development document. ([\#12179](matrix-org/synapse#12179))
- Document that the `typing`, `to_device`, `account_data`, `receipts`, and `presence` stream writer can only be used on a single worker. ([\#12196](matrix-org/synapse#12196))
- Document that contributors can sign off privately by email. ([\#12204](matrix-org/synapse#12204))


Deprecations and Removals
-------------------------

- **Remove workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. Breaks compatibility with Mjolnir 1.3.1 and earlier. ([\#11700](matrix-org/synapse#11700
- **`synctl` has been moved into into `synapse._scripts` and is exposed as an entry point; see [upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#synctl-script-has-been-moved). ([\#12140](matrix-org/synapse#12140))
- Remove backwards compatibilty with pagination tokens from the `/relations` and `/aggregations` endpoints generated from Synapse < v1.52.0. ([\#12138](matrix-org/synapse#12138))
- The groups/communities feature in Synapse has been deprecated. ([\#12200](matrix-org/synapse#12200))


Internal Changes
----------------

- Simplify the `ApplicationService` class' set of public methods related to interest checking. ([\#11915](matrix-org/synapse#11915))
- Add config settings for background update parameters. ([\#11980](matrix-org/synapse#11980))
- Correct type hints for txredis. ([\#12042](matrix-org/synapse#12042))
- Limit the size of `aggregation_key` on annotations. ([\#12101](matrix-org/synapse#12101))
- Add type hints to tests files. ([\#12108](matrix-org/synapse#12108), [\#12146](matrix-org/synapse#12146), [\#12207](matrix-org/synapse#12207), [\#12208](matrix-org/synapse#12208))
- Move scripts to Synapse package and expose as setuptools entry points. ([\#12118](matrix-org/synapse#12118))
- Add support for cancellation to `ReadWriteLock`. ([\#12120](matrix-org/synapse#12120))
- Fix data validation to compare to lists, not sequences. ([\#12128](matrix-org/synapse#12128))
- Fix CI not attaching source distributions and wheels to the GitHub releases. ([\#12131](matrix-org/synapse#12131))
- Remove unused mocks from `test_typing`. ([\#12136](matrix-org/synapse#12136))
- Give `scripts-dev` scripts suffixes for neater CI config. ([\#12137](matrix-org/synapse#12137))
- Move the snapcraft configuration file to `contrib`. ([\#12142](matrix-org/synapse#12142))
- Enable [MSC3030](matrix-org/matrix-spec-proposals#3030) Complement tests in CI. ([\#12144](matrix-org/synapse#12144))
- Enable [MSC2716](matrix-org/matrix-spec-proposals#2716) Complement tests in CI. ([\#12145](matrix-org/synapse#12145))
- Add test for `ObservableDeferred`'s cancellation behaviour. ([\#12149](matrix-org/synapse#12149))
- Use `ParamSpec` in type hints for `synapse.logging.context`. ([\#12150](matrix-org/synapse#12150))
- Prune unused jobs from `tox` config. ([\#12152](matrix-org/synapse#12152))
- Move CI checks out of tox, to facilitate a move to using poetry. ([\#12153](matrix-org/synapse#12153))
- Avoid generating state groups for local out-of-band leaves. ([\#12154](matrix-org/synapse#12154))
- Avoid trying to calculate the state at outlier events. ([\#12155](matrix-org/synapse#12155), [\#12173](matrix-org/synapse#12173), [\#12202](matrix-org/synapse#12202))
- Fix some type annotations. ([\#12156](matrix-org/synapse#12156))
- Add type hints for `ObservableDeferred` attributes. ([\#12159](matrix-org/synapse#12159))
- Use a prebuilt Action for the `tests-done` CI job. ([\#12161](matrix-org/synapse#12161))
- Reduce number of DB queries made during processing of `/sync`. ([\#12163](matrix-org/synapse#12163))
- Add `delay_cancellation` utility function, which behaves like `stop_cancellation` but waits until the original `Deferred` resolves before raising a `CancelledError`. ([\#12180](matrix-org/synapse#12180))
- Retry HTTP replication failures, this should prevent 502's when restarting stateful workers (main, event persisters, stream writers). Contributed by Nick @ Beeper. ([\#12182](matrix-org/synapse#12182))
- Add cancellation support to `@cached` and `@cachedList` decorators. ([\#12183](matrix-org/synapse#12183))
- Remove unused variables. ([\#12187](matrix-org/synapse#12187))
- Add combined test for HTTP pusher and push rule. Contributed by Nick @ Beeper. ([\#12188](matrix-org/synapse#12188))
- Rename `HomeServer.get_tcp_replication` to `get_replication_command_handler`. ([\#12192](matrix-org/synapse#12192))
- Remove some dead code. ([\#12197](matrix-org/synapse#12197))
- Fix a misleading comment in the function `check_event_for_spam`. ([\#12203](matrix-org/synapse#12203))
- Remove unnecessary `pass` statements. ([\#12206](matrix-org/synapse#12206))
- Update the SSO username picker template to comply with SIWA guidelines. ([\#12210](matrix-org/synapse#12210))
- Improve code documentation for the typing stream over replication. ([\#12211](matrix-org/synapse#12211))
Fizzadar added a commit to Fizzadar/synapse that referenced this pull request Apr 25, 2022
@Fizzadar
Merge tag 'v1.55.0' into beeper-master
86d3eb3 
Synapse 1.55.0 (2022-03-22)
===========================

This release removes a workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. **This breaks compatibility with Mjolnir 1.3.1 and earlier. ([\matrix-org#11700](matrix-org#11700; Mjolnir users should upgrade Mjolnir before upgrading Synapse to this version.

This release also moves the location of the `synctl` script; see the [upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#synctl-script-has-been-moved) for more details.

Internal Changes
----------------

- Tweak copy for default Single Sign-On account details template to better adhere to mobile app store guidelines. ([\matrix-org#12265](matrix-org#12265), [\matrix-org#12260](matrix-org#12260))

Synapse 1.55.0rc1 (2022-03-15)
==============================

Features
--------

- Add third-party rules callbacks `check_can_shutdown_room` and `check_can_deactivate_user`. ([\matrix-org#12028](matrix-org#12028))
- Improve performance of logging in for large accounts. ([\matrix-org#12132](matrix-org#12132))
- Add experimental env var `SYNAPSE_ASYNC_IO_REACTOR` that causes Synapse to use the asyncio reactor for Twisted. ([\matrix-org#12135](matrix-org#12135))
- Support the stable identifiers from [MSC3440](matrix-org/matrix-spec-proposals#3440): threads. ([\matrix-org#12151](matrix-org#12151))
- Add a new Jinja2 template filter to extract the local part of an email address. ([\matrix-org#12212](matrix-org#12212))

Bugfixes
--------

- Use the proper serialization format for bundled thread aggregations. The bug has existed since Synapse v1.48.0. ([\matrix-org#12090](matrix-org#12090))
- Fix a long-standing bug when redacting events with relations. ([\matrix-org#12113](matrix-org#12113), [\matrix-org#12121](matrix-org#12121), [\matrix-org#12130](matrix-org#12130), [\matrix-org#12189](matrix-org#12189))
- Fix a bug introduced in Synapse 1.7.2 whereby background updates are never run with the default background batch size. ([\matrix-org#12157](matrix-org#12157))
- Fix a bug where non-standard information was returned from the `/hierarchy` API. Introduced in Synapse v1.41.0. ([\matrix-org#12175](matrix-org#12175))
- Fix a bug introduced in Synapse 1.54.0 that broke background updates on sqlite homeservers while search was disabled. ([\matrix-org#12215](matrix-org#12215))
- Fix a long-standing bug when a `filter` argument with `event_fields` which did not include the `unsigned` field could result in a 500 error on `/sync`. ([\matrix-org#12234](matrix-org#12234))

Improved Documentation
----------------------

- Fix complexity checking config example in [Resource Constrained Devices](https://matrix-org.github.io/synapse/v1.54/other/running_synapse_on_single_board_computers.html) docs page. ([\matrix-org#11998](matrix-org#11998))
- Improve documentation for demo scripts. ([\matrix-org#12143](matrix-org#12143))
- Updates to the Room DAG concepts development document. ([\matrix-org#12179](matrix-org#12179))
- Document that the `typing`, `to_device`, `account_data`, `receipts`, and `presence` stream writer can only be used on a single worker. ([\matrix-org#12196](matrix-org#12196))
- Document that contributors can sign off privately by email. ([\matrix-org#12204](matrix-org#12204))

Deprecations and Removals
-------------------------

- **Remove workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. Breaks compatibility with Mjolnir 1.3.1 and earlier. ([\matrix-org#11700](matrix-org#11700
- **`synctl` has been moved into into `synapse._scripts` and is exposed as an entry point; see [upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#synctl-script-has-been-moved). ([\matrix-org#12140](matrix-org#12140))
- Remove backwards compatibilty with pagination tokens from the `/relations` and `/aggregations` endpoints generated from Synapse < v1.52.0. ([\matrix-org#12138](matrix-org#12138))
- The groups/communities feature in Synapse has been deprecated. ([\matrix-org#12200](matrix-org#12200))

Internal Changes
----------------

- Simplify the `ApplicationService` class' set of public methods related to interest checking. ([\matrix-org#11915](matrix-org#11915))
- Add config settings for background update parameters. ([\matrix-org#11980](matrix-org#11980))
- Correct type hints for txredis. ([\matrix-org#12042](matrix-org#12042))
- Limit the size of `aggregation_key` on annotations. ([\matrix-org#12101](matrix-org#12101))
- Add type hints to tests files. ([\matrix-org#12108](matrix-org#12108), [\matrix-org#12146](matrix-org#12146), [\matrix-org#12207](matrix-org#12207), [\matrix-org#12208](matrix-org#12208))
- Move scripts to Synapse package and expose as setuptools entry points. ([\matrix-org#12118](matrix-org#12118))
- Add support for cancellation to `ReadWriteLock`. ([\matrix-org#12120](matrix-org#12120))
- Fix data validation to compare to lists, not sequences. ([\matrix-org#12128](matrix-org#12128))
- Fix CI not attaching source distributions and wheels to the GitHub releases. ([\matrix-org#12131](matrix-org#12131))
- Remove unused mocks from `test_typing`. ([\matrix-org#12136](matrix-org#12136))
- Give `scripts-dev` scripts suffixes for neater CI config. ([\matrix-org#12137](matrix-org#12137))
- Move the snapcraft configuration file to `contrib`. ([\matrix-org#12142](matrix-org#12142))
- Enable [MSC3030](matrix-org/matrix-spec-proposals#3030) Complement tests in CI. ([\matrix-org#12144](matrix-org#12144))
- Enable [MSC2716](matrix-org/matrix-spec-proposals#2716) Complement tests in CI. ([\matrix-org#12145](matrix-org#12145))
- Add test for `ObservableDeferred`'s cancellation behaviour. ([\matrix-org#12149](matrix-org#12149))
- Use `ParamSpec` in type hints for `synapse.logging.context`. ([\matrix-org#12150](matrix-org#12150))
- Prune unused jobs from `tox` config. ([\matrix-org#12152](matrix-org#12152))
- Move CI checks out of tox, to facilitate a move to using poetry. ([\matrix-org#12153](matrix-org#12153))
- Avoid generating state groups for local out-of-band leaves. ([\matrix-org#12154](matrix-org#12154))
- Avoid trying to calculate the state at outlier events. ([\matrix-org#12155](matrix-org#12155), [\matrix-org#12173](matrix-org#12173), [\matrix-org#12202](matrix-org#12202))
- Fix some type annotations. ([\matrix-org#12156](matrix-org#12156))
- Add type hints for `ObservableDeferred` attributes. ([\matrix-org#12159](matrix-org#12159))
- Use a prebuilt Action for the `tests-done` CI job. ([\matrix-org#12161](matrix-org#12161))
- Reduce number of DB queries made during processing of `/sync`. ([\matrix-org#12163](matrix-org#12163))
- Add `delay_cancellation` utility function, which behaves like `stop_cancellation` but waits until the original `Deferred` resolves before raising a `CancelledError`. ([\matrix-org#12180](matrix-org#12180))
- Retry HTTP replication failures, this should prevent 502's when restarting stateful workers (main, event persisters, stream writers). Contributed by Nick @ Beeper. ([\matrix-org#12182](matrix-org#12182))
- Add cancellation support to `@cached` and `@cachedList` decorators. ([\matrix-org#12183](matrix-org#12183))
- Remove unused variables. ([\matrix-org#12187](matrix-org#12187))
- Add combined test for HTTP pusher and push rule. Contributed by Nick @ Beeper. ([\matrix-org#12188](matrix-org#12188))
- Rename `HomeServer.get_tcp_replication` to `get_replication_command_handler`. ([\matrix-org#12192](matrix-org#12192))
- Remove some dead code. ([\matrix-org#12197](matrix-org#12197))
- Fix a misleading comment in the function `check_event_for_spam`. ([\matrix-org#12203](matrix-org#12203))
- Remove unnecessary `pass` statements. ([\matrix-org#12206](matrix-org#12206))
- Update the SSO username picker template to comply with SIWA guidelines. ([\matrix-org#12210](matrix-org#12210))
- Improve code documentation for the typing stream over replication. ([\matrix-org#12211](matrix-org#12211))

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmI53SwACgkQiISIDS7+
# X/RlgQ/9HX3moAXQw67uIII5BnSGAYdgovL2uM990urQ6HVYRxcnidRlWQUyqlJ0
# qbvpOrJXRpK5MvoFyrJg+ryJLEncoUiccs5WQHNt4t22zmUXusLH/Bxxc/U24s5D
# 3gQVhYdjiyyR0dIzr/EwZsYGSgl9wANoKVJXbRInrE7gdMP5RT/IcifWBZETNWjQ
# zPGp3R5RDYzFecDiZUcGJCldXC+WPOw0YlhMdR7Rm5NkiE6eM1kAREH+K5vxBM7i
# H+nP//NLl+wL4nxb3f3paEgCakPkB+xuC6bZZ3NA7LfMw2OKBHdUqEl4rmXr4vCS
# lboF9kDhb42uIKgVPBnVcf3hv4m5NbVA7f6k3tpFJOvqk9G58UhnqcjdXS8DBso6
# gQpk7Zk6ILU0j8i7N21R/aZLSwmQ96h1O3lNO+43usPbh6WjqMIHR/kv/ADyZEhZ
# /6taXiOHtg7KaqtpHAy2UVuWjJ8c3a0vHbReRe77S+yMD7iggHbBTpSNMfgcM4yi
# gimKq55Z0qhPLI7vjr4IOB5irEd64ciKnUI3IQfkyFoCMyJHo0fw7+boC/Kc5FgM
# O7V69r+CBfJ9SsGeSlheiCTv6fGyh4n960ya2BQY+BHDjTFW+BUkTyjXkIeiIJ04
# CihTgluISePbrdRKL4JkceL+Im6NmpJhTOe1+Pjsncy/8WkOeos=
# =HvNr
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue Mar 22 14:29:00 2022 GMT
# gpg:                using RSA key 8101B7D59DF5ECDAD232DD108884880D2EFE5FF4
# gpg: Can't check signature: No public key

# Conflicts:
#	synapse/appservice/__init__.py
#	synapse/handlers/pagination.py
#	synapse/push/baserules.py
babolivier added a commit to matrix-org/synapse-dinsic that referenced this pull request Apr 28, 2022
@babolivier
Merge tag 'v1.55.0' into babolivier/dinsic_1.58
277369b 
Synapse 1.55.0 (2022-03-22)
===========================

This release removes a workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. **This breaks compatibility with Mjolnir 1.3.1 and earlier. ([\#11700](matrix-org/synapse#11700; Mjolnir users should upgrade Mjolnir before upgrading Synapse to this version.

This release also moves the location of the `synctl` script; see the [upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#synctl-script-has-been-moved) for more details.

Internal Changes
----------------

- Tweak copy for default Single Sign-On account details template to better adhere to mobile app store guidelines. ([\#12265](matrix-org/synapse#12265), [\#12260](matrix-org/synapse#12260))

Synapse 1.55.0rc1 (2022-03-15)
==============================

Features
--------

- Add third-party rules callbacks `check_can_shutdown_room` and `check_can_deactivate_user`. ([\#12028](matrix-org/synapse#12028))
- Improve performance of logging in for large accounts. ([\#12132](matrix-org/synapse#12132))
- Add experimental env var `SYNAPSE_ASYNC_IO_REACTOR` that causes Synapse to use the asyncio reactor for Twisted. ([\#12135](matrix-org/synapse#12135))
- Support the stable identifiers from [MSC3440](matrix-org/matrix-spec-proposals#3440): threads. ([\#12151](matrix-org/synapse#12151))
- Add a new Jinja2 template filter to extract the local part of an email address. ([\#12212](matrix-org/synapse#12212))

Bugfixes
--------

- Use the proper serialization format for bundled thread aggregations. The bug has existed since Synapse v1.48.0. ([\#12090](matrix-org/synapse#12090))
- Fix a long-standing bug when redacting events with relations. ([\#12113](matrix-org/synapse#12113), [\#12121](matrix-org/synapse#12121), [\#12130](matrix-org/synapse#12130), [\#12189](matrix-org/synapse#12189))
- Fix a bug introduced in Synapse 1.7.2 whereby background updates are never run with the default background batch size. ([\#12157](matrix-org/synapse#12157))
- Fix a bug where non-standard information was returned from the `/hierarchy` API. Introduced in Synapse v1.41.0. ([\#12175](matrix-org/synapse#12175))
- Fix a bug introduced in Synapse 1.54.0 that broke background updates on sqlite homeservers while search was disabled. ([\#12215](matrix-org/synapse#12215))
- Fix a long-standing bug when a `filter` argument with `event_fields` which did not include the `unsigned` field could result in a 500 error on `/sync`. ([\#12234](matrix-org/synapse#12234))

Improved Documentation
----------------------

- Fix complexity checking config example in [Resource Constrained Devices](https://matrix-org.github.io/synapse/v1.54/other/running_synapse_on_single_board_computers.html) docs page. ([\#11998](matrix-org/synapse#11998))
- Improve documentation for demo scripts. ([\#12143](matrix-org/synapse#12143))
- Updates to the Room DAG concepts development document. ([\#12179](matrix-org/synapse#12179))
- Document that the `typing`, `to_device`, `account_data`, `receipts`, and `presence` stream writer can only be used on a single worker. ([\#12196](matrix-org/synapse#12196))
- Document that contributors can sign off privately by email. ([\#12204](matrix-org/synapse#12204))

Deprecations and Removals
-------------------------

- **Remove workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. Breaks compatibility with Mjolnir 1.3.1 and earlier. ([\#11700](matrix-org/synapse#11700
- **`synctl` has been moved into into `synapse._scripts` and is exposed as an entry point; see [upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#synctl-script-has-been-moved). ([\#12140](matrix-org/synapse#12140))
- Remove backwards compatibilty with pagination tokens from the `/relations` and `/aggregations` endpoints generated from Synapse < v1.52.0. ([\#12138](matrix-org/synapse#12138))
- The groups/communities feature in Synapse has been deprecated. ([\#12200](matrix-org/synapse#12200))

Internal Changes
----------------

- Simplify the `ApplicationService` class' set of public methods related to interest checking. ([\#11915](matrix-org/synapse#11915))
- Add config settings for background update parameters. ([\#11980](matrix-org/synapse#11980))
- Correct type hints for txredis. ([\#12042](matrix-org/synapse#12042))
- Limit the size of `aggregation_key` on annotations. ([\#12101](matrix-org/synapse#12101))
- Add type hints to tests files. ([\#12108](matrix-org/synapse#12108), [\#12146](matrix-org/synapse#12146), [\#12207](matrix-org/synapse#12207), [\#12208](matrix-org/synapse#12208))
- Move scripts to Synapse package and expose as setuptools entry points. ([\#12118](matrix-org/synapse#12118))
- Add support for cancellation to `ReadWriteLock`. ([\#12120](matrix-org/synapse#12120))
- Fix data validation to compare to lists, not sequences. ([\#12128](matrix-org/synapse#12128))
- Fix CI not attaching source distributions and wheels to the GitHub releases. ([\#12131](matrix-org/synapse#12131))
- Remove unused mocks from `test_typing`. ([\#12136](matrix-org/synapse#12136))
- Give `scripts-dev` scripts suffixes for neater CI config. ([\#12137](matrix-org/synapse#12137))
- Move the snapcraft configuration file to `contrib`. ([\#12142](matrix-org/synapse#12142))
- Enable [MSC3030](matrix-org/matrix-spec-proposals#3030) Complement tests in CI. ([\#12144](matrix-org/synapse#12144))
- Enable [MSC2716](matrix-org/matrix-spec-proposals#2716) Complement tests in CI. ([\#12145](matrix-org/synapse#12145))
- Add test for `ObservableDeferred`'s cancellation behaviour. ([\#12149](matrix-org/synapse#12149))
- Use `ParamSpec` in type hints for `synapse.logging.context`. ([\#12150](matrix-org/synapse#12150))
- Prune unused jobs from `tox` config. ([\#12152](matrix-org/synapse#12152))
- Move CI checks out of tox, to facilitate a move to using poetry. ([\#12153](matrix-org/synapse#12153))
- Avoid generating state groups for local out-of-band leaves. ([\#12154](matrix-org/synapse#12154))
- Avoid trying to calculate the state at outlier events. ([\#12155](matrix-org/synapse#12155), [\#12173](matrix-org/synapse#12173), [\#12202](matrix-org/synapse#12202))
- Fix some type annotations. ([\#12156](matrix-org/synapse#12156))
- Add type hints for `ObservableDeferred` attributes. ([\#12159](matrix-org/synapse#12159))
- Use a prebuilt Action for the `tests-done` CI job. ([\#12161](matrix-org/synapse#12161))
- Reduce number of DB queries made during processing of `/sync`. ([\#12163](matrix-org/synapse#12163))
- Add `delay_cancellation` utility function, which behaves like `stop_cancellation` but waits until the original `Deferred` resolves before raising a `CancelledError`. ([\#12180](matrix-org/synapse#12180))
- Retry HTTP replication failures, this should prevent 502's when restarting stateful workers (main, event persisters, stream writers). Contributed by Nick @ Beeper. ([\#12182](matrix-org/synapse#12182))
- Add cancellation support to `@cached` and `@cachedList` decorators. ([\#12183](matrix-org/synapse#12183))
- Remove unused variables. ([\#12187](matrix-org/synapse#12187))
- Add combined test for HTTP pusher and push rule. Contributed by Nick @ Beeper. ([\#12188](matrix-org/synapse#12188))
- Rename `HomeServer.get_tcp_replication` to `get_replication_command_handler`. ([\#12192](matrix-org/synapse#12192))
- Remove some dead code. ([\#12197](matrix-org/synapse#12197))
- Fix a misleading comment in the function `check_event_for_spam`. ([\#12203](matrix-org/synapse#12203))
- Remove unnecessary `pass` statements. ([\#12206](matrix-org/synapse#12206))
- Update the SSO username picker template to comply with SIWA guidelines. ([\#12210](matrix-org/synapse#12210))
- Improve code documentation for the typing stream over replication. ([\#12211](matrix-org/synapse#12211))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@MadLittleMods MadLittleMods MadLittleMods left review comments

@clokep clokep clokep approved these changes

Assignees
No one assigned
Labels
A-Docs things relating to the documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@richvdh @clokep @MadLittleMods