Add a class UnpersistedEventContext to allow for the batching up of storing state groups #14675

H-Shay · 2022-12-13T23:36:56Z

This is preparatory work for the batching up of storing state groups: #13961. This PR adds a class, UnpersistedEventContext, which is created in handle_new_client_event.

Before this change, event contexts in handle_new_client_event were created by calling compute_event_context, which calculates and stores the state groups for each event individually. To allow for more flexibility when that function is called and state groups are stored I have created the class UnpersistedEventContext, which contains enough information to verify the associated event but has not had it's current state group calculated or stored.

To transform an UnpersistedEventContext to an EventContext the function persist is called on the unpersisted context. It will calculate and return an event context suitable for sending to the database with the event.

This PR should be reviewable commit-by-commit.

…-org/synapse into shay/rework_event_context

DMRobertson

I haven't been following your work on this---it looks deep, tricky, and carefully considered.

It looks sane, though I've left a few drive-by comments. I think we probably want Erik to take a look at this in the new year though(?)

synapse/events/snapshot.py

synapse/handlers/federation.py

synapse/handlers/message.py

synapse/state/__init__.py

synapse/handlers/message.py

…isted event context and then persists it

erikjohnston

This is looking really good thanks! A bunch of comments are just around wording/docs, otherwise I think there are a few potential simplifications. I think some of the changes aren't necessary anymore?

synapse/events/snapshot.py

synapse/handlers/message.py

synapse/state/__init__.py

synapse/handlers/message.py

synapse/state/__init__.py

synapse/events/snapshot.py

synapse/events/third_party_rules.py

synapse/state/__init__.py

synapse/events/snapshot.py

synapse/state/__init__.py

H-Shay · 2023-01-26T20:33:01Z

The complement failure on this is a known flake.

Edit: hopefully fixed by matrix-org/complement#591

erikjohnston

Almost there

synapse/events/snapshot.py

synapse/state/__init__.py

…-org/synapse into shay/rework_event_context

H-Shay · 2023-02-08T19:08:05Z

I think this is correct? Brain melting aside?

erikjohnston

Other than docstrings, LGTM!

synapse/events/snapshot.py

@thezaidbintariq

Synapse 1.78.0 (2023-02-28) =========================== Bugfixes -------- - Fix a bug introduced in Synapse 1.76 where 5s delays would occasionally occur in deployments using workers. ([\#15150](matrix-org/synapse#15150)) Synapse 1.78.0rc1 (2023-02-21) ============================== Features -------- - Implement the experimental `exact_event_match` push rule condition from [MSC3758](matrix-org/matrix-spec-proposals#3758). ([\#14964](matrix-org/synapse#14964)) - Add account data to the command line [user data export tool](https://matrix-org.github.io/synapse/v1.78/usage/administration/admin_faq.html#how-can-i-export-user-data). ([\#14969](matrix-org/synapse#14969)) - Implement [MSC3873](matrix-org/matrix-spec-proposals#3873) to disambiguate push rule keys with dots in them. ([\#15004](matrix-org/synapse#15004)) - Allow Synapse to use a specific Redis [logical database](https://redis.io/commands/select/) in worker-mode deployments. ([\#15034](matrix-org/synapse#15034)) - Tag opentracing spans for federation requests with the name of the worker serving the request. ([\#15042](matrix-org/synapse#15042)) - Implement the experimental `exact_event_property_contains` push rule condition from [MSC3966](matrix-org/matrix-spec-proposals#3966). ([\#15045](matrix-org/synapse#15045)) - Remove spurious `dont_notify` action from the defaults for the `.m.rule.reaction` pushrule. ([\#15073](matrix-org/synapse#15073)) - Update the error code returned when user sends a duplicate annotation. ([\#15075](matrix-org/synapse#15075)) Bugfixes -------- - Prevent clients from reporting nonexistent events. ([\#13779](matrix-org/synapse#13779)) - Return spec-compliant JSON errors when unknown endpoints are requested. ([\#14605](matrix-org/synapse#14605)) - Fix a long-standing bug where the room aliases returned could be corrupted. ([\#15038](matrix-org/synapse#15038)) - Fix a bug introduced in Synapse 1.76.0 where partially-joined rooms could not be deleted using the [purge room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api). ([\#15068](matrix-org/synapse#15068)) - Fix a long-standing bug where federated joins would fail if the first server in the list of servers to try is not in the room. ([\#15074](matrix-org/synapse#15074)) - Fix a bug introduced in Synapse v1.74.0 where searching with colons when using ICU for search term tokenisation would fail with an error. ([\#15079](matrix-org/synapse#15079)) - Reduce the likelihood of a rare race condition where rejoining a restricted room over federation would fail. ([\#15080](matrix-org/synapse#15080)) - Fix a bug introduced in Synapse 1.76 where workers would fail to start if the `health` listener was configured. ([\#15096](matrix-org/synapse#15096)) - Fix a bug introduced in Synapse 1.75 where the [portdb script](https://matrix-org.github.io/synapse/release-v1.78/postgres.html#porting-from-sqlite) would fail to run after a room had been faster-joined. ([\#15108](matrix-org/synapse#15108)) Improved Documentation ---------------------- - Document how to start Synapse with Poetry. Contributed by @thezaidbintariq. ([\#14892](matrix-org/synapse#14892), [\#15022](matrix-org/synapse#15022)) - Update delegation documentation to clarify that SRV DNS delegation does not eliminate all needs to serve files from .well-known locations. Contributed by @williamkray. ([\#14959](matrix-org/synapse#14959)) - Fix a mistake in registration_shared_secret_path docs. ([\#15078](matrix-org/synapse#15078)) - Refer to a more recent blog post on the [Database Maintenance Tools](https://matrix-org.github.io/synapse/latest/usage/administration/database_maintenance_tools.html) page. Contributed by @jahway603. ([\#15083](matrix-org/synapse#15083)) Internal Changes ---------------- - Re-type hint some collections as read-only. ([\#13755](matrix-org/synapse#13755)) - Faster joins: don't stall when another user joins during a partial-state room resync. ([\#14606](matrix-org/synapse#14606)) - Add a class `UnpersistedEventContext` to allow for the batching up of storing state groups. ([\#14675](matrix-org/synapse#14675)) - Add a check to ensure that locked dependencies have source distributions available. ([\#14742](matrix-org/synapse#14742)) - Tweak comment on `_is_local_room_accessible` as part of room visibility in `/hierarchy` to clarify the condition for a room being visible. ([\#14834](matrix-org/synapse#14834)) - Prevent `WARNING: there is already a transaction in progress` lines appearing in PostgreSQL's logs on some occasions. ([\#14840](matrix-org/synapse#14840)) - Use `StrCollection` to avoid potential bugs with `Collection[str]`. ([\#14929](matrix-org/synapse#14929)) - Improve performance of `/sync` in a few situations. ([\#14973](matrix-org/synapse#14973)) - Limit concurrent event creation for a room to avoid state resolution when sending bursts of events to a local room. ([\#14977](matrix-org/synapse#14977)) - Skip calculating unread push actions in /sync when enable_push is false. ([\#14980](matrix-org/synapse#14980)) - Add a schema dump symlinks inside `contrib`, to make it easier for IDEs to interrogate Synapse's database schema. ([\#14982](matrix-org/synapse#14982)) - Improve type hints. ([\#15008](matrix-org/synapse#15008), [\#15026](matrix-org/synapse#15026), [\#15027](matrix-org/synapse#15027), [\#15028](matrix-org/synapse#15028), [\#15031](matrix-org/synapse#15031), [\#15035](matrix-org/synapse#15035), [\#15052](matrix-org/synapse#15052), [\#15072](matrix-org/synapse#15072), [\#15084](matrix-org/synapse#15084)) - Update [MSC3952](matrix-org/matrix-spec-proposals#3952) support based on changes to the MSC. ([\#15037](matrix-org/synapse#15037)) - Avoid mutating a cached value in `get_user_devices_from_cache`. ([\#15040](matrix-org/synapse#15040)) - Fix a rare exception in logs on start up. ([\#15041](matrix-org/synapse#15041)) - Update pyo3-log to v0.8.1. ([\#15043](matrix-org/synapse#15043)) - Avoid mutating cached values in `_generate_sync_entry_for_account_data`. ([\#15047](matrix-org/synapse#15047)) - Refactor arguments of `try_unbind_threepid` and `_try_unbind_threepid_with_id_server` to not use dictionaries. ([\#15053](matrix-org/synapse#15053)) - Merge debug logging from the hotfixes branch. ([\#15054](matrix-org/synapse#15054)) - Faster joins: omit device list updates originating from partial state rooms in /sync responses without lazy loading of members enabled. ([\#15069](matrix-org/synapse#15069)) - Fix clashing database transaction name. ([\#15070](matrix-org/synapse#15070)) - Upper-bound frozendict dependency. This works around us being unable to test installing our wheels against Python 3.11 in CI. ([\#15114](matrix-org/synapse#15114)) - Tweak logging for when a worker waits for its view of a replication stream to catch up. ([\#15120](matrix-org/synapse#15120)) <details><summary>Locked dependency updates</summary> - Bump bleach from 5.0.1 to 6.0.0. ([\#15059](matrix-org/synapse#15059)) - Bump cryptography from 38.0.4 to 39.0.1. ([\#15020](matrix-org/synapse#15020)) - Bump ruff version from 0.0.230 to 0.0.237. ([\#15033](matrix-org/synapse#15033)) - Bump dtolnay/rust-toolchain from 9cd00a88a73addc8617065438eff914dd08d0955 to 25dc93b901a87e864900a8aec6c12e9aa794c0c3. ([\#15060](matrix-org/synapse#15060)) - Bump systemd-python from 234 to 235. ([\#15061](matrix-org/synapse#15061)) - Bump serde_json from 1.0.92 to 1.0.93. ([\#15062](matrix-org/synapse#15062)) - Bump types-requests from 2.28.11.8 to 2.28.11.12. ([\#15063](matrix-org/synapse#15063)) - Bump types-pillow from 9.4.0.5 to 9.4.0.10. ([\#15064](matrix-org/synapse#15064)) - Bump sentry-sdk from 1.13.0 to 1.15.0. ([\#15065](matrix-org/synapse#15065)) - Bump types-jsonschema from 4.17.0.3 to 4.17.0.5. ([\#15099](matrix-org/synapse#15099)) - Bump types-bleach from 5.0.3.1 to 6.0.0.0. ([\#15100](matrix-org/synapse#15100)) - Bump dtolnay/rust-toolchain from 25dc93b901a87e864900a8aec6c12e9aa794c0c3 to e12eda571dc9a5ee5d58eecf4738ec291c66f295. ([\#15101](matrix-org/synapse#15101)) - Bump dawidd6/action-download-artifact from 2.24.3 to 2.25.0. ([\#15102](matrix-org/synapse#15102)) - Bump types-pillow from 9.4.0.10 to 9.4.0.13. ([\#15104](matrix-org/synapse#15104)) - Bump types-setuptools from 67.1.0.0 to 67.3.0.1. ([\#15105](matrix-org/synapse#15105)) </details>

H-Shay added 6 commits December 13, 2022 13:38

add class UnpersistedEventContext

286469f

modify create new client event to create unpersistedeventcontexts

6322dac

persist event contexts after creation

54fa7f0

fix tests to persist unpersisted event contexts

d23d1be

cleanup

b483243

misc lints + cleanup

c008b6f

H-Shay requested a review from a team as a code owner December 13, 2022 23:36

H-Shay added 4 commits December 13, 2022 15:42

Merge branch 'develop' into shay/rework_event_context

c6ef7bb

changelog + fix comments

0f07c0e

Merge branch 'shay/rework_event_context' of https://github.com/matrix…

a66cc71

…-org/synapse into shay/rework_event_context

lints

78a71ed

H-Shay marked this pull request as draft December 14, 2022 02:39

H-Shay removed the request for review from a team December 14, 2022 03:09

H-Shay added 2 commits December 13, 2022 21:13

fix batch insertion?

6321576

reduce redundant calculation

d6454ce

H-Shay marked this pull request as ready for review December 14, 2022 23:27

H-Shay requested a review from a team December 14, 2022 23:27

DMRobertson reviewed Dec 15, 2022

View reviewed changes

DMRobertson requested a review from a team December 15, 2022 13:03

erikjohnston self-assigned this Jan 3, 2023

H-Shay added 7 commits January 9, 2023 10:35

add unpersisted event classes

8b99436

rework compute_event_context, split into function that returns unpers…

1668fd2

…isted event context and then persists it

use calculate_context_info to create unpersisted event contexts

7e494fc

update typing

61c8de0

Merge branch 'develop' into shay/rework_event_context

ed2e305

$%#^&*

f7eafcf

black

0402917

erikjohnston reviewed Jan 11, 2023

View reviewed changes

H-Shay added 2 commits January 11, 2023 20:29

fix comments and consolidate classes, use attr.s for class

373e331

requested changes

408fa21

lint

345d746

H-Shay requested a review from erikjohnston January 12, 2023 05:22

erikjohnston reviewed Jan 12, 2023

View reviewed changes

H-Shay added 2 commits January 19, 2023 16:52

Merge branch 'develop' into shay/rework_event_context

b9ee9d3

requested changes

cdce9a2

H-Shay requested a review from erikjohnston January 20, 2023 04:11

erikjohnston reviewed Jan 20, 2023

View reviewed changes

synapse/events/snapshot.py Outdated Show resolved Hide resolved

synapse/events/snapshot.py Outdated Show resolved Hide resolved

synapse/state/__init__.py Outdated Show resolved Hide resolved

H-Shay added 2 commits January 23, 2023 12:46

requested changes

b69a87c

refactor to be stupidly explicit

8830c94

H-Shay requested a review from erikjohnston January 26, 2023 01:35

H-Shay mentioned this pull request Jan 26, 2023

Batch up storing state groups when creating new room #14918

Merged

Merge branch 'develop' into shay/rework_event_context

d4c380d

H-Shay mentioned this pull request Feb 6, 2023

Reduce any possible duplication in logic in create_new_client_event #14688

Closed

Merge branch 'develop' into shay/rework_event_context

f439b23

erikjohnston reviewed Feb 7, 2023

View reviewed changes

synapse/events/snapshot.py Outdated Show resolved Hide resolved

synapse/events/snapshot.py Show resolved Hide resolved

synapse/events/snapshot.py Outdated Show resolved Hide resolved

H-Shay commented Feb 7, 2023

View reviewed changes

synapse/state/__init__.py Outdated Show resolved Hide resolved

H-Shay added 2 commits February 8, 2023 10:21

clearer renaming and flow

3614ed0

Merge branch 'shay/rework_event_context' of https://github.com/matrix…

a0bcc0a

…-org/synapse into shay/rework_event_context

H-Shay requested a review from erikjohnston February 8, 2023 19:07

make partial state non-optional

427d007

erikjohnston approved these changes Feb 9, 2023

View reviewed changes

synapse/events/snapshot.py Outdated Show resolved Hide resolved

update docstrings

c3bdb6e

H-Shay merged commit 03bccd5 into develop Feb 9, 2023

H-Shay deleted the shay/rework_event_context branch February 9, 2023 21:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a class UnpersistedEventContext to allow for the batching up of storing state groups #14675

Add a class UnpersistedEventContext to allow for the batching up of storing state groups #14675

H-Shay commented Dec 13, 2022 •

edited

Loading

DMRobertson left a comment

erikjohnston left a comment

H-Shay commented Jan 26, 2023 •

edited

Loading

erikjohnston left a comment

H-Shay commented Feb 8, 2023

erikjohnston left a comment

Add a class UnpersistedEventContext to allow for the batching up of storing state groups #14675

Add a class UnpersistedEventContext to allow for the batching up of storing state groups #14675

Conversation

H-Shay commented Dec 13, 2022 • edited Loading

DMRobertson left a comment

Choose a reason for hiding this comment

erikjohnston left a comment

Choose a reason for hiding this comment

H-Shay commented Jan 26, 2023 • edited Loading

erikjohnston left a comment

Choose a reason for hiding this comment

H-Shay commented Feb 8, 2023

erikjohnston left a comment

Choose a reason for hiding this comment

H-Shay commented Dec 13, 2022 •

edited

Loading

H-Shay commented Jan 26, 2023 •

edited

Loading