Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Backfilled events marked as outliers break backfill #8894

Closed
anoadragon453 opened this issue Dec 8, 2020 · 4 comments · Fixed by #11632
Closed

Backfilled events marked as outliers break backfill #8894

anoadragon453 opened this issue Dec 8, 2020 · 4 comments · Fixed by #11632
Labels
S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.

Comments

@anoadragon453
Copy link
Member

A homeserver backfilling from another homeserver was found to be raising an AssertionError at:

synapse/synapse/handlers/federation.py

Lines 921 to 923 in 30fba62

# For paranoia we ensure that these events are marked as
# non-outliers
assert not event.internal_metadata.is_outlier()

It's currently unclear which event is problematic and why it might be marked as an outlier. The logs of the backfilling server also contained many instances of event signature validation failures. This was due to a remote server changing their signing key without also changing the key ID (or putting the old key in old_signing_keys). This may be a red herring however.

The assert and a similar assert at:

synapse/synapse/handlers/federation.py

Lines 891 to 894 in 30fba62

# For paranoia we ensure that these events are marked as
# non-outliers
ev = event_map[e_id]
assert not ev.internal_metadata.is_outlier()

both do not give any information about the failing event.

Thus the first step is to improve these assert statements and figure out why these backfilled events are marked as outliers. After that, we may want to make the backfill code drop these events instead of simple raising, similar to what was done in #8350.
@anoadragon453 anoadragon453 added the z-bug (Deprecated Label) label Dec 8, 2020
@callahad
Copy link
Contributor

@anoadragon453 Any chance you have a stack trace for this?

@anoadragon453
Copy link
Member Author

The stack trace is quite unhelpful:

AssertionError: null
  File "synapse/handlers/federation.py", line 1097, in try_backfill
    await self.backfill(
  File "synapse/handlers/federation.py", line 920, in backfill
    assert not event.internal_metadata.is_outlier()

However you can see a little bit of surrounding logs and other metadata at https://sentry.matrix.org/sentry/synapse-modular/issues/194313/ and https://sentry.matrix.org/sentry/synapse-modular/issues/194314/ (for those that have access).

@callahad
Copy link
Contributor

callahad commented Jan 6, 2021

Deferring for now, as one of our quarterly goals is to tame our Sentry alert rate, we should return to this in due course.

@richvdh richvdh added S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. and removed z-bug (Deprecated Label) z-info-needed labels Apr 9, 2021
@richvdh
Copy link
Member

richvdh commented Dec 22, 2021

Looks like this was introduced in #8350.

@richvdh richvdh mentioned this issue Dec 22, 2021
Merged
richvdh added a commit that referenced this issue Dec 22, 2021
@richvdh
FederationClient.backfill: stop flagging events as outliers
7bc58b9 
Events returned by `backfill` should not be flagged as outliers.

Fixes:

```
AssertionError: null
  File "synapse/handlers/federation.py", line 313, in try_backfill
    dom, room_id, limit=100, extremities=extremities
  File "synapse/handlers/federation_event.py", line 517, in backfill
    await self._process_pulled_events(dest, events, backfilled=True)
  File "synapse/handlers/federation_event.py", line 642, in _process_pulled_events
    await self._process_pulled_event(origin, ev, backfilled=backfilled)
  File "synapse/handlers/federation_event.py", line 669, in _process_pulled_event
    assert not event.internal_metadata.is_outlier()
```

See https://sentry.matrix.org/sentry/synapse-matrixorg/issues/231992

Fixes #8894.
richvdh added a commit that referenced this issue Jan 4, 2022
@richvdh
FederationClient.backfill: stop flagging events as outliers (#11632)
878aa55 
Events returned by `backfill` should not be flagged as outliers.

Fixes:

```
AssertionError: null
  File "synapse/handlers/federation.py", line 313, in try_backfill
    dom, room_id, limit=100, extremities=extremities
  File "synapse/handlers/federation_event.py", line 517, in backfill
    await self._process_pulled_events(dest, events, backfilled=True)
  File "synapse/handlers/federation_event.py", line 642, in _process_pulled_events
    await self._process_pulled_event(origin, ev, backfilled=backfilled)
  File "synapse/handlers/federation_event.py", line 669, in _process_pulled_event
    assert not event.internal_metadata.is_outlier()
```

See https://sentry.matrix.org/sentry/synapse-matrixorg/issues/231992

Fixes #8894.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

FederationClient.backfill: stop flagging events as outliers matrix-org/synapse
3 participants
@callahad @anoadragon453 @richvdh