Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Server DoSed by incoming 'groups' federation requests #5933

Closed
MurzNN opened this issue Aug 29, 2019 · 6 comments
Closed

Server DoSed by incoming 'groups' federation requests #5933

MurzNN opened this issue Aug 29, 2019 · 6 comments
Labels
A-Performance Performance, both client-facing and admin-facing T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.

Comments

@MurzNN
Copy link

MurzNN commented Aug 29, 2019 •
edited
Loading

My ru-matrix.org homeserver very often do rate limiting to incoming federation requests from matrix.org server, I see a lot of SynapseError: 429 - Too Many Requests in logs.

Even if I increase limits to:

federation_rc_sleep_limit: 300 # from 10
federation_rc_reject_limit: 500 # from 50
federation_rc_sleep_delay: 100 # from 500
  • this isn't help.

But when I grep logs, I see that it contains very much groups federation requests every second, here is example of one minute range grep from logs (143 queries per minute!):

2019-08-30 00:54:00,016 - synapse.access.http.8008 - 302 - INFO - POST-97721- 94.237.60.124 - 8008 - {matrix.org} Processed request: 19.332sec/0.218sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2Bp4hz9dqgx3oy4nzzzgyogjrc39v5zw53zzb9oen6%3Aru-matrix.org/renew_attestation/%40cindy-lou-who%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,019 - synapse.access.http.8008 - 302 - INFO - POST-97724- 94.237.60.124 - 8008 - {matrix.org} Processed request: 19.488sec/0.066sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2Bru%3Amatrix.org/renew_attestation/%40blambo_123%3Aru-matrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,020 - synapse.http.server - 79 - INFO - POST-97725- <XForwardedForRequest at 0x7f3e626503c8 method='POST' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40hypersonic%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,091 - synapse.access.http.8008 - 302 - INFO - POST-97723- 94.237.60.124 - 8008 - {matrix.org} Processed request: 19.414sec/0.213sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40valium%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,107 - synapse.http.server - 79 - INFO - POST-97726- <XForwardedForRequest at 0x7f3e62650588 method='POST' uri='/_matrix/federation/v1/groups/%2Bp4hz9dqgx3oy4nzzzgyogjrc39v5zw53zzb9oen6%3Aru-matrix.org/renew_attestation/%40tylertyler%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,359 - synapse.access.http.8008 - 302 - INFO - POST-97725- 94.237.60.124 - 8008 - {matrix.org} Processed request: 19.555sec/0.337sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40hypersonic%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,362 - synapse.access.http.8008 - 302 - INFO - POST-97726- 94.237.60.124 - 8008 - {matrix.org} Processed request: 19.641sec/0.255sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2Bp4hz9dqgx3oy4nzzzgyogjrc39v5zw53zzb9oen6%3Aru-matrix.org/renew_attestation/%40tylertyler%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,364 - synapse.http.server - 79 - INFO - GET-97729- <XForwardedForRequest at 0x7f3f2b106828 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,404 - synapse.access.http.8008 - 302 - INFO - GET-97729- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.956sec/0.038sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,415 - synapse.http.server - 79 - INFO - POST-97730- <XForwardedForRequest at 0x7f3dc57e42e8 method='POST' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40the-comfy-couch%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,488 - synapse.access.http.8008 - 302 - INFO - POST-97730- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.005sec/0.072sec (0.004sec, 0.001sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40the-comfy-couch%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,495 - synapse.http.server - 79 - INFO - POST-97731- <XForwardedForRequest at 0x7f3dc57e4d68 method='POST' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40ysengrimus%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,624 - synapse.access.http.8008 - 302 - INFO - POST-97731- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.082sec/0.129sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40ysengrimus%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,631 - synapse.http.server - 79 - INFO - POST-97734- <XForwardedForRequest at 0x7f3dd1e430f0 method='POST' uri='/_matrix/federation/v1/groups/%2Bp4hz9dqgx3oy4nzzzgyogjrc39v5zw53zzb9oen6%3Aru-matrix.org/renew_attestation/%40alexandrepickles%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,663 - synapse.access.http.8008 - 302 - INFO - POST-97734- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.160sec/0.032sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2Bp4hz9dqgx3oy4nzzzgyogjrc39v5zw53zzb9oen6%3Aru-matrix.org/renew_attestation/%40alexandrepickles%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,677 - synapse.http.server - 79 - INFO - POST-97735- <XForwardedForRequest at 0x7f3dd1e439b0 method='POST' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40marcensen%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,723 - synapse.access.http.8008 - 302 - INFO - POST-97735- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.205sec/0.046sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40marcensen%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,725 - synapse.http.server - 79 - INFO - POST-97736- <XForwardedForRequest at 0x7f3e37dab0f0 method='POST' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40ladyviper%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,752 - synapse.access.http.8008 - 302 - INFO - POST-97736- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.250sec/0.027sec (0.002sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40ladyviper%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,756 - synapse.http.server - 79 - INFO - POST-97737- <XForwardedForRequest at 0x7f3dc6b53080 method='POST' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40floofox%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,777 - synapse.access.http.8008 - 302 - INFO - POST-97737- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.287sec/0.014sec (0.006sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40floofox%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,788 - synapse.http.server - 79 - INFO - GET-97738- <XForwardedForRequest at 0x7f3e39d70358 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,806 - synapse.access.http.8008 - 302 - INFO - GET-97738- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.311sec/0.017sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,810 - synapse.http.server - 79 - INFO - POST-97739- <XForwardedForRequest at 0x7f3f2cbb6e10 method='POST' uri='/_matrix/federation/v1/groups/%2Bp4hz9dqgx3oy4nzzzgyogjrc39v5zw53zzb9oen6%3Aru-matrix.org/renew_attestation/%40kyosai7%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,821 - synapse.access.http.8008 - 302 - INFO - POST-97739- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.333sec/0.010sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2Bp4hz9dqgx3oy4nzzzgyogjrc39v5zw53zzb9oen6%3Aru-matrix.org/renew_attestation/%40kyosai7%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,827 - synapse.http.server - 79 - INFO - POST-97740- <XForwardedForRequest at 0x7f3e32f80518 method='POST' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40princessofthevoid%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,835 - synapse.access.http.8008 - 302 - INFO - POST-97740- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.349sec/0.008sec (0.015sec, 0.004sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40princessofthevoid%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,840 - synapse.http.server - 79 - INFO - POST-97741- <XForwardedForRequest at 0x7f3f3b422048 method='POST' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40d44%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,856 - synapse.access.http.8008 - 302 - INFO - POST-97741- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.366sec/0.010sec (0.005sec, 0.001sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40d44%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,862 - synapse.http.server - 79 - INFO - POST-97755- <XForwardedForRequest at 0x7f3ebaf79908 method='POST' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40dawn118%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,871 - synapse.access.http.8008 - 302 - INFO - POST-97755- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.339sec/0.007sec (0.007sec, 0.001sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40dawn118%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,885 - synapse.http.server - 79 - INFO - POST-97756- <XForwardedForRequest at 0x7f3dc0eb7b70 method='POST' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40cindy-lou-who-question-mark%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,901 - synapse.access.http.8008 - 302 - INFO - POST-97756- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.357sec/0.016sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40cindy-lou-who-question-mark%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,919 - synapse.http.server - 79 - INFO - POST-97757- <XForwardedForRequest at 0x7f3dc0eb7908 method='POST' uri='/_matrix/federation/v1/groups/%2Bp4hz9dqgx3oy4nzzzgyogjrc39v5zw53zzb9oen6%3Aru-matrix.org/renew_attestation/%401000ero%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,924 - synapse.access.http.8008 - 302 - INFO - POST-97757- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.390sec/0.004sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2Bp4hz9dqgx3oy4nzzzgyogjrc39v5zw53zzb9oen6%3Aru-matrix.org/renew_attestation/%401000ero%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,938 - synapse.http.server - 79 - INFO - POST-97758- <XForwardedForRequest at 0x7f3d5b8085f8 method='POST' uri='/_matrix/federation/v1/groups/%2Bp4hz9dqgx3oy4nzzzgyogjrc39v5zw53zzb9oen6%3Aru-matrix.org/renew_attestation/%40adrianmara%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,954 - synapse.access.http.8008 - 302 - INFO - POST-97758- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.409sec/0.015sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2Bp4hz9dqgx3oy4nzzzgyogjrc39v5zw53zzb9oen6%3Aru-matrix.org/renew_attestation/%40adrianmara%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:00,966 - synapse.http.server - 79 - INFO - POST-97759- <XForwardedForRequest at 0x7f3d5b808400 method='POST' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40flubberyowl%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:00,982 - synapse.access.http.8008 - 302 - INFO - POST-97759- 94.237.60.124 - 8008 - {matrix.org} Processed request: 1.435sec/0.014sec (0.006sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "POST /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/renew_attestation/%40flubberyowl%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:02,335 - synapse.http.server - 79 - INFO - GET-97827- <XForwardedForRequest at 0x7f3deb7670f0 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:02,457 - synapse.access.http.8008 - 302 - INFO - GET-97827- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.005sec/0.116sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:02,982 - synapse.http.server - 79 - INFO - GET-97846- <XForwardedForRequest at 0x7f3f3b9df438 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:02,983 - synapse.access.http.8008 - 302 - INFO - GET-97846- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.001sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:04,265 - synapse.http.server - 79 - INFO - GET-97855- <XForwardedForRequest at 0x7f3dc1371780 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:04,266 - synapse.access.http.8008 - 302 - INFO - GET-97855- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.002sec/0.000sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:06,762 - synapse.http.server - 79 - INFO - GET-97881- <XForwardedForRequest at 0x7f3e06bac588 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:06,763 - synapse.access.http.8008 - 302 - INFO - GET-97881- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.002sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:07,302 - synapse.http.server - 79 - INFO - GET-97888- <XForwardedForRequest at 0x7f3dc548f908 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:07,304 - synapse.access.http.8008 - 302 - INFO - GET-97888- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.004sec/0.001sec (0.002sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:07,740 - synapse.http.server - 79 - INFO - GET-97894- <XForwardedForRequest at 0x7f3ec6aa1b00 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:07,742 - synapse.access.http.8008 - 302 - INFO - GET-97894- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.006sec/0.001sec (0.002sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:09,134 - synapse.http.server - 79 - INFO - GET-97905- <XForwardedForRequest at 0x7f3dc4e0ef28 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:09,139 - synapse.access.http.8008 - 302 - INFO - GET-97905- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.025sec/0.001sec (0.003sec, 0.000sec) (0.006sec/0.010sec/1) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:11,828 - synapse.http.server - 79 - INFO - GET-97932- <XForwardedForRequest at 0x7f3f0a2b80f0 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:11,830 - synapse.access.http.8008 - 302 - INFO - GET-97932- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.005sec/0.001sec (0.002sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40ganymede3301777%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:17,466 - synapse.http.server - 79 - INFO - GET-97966- <XForwardedForRequest at 0x7f3de0448588 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:17,474 - synapse.access.http.8008 - 302 - INFO - GET-97966- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.003sec/0.007sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:18,245 - synapse.http.server - 79 - INFO - GET-97973- <XForwardedForRequest at 0x7f3e07afec18 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:18,247 - synapse.access.http.8008 - 302 - INFO - GET-97973- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.002sec/0.002sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:19,130 - synapse.http.server - 79 - INFO - GET-97984- <XForwardedForRequest at 0x7f3de0448ef0 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:19,131 - synapse.access.http.8008 - 302 - INFO - GET-97984- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.002sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:23,052 - synapse.http.server - 79 - INFO - GET-98012- <XForwardedForRequest at 0x7f3f2b97b3c8 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:23,066 - synapse.access.http.8008 - 302 - INFO - GET-98012- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.002sec/0.014sec (0.002sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:29,775 - synapse.http.server - 79 - INFO - GET-98024- <XForwardedForRequest at 0x7f3dd331a390 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:29,780 - synapse.http.server - 79 - INFO - GET-98026- <XForwardedForRequest at 0x7f3f14a97160 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40jean_luc_picard%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:29,785 - synapse.access.http.8008 - 302 - INFO - GET-98024- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.006sec/0.009sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:29,786 - synapse.access.http.8008 - 302 - INFO - GET-98026- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.007sec/0.002sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40jean_luc_picard%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:29,998 - synapse.http.server - 79 - INFO - GET-98028- <XForwardedForRequest at 0x7f3e55caa320 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40prodigaljohn%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:30,006 - synapse.access.http.8008 - 302 - INFO - GET-98028- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.004sec/0.007sec (0.001sec, 0.002sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40prodigaljohn%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:30,281 - synapse.http.server - 79 - INFO - GET-98030- <XForwardedForRequest at 0x7f3f31583748 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:30,286 - synapse.access.http.8008 - 302 - INFO - GET-98030- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.003sec/0.004sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:30,442 - synapse.http.server - 79 - INFO - GET-98031- <XForwardedForRequest at 0x7f3f3a5e49b0 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40jean_luc_picard%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:30,444 - synapse.access.http.8008 - 302 - INFO - GET-98031- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.003sec/0.002sec (0.002sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40jean_luc_picard%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]
2019-08-30 00:54:30,488 - synapse.http.server - 79 - INFO - GET-98032- <XForwardedForRequest at 0x7f3ddf381b00 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40prodigaljohn%3Amatrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 429 - Too Many Requests
2019-08-30 00:54:30,488 - synapse.access.http.8008 - 302 - INFO - GET-98032- 94.237.60.124 - 8008 - {matrix.org} Processed request: 0.002sec/0.000sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 77B 429 "GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40prodigaljohn%3Amatrix.org HTTP/1.0" "Synapse/1.3.1 (b=matrix-org-hotfixes,a81e0233e)" [0 dbevts]

Why those federation requests sends too often, without reacting to 429 error and delaying? I think that this is bug in Synapse, that don't apply rate limiting to outgoing federation requests of groups category?
@MurzNN
Copy link
Author

MurzNN commented Aug 29, 2019 •
edited
Loading

And here is not rate-limited groups federation one-minute spam (44 queries) example in logs:

2019-08-30 01:20:08,452 - synapse.http.site - 203 - WARNING - GET-105996- Error processing request <XForwardedForRequest at 0x7f3e988f81d0 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionDone'> Connection was closed cleanly.
2019-08-30 01:20:08,543 - synapse.access.http.8008 - 233 - INFO - GET-106285- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:08,549 - synapse.access.http.8008 - 233 - INFO - GET-106287- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:15,334 - synapse.access.http.8008 - 233 - INFO - GET-106289- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:15,341 - synapse.http.site - 203 - WARNING - GET-106017- Error processing request <XForwardedForRequest at 0x7f3e41b57a90 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40simplymiles%3Amatrix.org' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionDone'> Connection was closed cleanly.
2019-08-30 01:20:15,921 - synapse.access.http.8008 - 233 - INFO - GET-106292- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:16,135 - synapse.access.http.8008 - 233 - INFO - GET-106301- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40simplymiles%3Amatrix.org
2019-08-30 01:20:17,415 - synapse.access.http.8008 - 233 - INFO - GET-106307- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:20,232 - synapse.access.http.8008 - 233 - INFO - GET-106319- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:20,374 - synapse.access.http.8008 - 233 - INFO - GET-106324- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:24,367 - synapse.access.http.8008 - 233 - INFO - GET-106339- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:24,973 - synapse.access.http.8008 - 233 - INFO - GET-106345- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:25,619 - synapse.http.site - 203 - WARNING - GET-106072- Error processing request <XForwardedForRequest at 0x7f3dc0623940 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionDone'> Connection was closed cleanly.
2019-08-30 01:20:25,789 - synapse.http.site - 203 - WARNING - GET-106074- Error processing request <XForwardedForRequest at 0x7f3ddf7041d0 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40prodigaljohn%3Amatrix.org' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionDone'> Connection was closed cleanly.
2019-08-30 01:20:26,106 - synapse.access.http.8008 - 233 - INFO - GET-106348- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:26,605 - synapse.access.http.8008 - 233 - INFO - GET-106350- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org
2019-08-30 01:20:27,152 - synapse.access.http.8008 - 233 - INFO - GET-106352- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40prodigaljohn%3Amatrix.org
2019-08-30 01:20:29,389 - synapse.http.site - 203 - WARNING - GET-106102- Error processing request <XForwardedForRequest at 0x7f3c82fe59e8 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionDone'> Connection was closed cleanly.
2019-08-30 01:20:29,390 - synapse.http.site - 203 - WARNING - GET-106103- Error processing request <XForwardedForRequest at 0x7f3f2cf794a8 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionDone'> Connection was closed cleanly.
2019-08-30 01:20:29,456 - synapse.access.http.8008 - 233 - INFO - GET-106355- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40kyosai7%3Amatrix.org
2019-08-30 01:20:29,464 - synapse.access.http.8008 - 233 - INFO - GET-106358- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:29,464 - synapse.access.http.8008 - 233 - INFO - GET-106359- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%40prodigaljohn%3Amatrix.org
2019-08-30 01:20:30,216 - synapse.access.http.8008 - 233 - INFO - GET-106362- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:30,966 - synapse.access.http.8008 - 233 - INFO - GET-106367- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:31,024 - synapse.access.http.8008 - 233 - INFO - GET-106368- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:31,648 - synapse.http.site - 203 - WARNING - GET-106108- Error processing request <XForwardedForRequest at 0x7f3c1e70e860 method='GET' uri='/_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionDone'> Connection was closed cleanly.
2019-08-30 01:20:32,063 - synapse.access.http.8008 - 233 - INFO - GET-106373- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:32,337 - synapse.access.http.8008 - 233 - INFO - GET-106374- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:33,278 - synapse.access.http.8008 - 233 - INFO - GET-106378- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:33,742 - synapse.access.http.8008 - 233 - INFO - GET-106380- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:36,888 - synapse.access.http.8008 - 233 - INFO - GET-106384- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:37,408 - synapse.access.http.8008 - 233 - INFO - GET-106387- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:37,411 - synapse.access.http.8008 - 233 - INFO - GET-106389- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:38,690 - synapse.access.http.8008 - 233 - INFO - GET-106393- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:41,133 - synapse.access.http.8008 - 233 - INFO - GET-106398- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:41,356 - synapse.access.http.8008 - 233 - INFO - GET-106400- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:42,034 - synapse.access.http.8008 - 233 - INFO - GET-106404- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:42,169 - synapse.access.http.8008 - 233 - INFO - GET-106406- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:45,099 - synapse.access.http.8008 - 233 - INFO - GET-106409- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:45,101 - synapse.access.http.8008 - 233 - INFO - GET-106410- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:46,005 - synapse.access.http.8008 - 233 - INFO - GET-106416- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:49,293 - synapse.access.http.8008 - 233 - INFO - GET-106421- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:49,304 - synapse.access.http.8008 - 233 - INFO - GET-106422- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:49,991 - synapse.access.http.8008 - 233 - INFO - GET-106430- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:52,468 - synapse.access.http.8008 - 233 - INFO - GET-106432- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:53,779 - synapse.access.http.8008 - 233 - INFO - GET-106451- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org
2019-08-30 01:20:56,540 - synapse.access.http.8008 - 233 - INFO - GET-106463- 94.237.60.124 - 8008 - Received request: GET /_matrix/federation/v1/groups/%2B4d26pq2okn2c2bz747ew3mgu5owftg%3Aru-matrix.org/profile?requester_user_id=%401000ero%3Amatrix.org

and more than 1600 same groups queries in last 2 hours.

@richvdh
Copy link
Member

richvdh commented Aug 30, 2019

yes, looks like the groups code does not handle 429s (or failures of any kind) gracefully.

And here is not rate-limited groups federation one-minute spam (44 queries) example in logs:

Looking at these from the matrix.org end, all of them seem to end with a timeout or a 429.

I can't really suggest much except to stop using groups until the functionality is more mature.

Related #2654

@richvdh richvdh changed the title Too much 'groups' federation requests from matrix.org's Synapse Server DoSed by incoming 'groups' federation requests Aug 30, 2019
@richvdh richvdh mentioned this issue Aug 30, 2019
Closed
@MurzNN
Copy link
Author

MurzNN commented Aug 30, 2019 •
edited
Loading

@richvdh Thanks for describing!

I can't really suggest much except to stop using groups until the functionality is more mature.

Can you describe easier way to force stop using groups on homeserver, especially for not broke federation? Manually clean-up groups_* pgsql tables and restarting Synapse?

Or maybe simply block /_matrix/federation/v1/groups/* urls on nginx proxy side? What error code best to answer - 404?

@MurzNN
Copy link
Author

MurzNN commented Aug 31, 2019

So, I blocked groups queries in nginx proxy via this rule:

        location ~ ^/_matrix/federation/v1/groups {
                return 404;
        }

@MadLittleMods MadLittleMods added z-communities A-Performance Performance, both client-facing and admin-facing labels Jul 8, 2021
@erikjohnston erikjohnston added the T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. label Jul 23, 2021
@ghost

This comment has been minimized.

Sign in to view
@clokep
Copy link
Member

clokep commented Apr 26, 2022

Communities/groups are being removed, see #11584.

@clokep clokep closed this as completed Apr 26, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
A-Performance Performance, both client-facing and admin-facing T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@MurzNN @clokep @MadLittleMods @richvdh @erikjohnston