/_synapse/admin/v1/delete_group admin api fails with 500 error

[olmari]

Description

Trying to remove user from group sometimes fails, adjacently https://github.com/matrix-org/synapse/blob/master/docs/admin_api/delete_group.md fails too because of same rootcause. Troublesome users seems to be existing on matrix.org server rather than our own hackab.fi one.
ADD: seems at least one other user in one group is from jkl.hacklab.fi so this might be generally federating the removal issue)

Groups that had members only from hacklab.fi got removed just fine with the api.

Steps to reproduce

• try removing user from matrix.org HS from community on own HS
  and/or
• try using admin api delete group which fails on removing such user and fails itself

curl -w "\n\nResponse code: %{response_code}\n\n" -s \
-X POST -H "Authorization: Bearer <New_format_is_nicely_short>" \
-H "Content-Type: application/json" \
'http://localhost:8008/_synapse/admin/v1/delete_group/+community:hacklab.fi'

{"errcode":"M_UNKNOWN","error":"Internal server error"}

Response code: 500

2021-05-18 17:41:04,696 - synapse.http.matrixfederationclient - 618 - WARNING - POST-793486 - {POST-O-569346} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/federation/v1/groups/local/%2Bcommunity%3Ahacklab.fi/users/%40luminix%3Amatrix.org/remove: HttpResponseException('400: Bad Request')

Version information

• Homeserver: hacklab.fi

• Version: 1.34.0

• Install method:
  matrix.org provided debian repo w/ apt

[erikjohnston]

The reported error on matrix.org is Content not JSON. I think what is happening here is you're not providing a body and your server is blindly forwarding that to matrix.org. Does it work if you add -d '{}' to your curl?

[olmari]

Now I spotted these:

curl -w "\n\nResponse code: %{response_code}\n\n" -s \
-X POST \
-H "Authorization: Bearer <redacted>" \
-H "Content-Type: application/json" \
-d '{}' \
'http://localhost:8008/_synapse/admin/v1/delete_group/+community:hacklab.fi'

{"errcode":"M_UNKNOWN","error":"Internal server error"}

Response code: 500

2021-05-20 15:46:16,843 - synapse.http.server - 97 - ERROR - POST-8146200 - Failed handle request via 'DeleteGroupAdminRestServlet': <XForwardedForRequest at 0x7fa011143160 method='POST' uri='/_synapse/admin/v1/delete_group/+community:hacklab.fi' clientproto='HTTP/1.1' site='8008'>
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/http/server.py", line 258, in _async_render_wrapper
--
    raise exc
synapse.api.errors.HttpResponseException: 400: Bad Request
2021-05-20 15:46:16,843 - synapse.access.http.8008 - 408 - INFO - POST-8146200 - ::1 - 8008 - {@olmari:hacklab.fi} Processed request: 0.030sec/-0.000sec (0.010sec, 0.004sec) (0.007sec/0.040sec/12) 55B 500 "POST /_synapse/admin/v1/delete_group/+community:hacklab.fi HTTP/1.1" "curl/7.74.0" [0 dbevts]
2021-05-20 15:46:16,875 - synapse.http.matrixfederationclient - 539 - INFO - POST-8146200 - {POST-O-4105565} [matrix.org] Got response headers: 400 Bad Request
2021-05-20 15:46:16,875 - synapse.http.matrixfederationclient - 618 - WARNING - POST-8146200 - {POST-O-4105565} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/federation/v1/groups/local/%2Bcommunity%3Ahacklab.fi/users/%40digital_ta%3Amatrix.org/remove: HttpResponseException('400: Bad Request')
2021-05-20 15:46:16,878 - synapse.http.matrixfederationclient - 539 - INFO - POST-8146200 - {POST-O-4105569} [lyseo.edu.ouka.fi] Got response headers: 400 Bad Request
2021-05-20 15:46:16,878 - synapse.http.matrixfederationclient - 618 - WARNING - POST-8146200 - {POST-O-4105569} [lyseo.edu.ouka.fi] Request failed: POST matrix://lyseo.edu.ouka.fi/_matrix/federation/v1/groups/local/%2Bcommunity%3Ahacklab.fi/users/%40uumas%3Alyseo.edu.ouka.fi/remove: HttpResponseException('400: Bad Request')
2021-05-20 15:46:16,880 - synapse.http.matrixfederationclient - 539 - INFO - POST-8146200 - {POST-O-4105568} [matrix.org] Got response headers: 400 Bad Request
2021-05-20 15:46:16,880 - synapse.http.matrixfederationclient - 618 - WARNING - POST-8146200 - {POST-O-4105568} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/federation/v1/groups/local/%2Bcommunity%3Ahacklab.fi/users/%40semileppone%3Amatrix.org/remove: HttpResponseException('400: Bad Request')
2021-05-20 15:46:16,884 - synapse.http.matrixfederationclient - 539 - INFO - POST-8146200 - {POST-O-4105566} [matrix.org] Got response headers: 400 Bad Request
2021-05-20 15:46:16,884 - synapse.http.matrixfederationclient - 618 - WARNING - POST-8146200 - {POST-O-4105566} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/federation/v1/groups/local/%2Bcommunity%3Ahacklab.fi/users/%40trehacklab%3Amatrix.org/remove: HttpResponseException('400: Bad Request')
2021-05-20 15:46:16,887 - synapse.http.matrixfederationclient - 539 - INFO - POST-8146200 - {POST-O-4105567} [matrix.org] Got response headers: 400 Bad Request
2021-05-20 15:46:16,888 - synapse.http.matrixfederationclient - 618 - WARNING - POST-8146200 - {POST-O-4105567} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/federation/v1/groups/local/%2Bcommunity%3Ahacklab.fi/users/%40fixie%3Amatrix.org/remove: HttpResponseException('400: Bad Request')

[olmari]

admin api fails with 500 error

Like mentioned, trying to remove user from community through Element-desktop also fails, so it's not directly group deletion api that fails, but removing the user -part.. And then adjacently failing to remove community from existence after that (so, ultimately 2 separate issues relating together).

[dklimpel]

Related to #5144

[clokep]

Communities/groups are being removed, see #11584.