-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Add an admin api to delete local media. #8519
Changes from 4 commits
8e63746
1dc57c7
d8167de
e853bc5
be9ee45
542a11e
1cb5c44
3b9a691
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
Add an admin api `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete a single file from server. Contributed by @dklimpel. | ||
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -100,3 +100,82 @@ Response: | |
"num_quarantined": 10 # The number of media items successfully quarantined | ||
} | ||
``` | ||
|
||
# Delete local media | ||
This API deletes the *local* media from the disk of your own server. | ||
This includes any local thumbnails and copies of media downloaded from | ||
remote homeservers. | ||
This API will not affect media that has been uploaded to external | ||
media repositories (e.g https://github.com/turt2live/matrix-media-repo/). | ||
See also [purge_remote_media.rst](purge_remote_media.rst). | ||
|
||
## Delete a specific local media | ||
Delete a specific ``media_id``. | ||
|
||
anoadragon453 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Request: | ||
|
||
``` | ||
DELETE /_synapse/admin/v1/media/<server_name>/<media_id> | ||
|
||
{} | ||
``` | ||
|
||
URL Parameters | ||
|
||
* ``server_name``: string - The name of your local server (e.g ``matrix.org``) | ||
* ``media_id``: string - The ID of the media (e.g ``abcdefghijklmnopqrstuvwx``) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Although it renders fine, we typically use single backticks in markdown files. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have done an update. |
||
|
||
Response: | ||
|
||
```json | ||
{ | ||
"deleted_media":[ | ||
dklimpel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"abcdefghijklmnopqrstuvwx" | ||
], | ||
"total": 1 | ||
} | ||
``` | ||
|
||
The following fields are returned in the JSON response body: | ||
|
||
* ``deleted_media``: list of strings - List of deleted ``media_id`` | ||
dklimpel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* ``total``: integer - Total number of deleted ``media_id`` | ||
|
||
## Delete local media by date or size | ||
|
||
Request: | ||
|
||
``` | ||
POST /_synapse/admin/v1/media/<server_name>/delete?before_ts=<before_ts> | ||
|
||
{} | ||
``` | ||
|
||
URL Parameters | ||
|
||
* ``server_name``: string - The name of your local server (e.g ``matrix.org``) | ||
dklimpel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* ``before_ts``: string representing a positive integer - Unix timestamp in ms. | ||
Files that were last used before this timestamp will be deleted. It is the timestamp of | ||
last access and not the timestamp creation. | ||
* ``size_gt``: Optional - string representing a positive integer - Size of the media in bytes. | ||
Files that are larger will be deleted. Defaults to ``0``. | ||
* ``keep_profiles``: Optional- string representing a boolean - Switch to delete also files | ||
dklimpel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
that are still used in image data (e.g user profile, room avatar). | ||
If ``false`` thse files will be deleted. Defaults to ``true``. | ||
dklimpel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Response: | ||
|
||
```json | ||
{ | ||
"deleted_media":[ | ||
dklimpel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"abcdefghijklmnopqrstuvwx", | ||
"abcdefghijklmnopqrstuvwz" | ||
], | ||
"total": 2 | ||
} | ||
``` | ||
|
||
The following fields are returned in the JSON response body: | ||
|
||
* ``deleted_media``: list of strings - List of deleted ``media_id`` | ||
dklimpel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* ``total``: integer - Total number of deleted ``media_id`` |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -18,7 +18,7 @@ | |
import logging | ||
import os | ||
import shutil | ||
from typing import IO, Dict, Optional, Tuple | ||
from typing import IO, Dict, List, Optional, Tuple | ||
|
||
import twisted.internet.error | ||
import twisted.web.http | ||
|
@@ -767,6 +767,80 @@ async def delete_old_remote_media(self, before_ts): | |
|
||
return {"deleted": deleted} | ||
|
||
async def delete_local_media(self, media_id: str) -> Tuple[List[str], int]: | ||
""" | ||
Delete the given media_id from this server | ||
dklimpel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Args: | ||
media_id: The media ID to delete. | ||
Returns: | ||
List of deleted media_id | ||
Number of deleted media_id | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd probably word this something more like "A tuple of (list of deleted media IDs, total deleted media IDs)." Likewise below. |
||
""" | ||
logger.info("Deleting local media: %s", media_id) | ||
|
||
return await self._remove_local_media_from_disk([media_id]) | ||
|
||
async def delete_old_local_media( | ||
self, before_ts: int, size_gt: int = 0, keep_profiles: bool = True, | ||
) -> Tuple[List[str], int]: | ||
""" | ||
Delete old media_id from this server | ||
dklimpel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Args: | ||
before_ts: Unix timestamp in ms. | ||
Files that were last used before this timestamp will be deleted | ||
size_gt: Size of the media in bytes. Files that are larger will be deleted | ||
keep_profiles: Switch to delete also files that are still used in image data | ||
(e.g user profile, room avatar) | ||
If false thse files will be deleted | ||
dklimpel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Returns: | ||
List of deleted media_id | ||
Number of deleted media_id | ||
""" | ||
old_media = await self.store.get_local_media_before( | ||
before_ts, size_gt, keep_profiles, | ||
) | ||
logger.info("Deleting local media: %s", old_media) | ||
return await self._remove_local_media_from_disk(old_media) | ||
|
||
async def _remove_local_media_from_disk( | ||
self, media_ids: List[str] | ||
) -> Tuple[List[str], int]: | ||
""" | ||
Delete old media_id from this server | ||
dklimpel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Args: | ||
media_ids: List of media_id to delete | ||
Returns: | ||
List of deleted media_id | ||
Number of deleted media_id | ||
""" | ||
removed_media = [] | ||
for media_id in media_ids: | ||
logger.info("Deleting: %s", media_id) | ||
anoadragon453 marked this conversation as resolved.
Show resolved
Hide resolved
anoadragon453 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
full_path = self.filepaths.local_media_filepath(media_id) | ||
try: | ||
os.remove(full_path) | ||
except OSError as e: | ||
logger.warning("Failed to remove file: %r: %s", full_path, e) | ||
if e.errno == errno.ENOENT: | ||
pass | ||
else: | ||
continue | ||
|
||
thumbnail_dir = self.filepaths.local_media_thumbnail_dir(media_id) | ||
shutil.rmtree(thumbnail_dir, ignore_errors=True) | ||
|
||
await self.store.delete_remote_media(self.server_name, media_id) | ||
|
||
await self.store.delete_url_cache((media_id,)) | ||
await self.store.delete_url_cache_media((media_id,)) | ||
|
||
removed_media.append(media_id) | ||
|
||
return removed_media, len(removed_media) | ||
|
||
|
||
class MediaRepositoryResource(Resource): | ||
"""File uploading and downloading. | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -93,6 +93,7 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore): | |
|
||
def __init__(self, database: DatabasePool, db_conn, hs): | ||
super().__init__(database, db_conn, hs) | ||
self.server_name = hs.hostname | ||
|
||
async def get_local_media(self, media_id: str) -> Optional[Dict[str, Any]]: | ||
"""Get the metadata for a local piece of media | ||
|
@@ -115,6 +116,53 @@ async def get_local_media(self, media_id: str) -> Optional[Dict[str, Any]]: | |
desc="get_local_media", | ||
) | ||
|
||
async def get_local_media_before( | ||
self, before_ts: int, size_gt: int, keep_profiles: bool, | ||
) -> Optional[List[str]]: | ||
|
||
sql = """ | ||
SELECT media_id | ||
FROM local_media_repository AS lmr | ||
WHERE last_access_ts < ? and media_length > ? | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What about media that is never accessed? As far as I can tell this will have a |
||
""" | ||
|
||
if keep_profiles: | ||
sql_keep = """ | ||
AND ( | ||
NOT EXISTS | ||
(SELECT 1 | ||
FROM profiles | ||
WHERE profiles.avatar_url = '{media_prefix}' || lmr.media_id) | ||
AND NOT EXISTS | ||
(SELECT 1 | ||
FROM groups | ||
WHERE groups.avatar_url = '{media_prefix}' || lmr.media_id) | ||
AND NOT EXISTS | ||
(SELECT 1 | ||
FROM room_memberships | ||
WHERE room_memberships.avatar_url = '{media_prefix}' || lmr.media_id) | ||
AND NOT EXISTS | ||
(SELECT 1 | ||
FROM user_directory | ||
WHERE user_directory.avatar_url = '{media_prefix}' || lmr.media_id) | ||
AND NOT EXISTS | ||
(SELECT 1 | ||
FROM room_stats_state | ||
WHERE room_stats_state.avatar = '{media_prefix}' || lmr.media_id) | ||
) | ||
""".format( | ||
media_prefix="mxc://%s/" % (self.server_name,), | ||
) | ||
sql += sql_keep | ||
|
||
def _get_local_media_before_txn(txn): | ||
txn.execute(sql, (before_ts, size_gt)) | ||
return [row[0] for row in txn] | ||
|
||
return await self.db_pool.runInteraction( | ||
"get_local_media_before", _get_local_media_before_txn | ||
) | ||
|
||
async def store_local_media( | ||
self, | ||
media_id, | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs a small update.