From 7508d524a5e3c6d2df3eabad5ec7d551a5edd4e6 Mon Sep 17 00:00:00 2001 From: Taku <45324516+Taaku18@users.noreply.github.com> Date: Tue, 10 Oct 2023 17:23:32 -0700 Subject: [PATCH 1/5] Update readme with new documentation links, python version, and removed obsolete installion steps --- README.md | 119 +++++++++++++----------------------------------------- 1 file changed, 28 insertions(+), 91 deletions(-) diff --git a/README.md b/README.md index 062bf6d8a4..319a9742cf 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ - Made with Python 3.8 + Made with Python 3.10 @@ -50,11 +50,13 @@ Modmail is similar to Reddit's Modmail, both in functionality and purpose. It se This bot is free for everyone and always will be. If you like this project and would like to show your appreciation, you can support us on **[Patreon](https://www.patreon.com/kyber)**, cool benefits included! +For up-to-date setup instructions, please visit our [**documentation**](https://docs.modmail.dev/installation) page. + ## How does it work? When a member sends a direct message to the bot, Modmail will create a channel or "thread" into a designated category. All further DM messages will automatically relay to that channel; any available staff can respond within the channel. -Our Logviewer will save the threads so you can view previous threads through their corresponding log link. Here is an [**example**](https://logs.modmail.dev/example). +Our Logviewer will save the threads so you can view previous threads through their corresponding log link. ~~Here is an [**example**](https://logs.modmail.dev/example)~~ (demo not available at the moment). ## Features @@ -67,7 +69,7 @@ Our Logviewer will save the threads so you can view previous threads through the * Minimum length for members to be in the guild before allowed to contact Modmail (`guild_age`). * **Advanced Logging Functionality:** - * When you close a thread, Modmail will generate a [log link](https://logs.modmail.dev/example) and post it to your log channel. + * When you close a thread, Modmail will generate a log link and post it to your log channel. * Native Discord dark-mode feel. * Markdown/formatting support. * Login via Discord to protect your logs ([premium Patreon feature](https://patreon.com/kyber)). @@ -84,88 +86,34 @@ This list is ever-growing thanks to active development and our exceptional contr ## Installation -Q: Where can I find the Modmail bot invite link? +There are a number of options for hosting your very own dedicated Modmail bot. + +Visit our [**documentation**](https://docs.modmail.dev/installation) page for detailed guidance on how to deploy your Modmail bot. -A: Unfortunately, due to how this bot functions, it cannot be invited. The lack of an invite link is to ensure an individuality to your server and grant you full control over your bot and data. Nonetheless, you can quickly obtain a free copy of Modmail for your server by following one of the methods listed below (roughly takes 15 minutes of your time). +### Patreon Hosting -There are a few options for hosting your very own dedicated Modmail bot. +If you don't want the trouble of renting and configuring your server to host Modmail, we got a solution for you! We offer hosting and maintenance of your own, private Modmail bot (including a Logviewer) through [**Patreon**](https://patreon.com/kyber). -1. Patreon hosting -2. Local hosting (VPS, Dedicated Server, RPi, your computer, etc.) -3. PaaS (we provide a guide for Heroku) +## FAQ -### Patreon Hosting +**Q: Where can I find the Modmail bot invite link?** + +**A:** Unfortunately, due to how this bot functions, it cannot be invited. The lack of an invite link is to ensure an individuality to your server and grant you full control over your bot and data. Nonetheless, you can quickly obtain a free copy of Modmail for your server by following our [**documentation**](https://docs.modmail.dev/installation) steps or subscribe to [**Patreon**](https://patreon.com/kyber). + +**Q: Where can I find out more info about Modmail?** + +**A:** You can find more info about Modmail on our [**documentation**](https://docs.modmail.dev) page. If you run into any problems, join our [Modmail Discord Server](https://discord.gg/cnUpwrnpYb) for help and support. + +## Plugins + +Modmail supports the use of third-party plugins to extend or add functionalities to the bot. +Plugins allow niche features as well as anything else outside of the scope of the core functionality of Modmail. + +You can find a list of third-party plugins using the `?plugins registry` command or visit the [Unofficial List of Plugins](https://github.com/modmail-dev/modmail/wiki/Unofficial-List-of-Plugins) for a list of plugins contributed by the community. -If you don't want the trouble of renting and configuring your server to host Modmail, we got a solution for you! We offer hosting and maintenance of your own, private Modmail bot (including a Logviewer) through [**Patreon**](https://patreon.com/kyber). Join our [Modmail Discord Server](https://discord.gg/cnUpwrnpYb) for more info! - -### Local hosting (General Guide) - -Modmail can be hosted on any modern hardware, including your PC. For stability and reliability, we suggest purchasing a cloud server (VPS) for under $10/mo. If you need recommendations on choosing a VPS, join our [Discord server](https://discord.gg/cnUpwrnpYb), and we'll send you a list of non-affiliated hosting providers. Alternatively, we can host Modmail for you when you're subscribed to our [Patreon](https://patreon.com/kyber). - -This guide assumes you've downloaded [`Python 3.10`](https://www.python.org/downloads/release/python-376/) and added python and pip to PATH. - -1. Clone this repo - ```console - $ git clone https://github.com/modmail-dev/modmail - $ cd modmail - ``` -2. Create a Discord bot account, grant the necessary intents, and invite the bot ([guide](https://github.com/modmail-dev/modmail/wiki/Installation#2-discord-bot-account)) -3. Create a free MongoDB database ([guide](https://github.com/modmail-dev/modmail/wiki/Installation-(cont.)#3-create-a-database), follow it carefully!) -4. Rename the file `.env.example` to `.env` and fill it with appropriate values - - If you can't find `.env.example` because it's hidden, create a new text file named `.env`, then copy the contents of [this file](https://raw.githubusercontent.com/modmail-dev/modmail/master/.env.example) and replace the placeholders with their values - - If you're on Windows and cannot save the file as `.env`, save it as `.env.` instead (this only applies to Windows!) - - If you do not have a Logviewer yet, leave the `LOG_URL` field as-is -5. Update pip, install pipenv, and install dependencies using pipenv - ```console - $ pip install -U pip - $ pip install pipenv - $ pipenv install - ``` -6. Start the bot - ```console - $ pipenv run bot - ``` -7. Set up the Logviewer, see the [Logviewer installation guide](https://github.com/modmail-dev/logviewer) - -### Local Hosting (Docker) - -We provide support for Docker to simplify the deployment of Modmail and Logviewer. -We assume you already have Docker and Docker Compose Plugin installed, if not, see [here](https://docs.docker.com/get-docker/). - -1. Create a Discord bot account, grant the necessary intents, and invite the bot ([guide](https://github.com/modmail-dev/modmail/wiki/Installation#2-discord-bot-account)) -2. Create a file named `.env`, then copy the contents of [this file](https://raw.githubusercontent.com/modmail-dev/modmail/master/.env.example) and replace the placeholders with their values -3. Create a file named `docker-compose.yml`, then copy the contents of [this file](https://raw.githubusercontent.com/modmail-dev/modmail/master/docker-compose.yml), do not change anything! -4. Start the bot - ```console - $ docker compose up -d - ``` - - For older Docker versions, you may need to run `docker-compose up -d` instead -5. View the status of your bot, using `docker ps` and `docker logs [container-id]` - -Our Docker images are hosted on [GitHub Container Registry](ghcr.io), you can build your own image if you wish: -```console -$ docker build --tag=modmail:master . -``` - -Then simply remove `ghcr.io/modmail-dev/` from the `docker-compose.yml` file. - -### Local Hosting (OS-Specific) - -This guide is a WIP. Join our [Discord server](https://discord.gg/cnUpwrnpYb) for more info. - -### Platform as a Service (PaaS) - -You can host this bot on Heroku (no longer free). - -Installation via Heroku is possible with your web browser alone. -The [**installation guide**](https://github.com/modmail-dev/modmail/wiki/Installation) (which includes a video tutorial!) will guide you through the entire installation process. If you run into any problems, join our [Modmail Discord Server](https://discord.gg/cnUpwrnpYb) for help and support. - -When using Heroku, you can configure automatic updates: - - Login to [GitHub](https://github.com/) and verify your account. - - [Fork the repo](https://github.com/modmail-dev/modmail/fork). - - Install the [Pull app](https://github.com/apps/pull) for your fork. - - Then go to the Deploy tab in your [Heroku account](https://dashboard.heroku.com/apps) of your bot app, select GitHub and connect your fork (usually by typing "Modmail"). - - Turn on auto-deploy for the `master` branch. +To develop your own, check out the [plugins documentation](https://github.com/modmail-dev/modmail/wiki/Plugins). + +Plugins requests and support are available in our [Modmail Support Server](https://discord.gg/cnUpwrnpYb). ## Sponsors @@ -208,17 +156,6 @@ Discord Advice Center: Become a sponsor on [Patreon](https://patreon.com/kyber). -## Plugins - -Modmail supports the use of third-party plugins to extend or add functionalities to the bot. -Plugins allow niche features as well as anything else outside of the scope of the core functionality of Modmail. - -You can find a list of third-party plugins using the `?plugins registry` command or visit the [Unofficial List of Plugins](https://github.com/modmail-dev/modmail/wiki/Unofficial-List-of-Plugins) for a list of plugins contributed by the community. - -To develop your own, check out the [plugins documentation](https://github.com/modmail-dev/modmail/wiki/Plugins). - -Plugins requests and support are available in our [Modmail Support Server](https://discord.gg/cnUpwrnpYb). - ## Contributing Contributions to Modmail are always welcome, whether it be improvements to the documentation or new functionality, please feel free to make the change. Check out our [contributing guidelines](https://github.com/modmail-dev/modmail/blob/master/.github/CONTRIBUTING.md) before you get started. From 6d61cf29ff0d9e8a3b984eb44a6337402246bff0 Mon Sep 17 00:00:00 2001 From: Amy Date: Sun, 19 Nov 2023 05:45:13 +0000 Subject: [PATCH 2/5] Add JSON logging support (#3305) * Add JSON logging support This adds support for JSON logging, along with the relevant options required. This does not change the default log behaviour, so should be backwards compatible. It is opt in via the LOG_FORMAT option, which can be 'json' to use the new logger, or anything else to fallback to the old behaviour. This is implemented in terms of a custom formatter, which is optionally applied to the stdout stream. The debug stream is unaffected by this. * Allow JSON to be selected when creating handlers * Allow different formats to be selected for streams/files * Remove old / unused code * Add new config opts to helpfile * Formatting, basic typing and reorder for consistency in project. --------- Co-authored-by: Jerrie-Aries Co-authored-by: Taku <45324516+Taaku18@users.noreply.github.com> --- core/config.py | 2 + core/config_help.json | 18 +++++++ core/models.py | 110 ++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 125 insertions(+), 5 deletions(-) diff --git a/core/config.py b/core/config.py index 527b2dc19f..a9e16a0df1 100644 --- a/core/config.py +++ b/core/config.py @@ -178,6 +178,8 @@ class ConfigManager: "disable_updates": False, # Logging "log_level": "INFO", + "stream_log_format": "plain", + "file_log_format": "plain", "discord_log_level": "INFO", # data collection "data_collection": True, diff --git a/core/config_help.json b/core/config_help.json index f909d27821..501c265827 100644 --- a/core/config_help.json +++ b/core/config_help.json @@ -1129,6 +1129,24 @@ "This configuration can only to be set through `.env` file or environment (config) variables." ] }, + "stream_log_format": { + "default": "plain", + "description": "The logging format when through a stream, can be 'plain' or 'json'", + "examples": [ + ], + "notes": [ + "This configuration can only to be set through `.env` file or environment (config) variables." + ] + }, + "file_log_format": { + "default": "plain", + "description": "The logging format when logging to a file, can be 'plain' or 'json'", + "examples": [ + ], + "notes": [ + "This configuration can only to be set through `.env` file or environment (config) variables." + ] + }, "discord_log_level": { "default": "INFO", "description": "The `discord.py` library logging level for logging to stdout.", diff --git a/core/models.py b/core/models.py index 29f6af71bb..2a397541c1 100644 --- a/core/models.py +++ b/core/models.py @@ -1,3 +1,4 @@ +import json import logging import os import re @@ -9,7 +10,7 @@ from logging import FileHandler, StreamHandler, Handler from logging.handlers import RotatingFileHandler from string import Formatter -from typing import Optional +from typing import Dict, Optional import discord from discord.ext import commands @@ -74,6 +75,71 @@ def line(self, level="info"): ) +class JsonFormatter(logging.Formatter): + """ + Formatter that outputs JSON strings after parsing the LogRecord. + + Parameters + ---------- + fmt_dict : Optional[Dict[str, str]] + {key: logging format attribute} pairs. Defaults to {"message": "message"}. + time_format: str + time.strftime() format string. Default: "%Y-%m-%dT%H:%M:%S" + msec_format: str + Microsecond formatting. Appended at the end. Default: "%s.%03dZ" + """ + + def __init__( + self, + fmt_dict: Optional[Dict[str, str]] = None, + time_format: str = "%Y-%m-%dT%H:%M:%S", + msec_format: str = "%s.%03dZ", + ): + self.fmt_dict: Dict[str, str] = fmt_dict if fmt_dict is not None else {"message": "message"} + self.default_time_format: str = time_format + self.default_msec_format: str = msec_format + self.datefmt: Optional[str] = None + + def usesTime(self) -> bool: + """ + Overwritten to look for the attribute in the format dict values instead of the fmt string. + """ + return "asctime" in self.fmt_dict.values() + + def formatMessage(self, record) -> Dict[str, str]: + """ + Overwritten to return a dictionary of the relevant LogRecord attributes instead of a string. + KeyError is raised if an unknown attribute is provided in the fmt_dict. + """ + return {fmt_key: record.__dict__[fmt_val] for fmt_key, fmt_val in self.fmt_dict.items()} + + def format(self, record) -> str: + """ + Mostly the same as the parent's class method, the difference being that a dict is manipulated and dumped as JSON + instead of a string. + """ + record.message = record.getMessage() + + if self.usesTime(): + record.asctime = self.formatTime(record, self.datefmt) + + message_dict = self.formatMessage(record) + + if record.exc_info: + # Cache the traceback text to avoid converting it multiple times + # (it's constant anyway) + if not record.exc_text: + record.exc_text = self.formatException(record.exc_info) + + if record.exc_text: + message_dict["exc_info"] = record.exc_text + + if record.stack_info: + message_dict["stack_info"] = self.formatStack(record.stack_info) + + return json.dumps(message_dict, default=str) + + class FileFormatter(logging.Formatter): ansi_escape = re.compile(r"\x1B\[[0-?]*[ -/]*[@-~]") @@ -85,11 +151,25 @@ def format(self, record): log_stream_formatter = logging.Formatter( "%(asctime)s %(name)s[%(lineno)d] - %(levelname)s: %(message)s", datefmt="%m/%d/%y %H:%M:%S" ) + log_file_formatter = FileFormatter( "%(asctime)s %(name)s[%(lineno)d] - %(levelname)s: %(message)s", datefmt="%Y-%m-%d %H:%M:%S", ) +json_formatter = JsonFormatter( + { + "level": "levelname", + "message": "message", + "loggerName": "name", + "processName": "processName", + "processID": "process", + "threadName": "threadName", + "threadID": "thread", + "timestamp": "asctime", + } +) + def create_log_handler( filename: Optional[str] = None, @@ -98,6 +178,7 @@ def create_log_handler( level: int = logging.DEBUG, mode: str = "a+", encoding: str = "utf-8", + format: str = "plain", maxBytes: int = 28000000, backupCount: int = 1, **kwargs, @@ -124,6 +205,9 @@ def create_log_handler( encoding : str If this keyword argument is specified along with filename, its value is used when the `FileHandler` is created, and thus used when opening the output file. Defaults to 'utf-8'. + format : str + The format to output with, can either be 'json' or 'plain'. Will apply to whichever handler is created, + based on other conditional logic. maxBytes : int The max file size before the rollover occurs. Defaults to 28000000 (28MB). Rollover occurs whenever the current log file is nearly `maxBytes` in length; but if either of `maxBytes` or `backupCount` is zero, @@ -141,23 +225,28 @@ def create_log_handler( if filename is None: handler = StreamHandler(stream=sys.stdout, **kwargs) - handler.setFormatter(log_stream_formatter) + formatter = log_stream_formatter elif not rotating: handler = FileHandler(filename, mode=mode, encoding=encoding, **kwargs) - handler.setFormatter(log_file_formatter) + formatter = log_file_formatter else: handler = RotatingFileHandler( filename, mode=mode, encoding=encoding, maxBytes=maxBytes, backupCount=backupCount, **kwargs ) - handler.setFormatter(log_file_formatter) + formatter = log_file_formatter + + if format == "json": + formatter = json_formatter handler.setLevel(level) + handler.setFormatter(formatter) return handler logging.setLoggerClass(ModmailLogger) log_level = logging.INFO loggers = set() + ch = create_log_handler(level=log_level) ch_debug: Optional[RotatingFileHandler] = None @@ -173,7 +262,11 @@ def getLogger(name=None) -> ModmailLogger: def configure_logging(bot) -> None: - global ch_debug, log_level + global ch_debug, log_level, ch + + stream_log_format, file_log_format = bot.config["stream_log_format"], bot.config["file_log_format"] + if stream_log_format == "json": + ch.setFormatter(json_formatter) logger = getLogger(__name__) level_text = bot.config["log_level"].upper() @@ -198,8 +291,15 @@ def configure_logging(bot) -> None: logger.info("Log file: %s", bot.log_file_path) ch_debug = create_log_handler(bot.log_file_path, rotating=True) + + if file_log_format == "json": + ch_debug.setFormatter(json_formatter) + ch.setLevel(log_level) + logger.info("Stream log format: %s", stream_log_format) + logger.info("File log format: %s", file_log_format) + for log in loggers: log.setLevel(log_level) log.addHandler(ch_debug) From 5c710596c870121d2df864ceec8d86f0333fbb49 Mon Sep 17 00:00:00 2001 From: Taku <45324516+Taaku18@users.noreply.github.com> Date: Sat, 18 Nov 2023 21:47:34 -0800 Subject: [PATCH 3/5] Update changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9830415b10..0b3a61943a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,7 @@ however, insignificant breaking changes do not guarantee a major version bump, s - `REGISTRY_PLUGINS_ONLY`, environment variable, when set, restricts to only allow adding registry plugins. ([PR #3247](https://github.com/modmail-dev/modmail/pull/3247)) - `DISCORD_LOG_LEVEL` environment variable to set the log level of discord.py. ([PR #3216](https://github.com/modmail-dev/Modmail/pull/3216)) - New registry plugin: [`autoreact`](https://github.com/martinbndr/kyb3r-modmail-plugins/tree/master/autoreact). +- `STREAM_LOG_FORMAT` and `FILE_LOG_FORMAT` for settings the log format of the stream and file handlers respectively. Possible options are `json` and `plain` (default). ([PR #3305](https://github.com/modmail-dev/Modmail/pull/3305)) ### Changed - Repo moved to https://github.com/modmail-dev/modmail. From ae99060a3fe4a0afcd2d2fa35af85e8fce5b8aaa Mon Sep 17 00:00:00 2001 From: Jerrie <70805800+Jerrie-Aries@users.noreply.github.com> Date: Sun, 19 Nov 2023 16:38:42 +0800 Subject: [PATCH 4/5] Fix rate limit issue on raw reaction add/remove events. (#3306) * Fix rate limit issue on raw reaction add/remove events. * Pasd message object to `find_linked_messages` since it is already fetched. --------- Co-authored-by: Taku <45324516+Taaku18@users.noreply.github.com> --- bot.py | 40 +++++++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 15 deletions(-) diff --git a/bot.py b/bot.py index 45530a908f..3701d826ea 100644 --- a/bot.py +++ b/bot.py @@ -1222,25 +1222,36 @@ async def handle_reaction_events(self, payload): return channel = self.get_channel(payload.channel_id) - if not channel: # dm channel not in internal cache - _thread = await self.threads.find(recipient=user) - if not _thread: + thread = None + # dm channel not in internal cache + if not channel: + thread = await self.threads.find(recipient=user) + if not thread: + return + channel = await thread.recipient.create_dm() + if channel.id != payload.channel_id: + return + + from_dm = isinstance(channel, discord.DMChannel) + from_txt = isinstance(channel, discord.TextChannel) + if not from_dm and not from_txt: + return + + if not thread: + params = {"recipient": user} if from_dm else {"channel": channel} + thread = await self.threads.find(**params) + if not thread: return - channel = await _thread.recipient.create_dm() + # thread must exist before doing this API call try: message = await channel.fetch_message(payload.message_id) except (discord.NotFound, discord.Forbidden): return reaction = payload.emoji - close_emoji = await self.convert_emoji(self.config["close_emoji"]) - - if isinstance(channel, discord.DMChannel): - thread = await self.threads.find(recipient=user) - if not thread: - return + if from_dm: if ( payload.event_type == "REACTION_ADD" and message.embeds @@ -1248,7 +1259,7 @@ async def handle_reaction_events(self, payload): and self.config.get("recipient_thread_close") ): ts = message.embeds[0].timestamp - if thread and ts == thread.channel.created_at: + if ts == thread.channel.created_at: # the reacted message is the corresponding thread creation embed # closing thread return await thread.close(closer=user) @@ -1268,11 +1279,10 @@ async def handle_reaction_events(self, payload): logger.warning("Failed to find linked message for reactions: %s", e) return else: - thread = await self.threads.find(channel=channel) - if not thread: - return try: - _, *linked_messages = await thread.find_linked_messages(message.id, either_direction=True) + _, *linked_messages = await thread.find_linked_messages( + message1=message, either_direction=True + ) except ValueError as e: logger.warning("Failed to find linked message for reactions: %s", e) return From a8d7c26d8cfd863c13b4862ffae1cb998a13fda8 Mon Sep 17 00:00:00 2001 From: Taku <45324516+Taaku18@users.noreply.github.com> Date: Sun, 19 Nov 2023 02:21:42 -0800 Subject: [PATCH 5/5] Update changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b3a61943a..76a929b178 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ however, insignificant breaking changes do not guarantee a major version bump, s - Fixed a syntactic error in the close message when a thread is closed after a certain duration. ([PR #3233](https://github.com/modmail-dev/Modmail/pull/3233)) - Removed an extra space in the help command title when the command has no parameters. ([PR #3271](https://github.com/modmail-dev/Modmail/pull/3271)) - Corrected some incorrect config help descriptions. ([PR #3277](https://github.com/modmail-dev/Modmail/pull/3277)) +- Rate limit issue when fetch the messages due to reaction linking. ([PR #3306](https://github.com/modmail-dev/Modmail/pull/3306)) ### Added - `?log key ` to retrieve the log link and view a preview using a log key. ([PR #3196](https://github.com/modmail-dev/Modmail/pull/3196))