-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updated Quota Support to Latest Version of MiaB and resolving code review comments #2387
base: main
Are you sure you want to change the base?
Conversation
Looks like 15-mailbox.conf is just relocated and the script is already updated so it's fine as-is. |
Does not appear to be any reason why https://raw.githubusercontent.com/jrsupplee/mailinabox/master/setup/bootstrap.sh was being used instead of https://mailinabox.email/setup.sh?ping=1 -- my best guess is for reasons of pinning the version |
.gitignore
Outdated
@@ -5,5 +5,6 @@ tools/__pycache__/ | |||
externals/ | |||
.env | |||
.vagrant | |||
.idea/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
small change happy to remove if you want
Woops looks like basing off main was a bad idea. Gonna redo this and base instead off of v68 |
f82be1b
to
1795f8a
Compare
So the conflict is trivial to resolve (just bring in the first line of the |
strangely, I get a 500 error the first time I run |
When it 500s after setting the password, it seems it fails to create the admin user. Creating the admin user by hand seems to work?
|
Fixed the 500, it was related to missing the subprocess module. Fixed some conflicts with the spamassassin.sh script also |
I've resolved the conflict -- the remote chadfurman/mailinabox has a somewhat stable and tested version of this tagged as |
management/cli.py
Outdated
@@ -117,6 +123,14 @@ def setup_key_auth(mgmt_uri): | |||
if "admin" in user['privileges']: | |||
print(user['email']) | |||
|
|||
elif sys.argv[1] == "user" and sys.argv[2] == "quota" and len(sys.argv) == 4: | |||
# Set a user's quota |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this comment correct? I don't understand how this sets a user's quota.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this is going to display not set a users current quota.
Not sure if it helps or not but I've installed this with the v68a 'tag' on my production MiaB server and can confirm it works. Steps I've taken to install it: ssh into your box -
|
setup/mail-users.sh
Outdated
echo "CREATE TABLE aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path"; | ||
echo "CREATE TABLE mfa (id INTEGER PRIMARY KEY AUTOINCREMENT, user_id INTEGER NOT NULL, type TEXT NOT NULL, secret TEXT NOT NULL, mru_token TEXT, label TEXT, FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE);" | sqlite3 "$db_path"; | ||
echo "CREATE TABLE auto_aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path"; | ||
elif sqlite3 $db_path ".schema users" | grep --invert-match quota; then | ||
echo "ALTER TABLE users ADD COLUMN quota TEXT NOT NULL DEFAULT '0';" | sqlite3 $db_path; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You've opted to make the quota column type TEXT versus making it a numeric column. Just curious if there was a reason for this choice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pretty sure what is stored in sqlite is basically for the webui to 'display' with what is actually set with maildirsize file for dovecot.
Also don't forget this is based on proven code from @jrsupplee, so some of this code is obviously not written by Chad.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I see is we're storing actually a string like "3G" (3 gigabyte limit), not an integer number of bytes. So this is ok.
setup/mail-users.sh
Outdated
@@ -20,10 +20,12 @@ db_path=$STORAGE_ROOT/mail/users.sqlite | |||
# Create an empty database if it doesn't yet exist. | |||
if [ ! -f "$db_path" ]; then | |||
echo "Creating new user database: $db_path"; | |||
echo "CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, email TEXT NOT NULL UNIQUE, password TEXT NOT NULL, extra, privileges TEXT NOT NULL DEFAULT '');" | sqlite3 "$db_path"; | |||
echo "CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, email TEXT NOT NULL UNIQUE, password TEXT NOT NULL, extra, privileges TEXT NOT NULL DEFAULT '', quota TEXT NOT NULL DEFAULT '0');" | sqlite3 $db_path; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I understand correctly, '0' means unlimited and non-zero represents the quota. But there's no symbolic representation for "system default' from what I can see. So for example, if I made the system default 1GB, then the users would all get their DB value set to 1GB. If I later were to increase the system default to 10GB, the existing users would stay at 1GB and not inherit the new default.
Correct me if I've overlooked something and that's not how it works, but assuming that description is correct, then I'm wondering if the users that are getting the "system default quota" should have a symbolic representation of this in the DB, so if the default were to be changed, these users inherit the new value automatically.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Quota is not applied, by default so it is basically set to 0. 0 represents "no quota or 'unlimited' quota" (obviously the limit is the size of the hard-drive for miab server.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think my confusion stems from the fact that 0 means unlimited, but I also see a system default-quota
value that can be set, and I'm not sure what role that default value plays in this. How would I set a user to the "default-quota" value?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay I understand what you would like to see, but I believe this is a limitation of Dovecot / Postfix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dms00's question is right. The purpose of the default quota is going to cause confusion. What I see is that it's a default for new users. It's not a default for existing users (prior to this PR) or users without a quota set - they all get no quota. I think the way to resolve this is to just remove the default quota functionality to avoid confusion.
@MrMirhan's point is probably not solvable with this projects architecture because it's up to the mail client to do the send and then save the mail. As @stylnchris alluded to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand the confusion around the default quota bits. I have to look into this more. As I understand it, default quota could either be the default quota for new users or the default quota for all users that don't have a quota set. But since "not having a quota set" means "having your quota set to '0' because it's unlimited" then the question becomes: if, in fact, "default quota" is the quota for all users who don't have a quota set, then is it actually just ignored? And, alternatively, if "default quota" is just for the new users, then is it actually more confusing than useful?
I agree, removing default quota seems to make sense. I can look into this. Likely just a UI adjustment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So there's a bit of a question here -- if we remove the default value of '0' from the table, it will need to be replaced with something otherwise all users will require a quota. Replacing it with empty string or Null makes the most sense, but i'm not sure how this will ripple out, so this is TBD?
Alternatively, if we remove the "system default" API endpoints, CLI commands, etc, this would reduce confusion while allowing users to continue to default to unlimited which I think makes the most sense.
That said, ideally I could make the system default "unlimited" and then have a per-user quota and where users aren't specifically given a quota then when the system default changes the user's quota would also change for any user where the quota isn't set. That said, I'm not at all sure how to do this with dovecot. I can likely do this with the python / shell commands without too much trouble, but how dovecot will handle it is something I need to understand a bit better...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed here: 7c7b744 -- note that for this we just remove the ability to configure the system-default because it's confusing and default everyone to unlimited because it's simple.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I appreciate the effort and the patience while I found time to review. I'm doing this from airplane wifi! (It's the only time I don't have other responsibilities!)
Since there are some confusing things, I'd like to try to reduce the complexity of the PR to just side step those possible confusions. I'm not sure if using the old maildir backend is a blocker. It would require several changes. Moving the db migration to our migration script is a blocker so that all migrations are in the same place.
Thanks!
management/cli.py
Outdated
@@ -117,6 +123,14 @@ def setup_key_auth(mgmt_uri): | |||
if "admin" in user['privileges']: | |||
print(user['email']) | |||
|
|||
elif sys.argv[1] == "user" and sys.argv[2] == "quota" and len(sys.argv) == 4: | |||
# Set a user's quota |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this is going to display not set a users current quota.
management/mailconfig.py
Outdated
"box_quota": box_quota, | ||
"box_size": sizeof_fmt(box_size) if box_size != '?' else box_size, | ||
"percent": '%3.0f%%' % percent if type(percent) != str else percent, | ||
"box_count": box_count, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess this is the number of messages. Since we don't enforce a limit for this, it's a little confusing to report. I'd suggest removing it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's used in the management users.html template mostly. I'm not sure how useful this is to have there, but maybe @stylnchris could shed some light on this for us?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Number of messages can be removed. I do find it somewhat helpful but I do understand that its not part of the scope of adding quotas. This would be "another feature" I suppose.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The counting of messages has been removed here: c91db7b
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this file is mistakenly added. I think it duplicates a file two directories up. I didn't check if the files are different.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
resolved in 8a1e803
setup/mail-dovecot.sh
Outdated
@@ -66,7 +66,33 @@ tools/editconf.py /etc/dovecot/conf.d/10-mail.conf \ | |||
first_valid_uid=0 | |||
|
|||
# Create, subscribe, and mark as special folders: INBOX, Drafts, Sent, Trash, Spam and Archive. | |||
cp conf/dovecot-mailboxes.conf /etc/dovecot/conf.d/15-mailboxes.conf | |||
cp conf/dovecot/conf.d/15-mailboxes.conf /etc/dovecot/conf.d/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this change is mistakenly included along with the added file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
resolved in 8a1e803
setup/mail-users.sh
Outdated
@@ -20,10 +20,12 @@ db_path=$STORAGE_ROOT/mail/users.sqlite | |||
# Create an empty database if it doesn't yet exist. | |||
if [ ! -f "$db_path" ]; then | |||
echo "Creating new user database: $db_path"; | |||
echo "CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, email TEXT NOT NULL UNIQUE, password TEXT NOT NULL, extra, privileges TEXT NOT NULL DEFAULT '');" | sqlite3 "$db_path"; | |||
echo "CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, email TEXT NOT NULL UNIQUE, password TEXT NOT NULL, extra, privileges TEXT NOT NULL DEFAULT '', quota TEXT NOT NULL DEFAULT '0');" | sqlite3 $db_path; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dms00's question is right. The purpose of the default quota is going to cause confusion. What I see is that it's a default for new users. It's not a default for existing users (prior to this PR) or users without a quota set - they all get no quota. I think the way to resolve this is to just remove the default quota functionality to avoid confusion.
@MrMirhan's point is probably not solvable with this projects architecture because it's up to the mail client to do the send and then save the mail. As @stylnchris alluded to.
setup/mail-users.sh
Outdated
echo "CREATE TABLE aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path"; | ||
echo "CREATE TABLE mfa (id INTEGER PRIMARY KEY AUTOINCREMENT, user_id INTEGER NOT NULL, type TEXT NOT NULL, secret TEXT NOT NULL, mru_token TEXT, label TEXT, FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE);" | sqlite3 "$db_path"; | ||
echo "CREATE TABLE auto_aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path"; | ||
elif sqlite3 $db_path ".schema users" | grep --invert-match quota; then | ||
echo "ALTER TABLE users ADD COLUMN quota TEXT NOT NULL DEFAULT '0';" | sqlite3 $db_path; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have a database migration script. This needs to move there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed in 1c66f69
setup/mail-users.sh
Outdated
echo "CREATE TABLE aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path"; | ||
echo "CREATE TABLE mfa (id INTEGER PRIMARY KEY AUTOINCREMENT, user_id INTEGER NOT NULL, type TEXT NOT NULL, secret TEXT NOT NULL, mru_token TEXT, label TEXT, FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE);" | sqlite3 "$db_path"; | ||
echo "CREATE TABLE auto_aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path"; | ||
elif sqlite3 $db_path ".schema users" | grep --invert-match quota; then | ||
echo "ALTER TABLE users ADD COLUMN quota TEXT NOT NULL DEFAULT '0';" | sqlite3 $db_path; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I see is we're storing actually a string like "3G" (3 gigabyte limit), not an integer number of bytes. So this is ok.
if ! grep -q "quota_status_success = DUNNO" /etc/dovecot/conf.d/90-quota.conf; then | ||
cat > /etc/dovecot/conf.d/90-quota.conf << EOF; | ||
plugin { | ||
quota = maildir |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a newer quota backend called "count" that is recommended. I'm reluctant to introduce a new feature that uses an out of date practice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Definitely gonna take a bit of investigation to make this happen.
The count backend:
https://github.com/dovecot/core/blob/076cf225aa8d18e8abb0cbdc04f68abdf1c62a5e/src/plugins/quota/quota-count.c#L8
The maildir backend:
https://github.com/dovecot/core/blob/076cf225aa8d18e8abb0cbdc04f68abdf1c62a5e/src/plugins/quota/quota-maildir.c
My C is quite rusty (no pun intended), so getting this in will be a bit of a trick. I have a path forward that would involve switching to the new backend, seeing what breaks, and then debugging it by looking at the C code and the database.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See #2387 (comment) for why I think it may be fine to leave the maildir backend
@JoshData I've addressed these comments: the migration issue here: 1c66f69 duplicate conf issue here: 8a1e803 CLI comment thing here: 4e9c564 (note that this was actually supposed to be a GET and a SET) Removal of the default fixed here: 7c7b744 -- note that for this we just remove the ability to configure the system-default because it's confusing and default everyone to unlimited because it's simple. The counting of messages has been removed here: c91db7b The only comment that I haven't addressed yet is around switching to the new count backend -- this is definitely a more involved change, if only because I need to test it thoroughly to make sure there's no unexpected side-effects. As far as I can tell, it might "just work" but I need to test it and make sure. Anything that starts acting up, I'll have to diff the two C files I linked and try to understand why/how/where etc. Not opposed to this, but waiting to hear from you if this is a blocker or not before I put in the time. |
Looking at https://doc.dovecot.org/configuration_manual/quota/quota_maildir/ it appears that, given we are using the maildir format, unless we want to shift away from using the maildir format in the future (which may be a bigger lift?) then we are likely fine to keep with the quota = maildir backend. The count backend will give us more flexibility which we don't need at the moment, and so given that the maildir backend for quota is the most commonly used backend for quota with the maildir format I'm thinking that it is safe to move forward without changing this. |
Note for the community: If anyone has used this branch before the migration change, you'll want to set |
@JoshData just checking in, here. Anything you still would like to see on this before merging? |
FWIW, here are a few items that came to mind when I was checking this out today.
|
Okay I'll take a look, thank you for the detailed review:)
…On Fri, Sep 6, 2024, 21:01 Downtown Allday ***@***.***> wrote:
FWIW, here are a few items that came to mind when I was checking this out
today.
1. The new argument method added to cli.py::mgmt() seriously breaks
other important commands that require POST (like 'user add', 'password',
etc). Removing the method argument and it's use inside mgmt() works
fine for me. I don't think it's needed.
2. cli.py user outputs unlimited for quotas set to 0 however, cli.py
user quota ***@***.*** outputs 0 for a quota set to 0, which is
inconsistent.
3. Including quota output in cli.py user output may break scripts that
are already expecting a certain output format.
4. It is confusing to allow a quota value in the admin UI to be just a
number without a unit (M or G).
5. The quota_grace setting should be "10%%" not "10%" becuse "%" is a
special character to dovecot.
6. I think it's important to add the quota_warning dovecot feature.
Currently, when a user exceeds their quota, incoming messages are bounced
by postfix (the sender receives an automated email from the server saying
the recipients mailbox is full). The problem is the user never knows
they're about to exceed their mailbox quota until it's too late. The
quota_warning feature of the dovecot quota plugin can alert the user that
they're close to exceeding their quota before they start losing messages.
See an example here:
https://unix.stackexchange.com/questions/363658/how-to-send-warning-with-dovecot-to-users-when-over-quota
.
—
Reply to this email directly, view it on GitHub
<#2387 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BGRID53OX7CJJIDYAZCBILLZVJF53AVCNFSM6AAAAABG4E3RPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZUHE3TQOJXGM>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
I also think Drafts and Sent folders should not count toward quota. Drafts because Roundcube automatically saves to Drafts every 5 minutes and you might be surprised that your draft wan't there if the message causes over quota, or an incoming message causes over quota. Roundcube does give a brief toast notification about the inability to save to draft, though. Sent becuase Roundcube automatically saves to Sent and it's probably more important to most that sent messages are retained. As-is you can send messages in an over quota situation, but the sent message fails to copy to Sent folder.
|
box_quota = 0 | ||
percent = '' | ||
try: | ||
dirsize_file = os.path.join(env['STORAGE_ROOT'], 'mail/mailboxes/%s/%s/maildirsize' % (domain, user)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of reading the "maildirsize" file directly, which is specific to the maildir quota backend, why not execute doveadm -f json quota get -u <email>
and parse the json, which contains both quota ("limit") in KiB, current size ("value") also in KiB, and percent used ("percent"). The "STORAGE" type is what you want. "MESSAGE" is a count of messages. The json looks like this:
[{"root":"User quota","type":"STORAGE","value":"2","limit":"5120","percent":"0"},{"root":"User quota","type":"MESSAGE","value":"1","limit":"-","percent":"0"}]```
Any help on this that pushes it over the finish line is appreciated, but please no scope creep other than what is required. On Sep 7, 2024, at 10:42 AM, Downtown Allday ***@***.***> wrote:
@downtownallday commented on this pull request.
In management/mailconfig.py:
active_accounts.add(email)
+ (user, domain) = email.split('@')
+ box_size = 0
+ box_quota = 0
+ percent = ''
+ try:
+ dirsize_file = os.path.join(env['STORAGE_ROOT'], 'mail/mailboxes/%s/%s/maildirsize' % (domain, user))
Instead of reading the "maildirsize" file directly, which is specific to the maildir quota backend, why not execute doveadm -f json quota get -u <email> and parse the json, which contains both quota ("limit") in KiB, current size ("value") also in KiB, and percent used ("percent"). The "STORAGE" type is what you want. "MESSAGE" is a count of messages. The json looks like this:
[{"root":"User quota","type":"STORAGE","value":"2","limit":"5120","percent":"0"},{"root":"User quota","type":"MESSAGE","value":"1","limit":"-","percent":"0"}]```
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
…is always unlimited.
This reverts commit a4a0898.
I've updated to v70 and fixed a few bugs:
Open questions:
I'm going to hold off on all the above changes until I hear from @JoshData -- for now, this PR is in maintenance mode and I will fix bugs and work to update it to new versions periodically; however, new features and major changes will only be made if it has a high probability of resulting in this PR getting merged to mainline. |
^ @jrsupplee
This PR is an update of #1568 ported to the latest version of MiaB.
TODO: