-
Notifications
You must be signed in to change notification settings - Fork 135
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: backup and restore scripts #72
base: master
Are you sure you want to change the base?
Conversation
cc @mserranom @gh2k |
local mongo_tmp_dir="$tmp_dir/backup/mongo" | ||
mkdir "$mongo_tmp_dir" | ||
|
||
"$TOOLKIT_ROOT/bin/docker-compose" up -d mongo |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we handle the case where there is an external Mongo DB here? I think there will be people running without the internal DBs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, would be useful to have a --skip-mongo
option for both backup/restore.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In addition, if we go with external redis, it could also have a --skip-redis
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking that we could look at the config in some way and back up the external mongo/redis using the same mechanism. Given that we've already checked the prerequisites for making a backup (the service is stopped so nothing will be in-flight) it makes sense to roll up the backup from the external DB at the same time. WDYT?
"$TOOLKIT_ROOT/bin/docker-compose" stop 2>/dev/null | ||
|
||
## Dump mongo | ||
echo "Dumping mongo..." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we also need to back up redis?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we have gone back and forth on this, the docs used to say it wasn't needed but were changed earlier this year: https://github.com/overleaf/overleaf/wiki/Backup-of-Data/_compare/dd55f820bf868464db70cbf8f584a608b6425c1b...f07e059281545c18907f9971e0fd18d9492b0127
I believe it's the case that if the editor is closed, users are disconnected, and then everything is shut down cleanly (in that order), nothing left in Redis is absolutely essential. But since we don't currently have a way to close the editor and disconnect users from a script, it seems safer to back up both, given various problems that admins have run into on support. (I think scripting the close editor and disconnect users actions was being discussed elsewhere – if we add that to the backup script then maybe Redis backup is indeed not needed.)
echo '[mongo is ready]' | ||
} | ||
|
||
function create-tmp-dir () { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could mktemp
be used here?
rm -rf /tmp/dump;' | ||
} | ||
|
||
function copy-data-files () { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are data files really needed for the backup? Most of them would be intermediate compile results right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I assumed that there would be e.g. filestore data here?
I think the The reason is that, according to docs, we need to gracefully close all editor sessions before attempting a backup. This should be done separately, maybe as part of |
Given that users are likely to want to run this on a cron, is there a way to send a disconnectAllUsers, prevent reconnection and wait for them to all disconnect, and then make a backup? If the only mechanism to do this is manual, then it's tricky to get a backup done on a sensible schedule. |
There isn't unfortunately, see https://github.com/overleaf/internal/issues/4951 |
How do we feel about resurrecting this work? @gh2k |
I guess this falls under @cmcisaac6's remit... I like the idea of having a canonical way to perform a backup though. |
Could we create a ticket to cover this so that it's tracked on the main board, please? |
Description
Work in progress. Do not merge.
This needs a lot more work to be usable in practice, but... the backup and restore process works, just like in our main monorepo.
Related issues / Pull Requests
Contributor Agreement