-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Revamped index management #357
Comments
Hi @wojcikstefan I'm going to ponder this one! I like it conceptually, however I'm not sold on have auto index generation as you say it can take time and maybe undesirable in production. Maybe one to review for 0.9. |
By "auto index generation" I meant the current status quo, i.e. MongoEngine ensures all of the defined indexes are actually created in the db at runtime. If you add something to |
+1 |
I have one more request related to indexes - there should be an easy way to globally disable automatic index creation for all documents. While it's a useful feature during development, it's dangerous for production environments... You want full control over when and what index is created (and how). It's also better to decouple index definitions from application logic if you have a reasonably big engineering team, because it allows indexes to be managed by a separate DevOps team. |
Wanted to bring this one back again as we also have just experienced pretty significant downtime due to automated foreground index creation on production. Just as @wojcikstefan said, at some point it is desirable to disable automated index creation completely and let devops control this. |
Few requirements for the updated index management:
Anything else y'all might think of @lafrech @thedrow @mclate @gukoff ? |
This comment has been minimized.
This comment has been minimized.
Speaking from experience, I'd definitely prefer no automated index creation as a sane default intended for production with the option to do automated background index creation if desired. Automated foreground index creation should be clearly documented as intended for development environments only. |
@wojcikstefan I agree with your suggestions. What about also removing field arguments related to indexes, so that all index info would be in one place -- |
I think the
On the one hand, I agree. The more scattered index creation is, the harder it is to grasp what indexes are defined in your code (which is particularly dangerous if indexes are auto-created). On the other hand, people might be used to index-related arguments on fields. For example, latest Django still offers Is dropping field options like |
I agree it would make index definitions more explicit, but it is against what most people coming to mongoengine from other "ORM"s would expect. That said, as long as it is fine breaking backwards compatibility, dropping those field options sounds good to me. |
The more I think about it, the more I like the idea of defining all of the indexes in This is similar to defining all of your indexes via
I don't mind introducing a breaking change if it's done alongside a major version bump. Perhaps we could get to MongoEngine v1.0.0 after this and some performance/functional/cleanup improvements... @bagerard what do you think? |
Hi @rozza!
Just wanted to pick your brain about the default index creation. Seems like non-background indexes are only useful at the very early stage of a project. When any of the collections grow fairly large (say above 1M documents),
createIndex
blocks the database for a significant amount of time.It's a bit annoying to add
{'index_background': True}
in each Document's meta and sometimes you're just not able to control it that way. E.g. changingallow_inheritance
will create a new non-background index (although I may be wrong whether_cls
-related indexes respect meta'sindex_background
or not).I'd argue that
{ background: true }
should be a default option in MongoEngine. What do you think?The text was updated successfully, but these errors were encountered: