-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mongoengine killed mongodb performance when used with pymongo 3.x #1446
Comments
#1457 First draft of changes, unfortunately I went to far and did change to switching db's and collections, but previous approach wasn't thread safe and was quite dirty. Test should pass but I didn't write any new as I prefer to get feedback if changes are going in right direction. |
Any news about this? Any workaround? |
Full history of the issue: Previously,
Most likely the best way to fix this issue is to implement some of the ideas mentioned in #357. |
Im also facing the same issue (based on . my understanding) Im using mongoengine==0.13.0. i have a collection that 28million documents and everytime, a new document is created mongodb is reindexing the entire data. Based on the
my model is
How can I avoid re-indexing the entire data on every save(). This is actually block me from reading the database. I dont want to remove unique=True though.. any thoughts !! |
Issue is quite old but I'm trying to understand the impact. Mongo index documentation mentions that
So unless you modify the indexes, the subsequent calls to create_indexes aren't actually re-creating the indexes. That leaves us with the overhead of a few python calls (involved in the Long story short, it is still valuable to improve this but it looks like skipping the call to |
Any updates on this? I'm not seeing a big performance hit on my database, but I'm seeing at leas 15ms added to every request that creates a document. |
The root problem is that mongoengine switched from pymongo
ensure_index
tocreate_index
method inside mongoengineensure_indexes
.ensure_index
had simple caching mechanizm butcreate_index
doesn't have it.This switch plus that
ensure_indexes
is called on eachdoc.save()
results in that mongodb need to handle withcreateIndex
which is quite heavy operation and in our case this results in 50% drop in performance.ensure_index
was added here #812 because there was problem with test and IMHO it was a workaround. Better solution could be to make proxy methoddrop_database
similar todrop_collection
which would resetcls._collection
on Document.Later I will prepare pull request with such change.
The text was updated successfully, but these errors were encountered: