Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add auto_indexes feature to LazyInstance #82

Closed
wants to merge 3 commits into from
Closed

Add auto_indexes feature to LazyInstance #82

wants to merge 3 commits into from

Conversation

lafrech
Copy link
Collaborator

@lafrech lafrech commented Dec 12, 2016

Closes #81.

What do you think about this ?

It works only on documents that are already registered when calling instance.init().

TODO:

  • Add tests for drivers other than PyMongo (currently only PyMongo is tested)

  • Document the feature

  • Also ensure indexes on documents registered after instance.init()?

@lafrech
Copy link
Collaborator Author

lafrech commented Dec 12, 2016

Obviously, it works only on documents that are already registered when calling instance.init().

I think this could be addressed in Instance._register_doc.

@touilleMan , waiting for your feedback before going any further. This is not critical to us.

@coveralls
Copy link

coveralls commented Dec 12, 2016

Coverage Status

Coverage increased (+0.07%) to 95.665% when pulling 7c50390 on Nobatek:dev_auto_indexes into 65c8e01 on Scille:master.

@touilleMan
Copy link
Member

I think this feature is pretty interesting given calling ensure_index by hand is cumbersome and error prone ;-)

Few remarks:

  1. With not implementing this for Instance as well as LazyLoaderInstance ?

  2. The trouble with implementing this at the generic instance level is ensure_indexes won't do the same thing depending of the framework used: the indexes will be up and ready for pymongo were for asyncio and twisted only coroutine has been registered (which will be executed in the future... maybe... or can raise a exception which won't be catched...).
    So I would have this feature in the builder given each framework has it own implementation:

class Instance:
    ...
    def register(self, template):
        # overload the default register to add a synchronous creation of indexes if needed
        implementation = super().register(template)
        if self.auto_indexes:
            self.builder.ensure_indexes(implementation)

class LazyLoaderInstance:
    ...
    def init(self, db):
        ...
        # Now we can finish initialization of all the documents
        for doc in self._doc_lookup.values():
            self.builder.ensure_indexes(doc)

class PyMongoBuilder(BaseBuilder):

    BASE_DOCUMENT_CLS = PyMongoDocument

    ...

    def ensure_indexes(self, implementation):
        # For async framework, we create a reactor just for this coroutine so 
        # we end up with something synchronous anyway
        implementation.ensure_indexes()    
  1. I don't like putting the auto_indexes option into the document's meta fields: I feel like document declaration should be independent of it execution as much as possible (we are defining a Template after all !). Using auto_indexes or not depend of which instance is used and not on what is the document made of.
    I think it would be better to add an option to the instance.register instead:
instance = Instance(db, auto_indexes=False)

@instance.register(force_auto_indexes_value=True)
class MyDoc(Document):
    ...

@lafrech
Copy link
Collaborator Author

lafrech commented Dec 17, 2016

Thank you for your feedback.

  1. Why not implement this for Instance as well as LazyLoaderInstance ?

I don't remember why I did not do it for Instance. I think I had a reason, probably a bad one...

  1. & 3.

Good points.

register(force_auto_indexes_value=True) was my first intent in #81, but then I changed my mind. I shouldn't have...

@touilleMan
Copy link
Member

I'm closing this because pymongo's ensure_index has been deprecated.

This makes creating automatically indexes much more complicated (plenty of complex concurrency bugs there, that's the reason pymongo dropped the feature in the first place).

For instance, this "don't bother user with this" approach created serious perf issues with mongoengine that are now complicated to fix.

@touilleMan touilleMan closed this Jan 21, 2018
@lafrech lafrech deleted the dev_auto_indexes branch January 21, 2018 20:10
@lafrech lafrech restored the dev_auto_indexes branch January 21, 2018 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants