Pep8 and more clean-up

AUTHORS

@@ -12,7 +12,7 @@ Laine Herron https://github.com/LaineHerron
 
 CONTRIBUTORS
 
-Dervived from the git logs, inevitably incomplete but all of whom and others
+Derived from the git logs, inevitably incomplete but all of whom and others
 have submitted patches, reported bugs and generally helped make MongoEngine
 that much better:
 

CONTRIBUTING.rst

@@ -29,7 +29,10 @@ Style Guide
 -----------
 
 MongoEngine aims to follow `PEP8 <http://www.python.org/dev/peps/pep-0008/>`_
-including 4 space indents and 79 character line limits.
+including 4 space indents. When possible we try to stick to 79 character line limits.
+However, screens got bigger and an ORM has a strong focus on readability and
+if it can help, we accept 119 as maximum line length, in a similar way as
+`django does <https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/coding-style/#python-style>`_
 
 Testing
 -------
@@ -38,6 +41,10 @@ All tests are run on `Travis <http://travis-ci.org/MongoEngine/mongoengine>`_
 and any pull requests are automatically tested by Travis. Any pull requests
 without tests will take longer to be integrated and might be refused.
 
+You may also submit a simple failing test as a PullRequest if you don't know
+how to fix it, it will be easier for other people to work on it and it may get
+fixed faster.
+
 General Guidelines
 ------------------
 
@@ -48,6 +55,7 @@ General Guidelines
   from the cmd line to run the test suite).
 - Ensure tests pass on every Python and PyMongo versions.
   You can test on these versions locally by executing ``tox``
+- Add enhancements or problematic bug fixes to docs/changelog.rst
 - Add yourself to AUTHORS :)
 
 Documentation

README.rst

@@ -26,7 +26,9 @@ a `tutorial <http://readthedocs.org/docs/mongoengine-odm/en/latest/tutorial.html
 
 Installation
 ============
-If you have `setuptools <http://peak.telecommunity.com/DevCenter/setuptools>`_
+We recommend the use of `virtualenv <https://virtualenv.pypa.io/>`_ and of
+`pip <https://pip.pypa.io/>`_. You can then use ``pip install -U mongoengine``.
+You may also have `setuptools <http://peak.telecommunity.com/DevCenter/setuptools>`_ and thus
 you can use ``easy_install -U mongoengine``. Otherwise, you can download the
 source from `GitHub <http://github.com/MongoEngine/mongoengine>`_ and run ``python
 setup.py install``.
@@ -114,7 +116,7 @@ Also use the -s argument if you want to print out whatever or access pdb while t
 
 .. code-block:: shell
 
-    $ python setup.py nosetests --tests tests/test_django.py:QuerySetTest.test_get_document_or_404 -s
+    $ python setup.py nosetests --tests tests/fields/fields.py:FieldTest.test_cls_field -s
 
 Community
 =========

docs/_themes/sphinx_rtd_theme/footer.html

@@ -2,7 +2,7 @@
   {% if next or prev %}
     <div class="rst-footer-buttons">
       {% if next %}
-        <a href="{{ next.link|e }}" class="btn btn-neutral float-right" title="{{ next.title|striptags|e }}"/>Next <span class="icon icon-circle-arrow-right"></span></a>
+        <a href="{{ next.link|e }}" class="btn btn-neutral float-right" title="{{ next.title|striptags|e }}">Next <span class="icon icon-circle-arrow-right"></span></a>
       {% endif %}
       {% if prev %}
         <a href="{{ prev.link|e }}" class="btn btn-neutral" title="{{ prev.title|striptags|e }}"><span class="icon icon-circle-arrow-left"></span> Previous</a>

docs/changelog.rst

@@ -2,9 +2,15 @@
 Changelog
 =========
 
+Changes in 0.10.1 - DEV
+=======================
 
-Changes in 0.9.X - DEV
-======================
+Changes in 0.10.0
+=================
+- Django support was removed and will be available as a separate extension. #958
+- Allow to load undeclared field with meta attribute 'strict': False #957
+- Support for PyMongo 3+ #946
+- Removed get_or_create() deprecated since 0.8.0. #300
 - Improve Document._created status when switch collection and db #1020
 - Queryset update doesn't go through field validation #453
 - Added support for specifying authentication source as option `authSource` in URI. #967
@@ -14,17 +20,14 @@ Changes in 0.9.X - DEV
 - Use sets for populating dbrefs to dereference
 - Fixed unpickled documents replacing the global field's list. #888
 - Fixed storage of microseconds in ComplexDateTimeField and unused separator option. #910
-- Django support was removed and will be available as a separate extension. #958
 - Don't send a "cls" option to ensureIndex (related to https://jira.mongodb.org/browse/SERVER-769)
 - Fix for updating sorting in SortedListField. #978
 - Added __ support to escape field name in fields lookup keywords that match operators names #949
-- Support for PyMongo 3+ #946
 - Fix for issue where FileField deletion did not free space in GridFS.
 - No_dereference() not respected on embedded docs containing reference. #517
 - Document save raise an exception if save_condition fails #1005
 - Fixes some internal _id handling issue. #961
 - Updated URL and Email Field regex validators, added schemes argument to URLField validation. #652
-- Removed get_or_create() deprecated since 0.8.0. #300
 - Capped collection multiple of 256. #1011
 - Added `BaseQuerySet.aggregate_sum` and `BaseQuerySet.aggregate_average` methods.
 - Fix for delete with write_concern {'w': 0}. #1008

docs/django.rst

@@ -15,5 +15,5 @@ The MongoEngine team is looking for help contributing and maintaining a new
 Django extension for MongoEngine! If you have Django experience and would like
 to help contribute to the project, please get in touch on the 
 `mailing list <http://groups.google.com/group/mongoengine-users>`_ or by 
-simpily contributing on 
+simply contributing on
 `GitHub <https://github.com/MongoEngine/django-mongoengine>`_.

docs/guide/querying.rst

@@ -598,7 +598,7 @@ Some variables are made available in the scope of the Javascript function:
 
 The following example demonstrates the intended usage of
 :meth:`~mongoengine.queryset.QuerySet.exec_js` by defining a function that sums
-over a field on a document (this functionality is already available throught
+over a field on a document (this functionality is already available through
 :meth:`~mongoengine.queryset.QuerySet.sum` but is shown here for sake of
 example)::
 

mongoengine/base/datastructures.py

@@ -1,5 +1,6 @@
 import weakref
 import itertools
+
 from mongoengine.common import _import_class
 from mongoengine.errors import DoesNotExist, MultipleObjectsReturned
 
@@ -20,7 +21,7 @@ def __init__(self, dict_items, instance, name):
         if isinstance(instance, (Document, EmbeddedDocument)):
             self._instance = weakref.proxy(instance)
         self._name = name
-        return super(BaseDict, self).__init__(dict_items)
+        super(BaseDict, self).__init__(dict_items)
 
     def __getitem__(self, key, *args, **kwargs):
         value = super(BaseDict, self).__getitem__(key)
@@ -65,15 +66,15 @@ def __setstate__(self, state):
 
     def clear(self, *args, **kwargs):
         self._mark_as_changed()
-        return super(BaseDict, self).clear(*args, **kwargs)
+        return super(BaseDict, self).clear()
 
     def pop(self, *args, **kwargs):
         self._mark_as_changed()
         return super(BaseDict, self).pop(*args, **kwargs)
 
     def popitem(self, *args, **kwargs):
         self._mark_as_changed()
-        return super(BaseDict, self).popitem(*args, **kwargs)
+        return super(BaseDict, self).popitem()
 
     def setdefault(self, *args, **kwargs):
         self._mark_as_changed()
@@ -189,7 +190,7 @@ def remove(self, *args, **kwargs):
 
     def reverse(self, *args, **kwargs):
         self._mark_as_changed()
-        return super(BaseList, self).reverse(*args, **kwargs)
+        return super(BaseList, self).reverse()
 
     def sort(self, *args, **kwargs):
         self._mark_as_changed()
@@ -368,45 +369,61 @@ class StrictDict(object):
     __slots__ = ()
     _special_fields = set(['get', 'pop', 'iteritems', 'items', 'keys', 'create'])
     _classes = {}
+
     def __init__(self, **kwargs):
-        for k,v in kwargs.iteritems():
+        for k, v in kwargs.iteritems():
             setattr(self, k, v)
+
     def __getitem__(self, key):
         key = '_reserved_' + key if key in self._special_fields else key
         try:
             return getattr(self, key)
         except AttributeError:
             raise KeyError(key)
+
     def __setitem__(self, key, value):
         key = '_reserved_' + key if key in self._special_fields else key
         return setattr(self, key, value)
+
     def __contains__(self, key):
         return hasattr(self, key)
+
     def get(self, key, default=None):
         try:
             return self[key]
         except KeyError:
             return default
+
     def pop(self, key, default=None):
         v = self.get(key, default)
         try:
             delattr(self, key)
         except AttributeError:
             pass
         return v
+
     def iteritems(self):
         for key in self:
             yield key, self[key]
+
     def items(self):
         return [(k, self[k]) for k in iter(self)]
+
+    def iterkeys(self):
+        return iter(self)
+
     def keys(self):
         return list(iter(self))
+
     def __iter__(self):
         return (key for key in self.__slots__ if hasattr(self, key))
+
     def __len__(self):
         return len(list(self.iteritems()))
+
     def __eq__(self, other):
         return self.items() == other.items()
+
     def __neq__(self, other):
         return self.items() != other.items()
 
@@ -417,15 +434,18 @@ def create(cls, allowed_keys):
         if allowed_keys not in cls._classes:
             class SpecificStrictDict(cls):
                 __slots__ = allowed_keys_tuple
+
                 def __repr__(self):
-                    return "{%s}" % ', '.join('"{0!s}": {0!r}'.format(k,v) for (k,v) in self.iteritems())
+                    return "{%s}" % ', '.join('"{0!s}": {0!r}'.format(k) for k in self.iterkeys())
+
             cls._classes[allowed_keys] = SpecificStrictDict
         return cls._classes[allowed_keys]
 
 
 class SemiStrictDict(StrictDict):
-    __slots__ = ('_extras')
+    __slots__ = ('_extras', )
     _classes = {}
+
     def __getattr__(self, attr):
         try:
             super(SemiStrictDict, self).__getattr__(attr)
@@ -434,6 +454,7 @@ def __getattr__(self, attr):
                 return self.__getattribute__('_extras')[attr]
             except KeyError as e:
                 raise AttributeError(e)
+
     def __setattr__(self, attr, value):
         try:
             super(SemiStrictDict, self).__setattr__(attr, value)

mongoengine/base/document.py

@@ -14,7 +14,6 @@
 from mongoengine.errors import (ValidationError, InvalidDocumentError,
                                 LookUpError, FieldDoesNotExist)
 from mongoengine.python_support import PY3, txt_type
-
 from mongoengine.base.common import get_document, ALLOW_INHERITANCE
 from mongoengine.base.datastructures import (
     BaseDict,
@@ -150,7 +149,6 @@ def __setattr__(self, name, value):
         # Handle dynamic data only if an initialised dynamic document
         if self._dynamic and not self._dynamic_lock:
 
-            field = None
             if not hasattr(self, name) and not name.startswith('_'):
                 DynamicField = _import_class("DynamicField")
                 field = DynamicField(db_field=name)
@@ -183,8 +181,8 @@ def __setattr__(self, name, value):
         except AttributeError:
             self__initialised = False
         # Check if the user has created a new instance of a class
-        if (self._is_document and self__initialised
-                and self__created and name == self._meta.get('id_field')):
+        if (self._is_document and self__initialised and
+                self__created and name == self._meta.get('id_field')):
             super(BaseDocument, self).__setattr__('_created', False)
 
         super(BaseDocument, self).__setattr__(name, value)
@@ -328,7 +326,7 @@ def to_mongo(self, use_db_field=True, fields=None):
 
             if value is not None:
 
-                if isinstance(field, (EmbeddedDocumentField)):
+                if isinstance(field, EmbeddedDocumentField):
                     if fields:
                         key = '%s.' % field_name
                         embedded_fields = [
@@ -417,10 +415,11 @@ def validate(self, clean=True):
 
     def to_json(self, *args, **kwargs):
         """Converts a document to JSON.
-        :param use_db_field: Set to True by default but enables the output of the json structure with the field names and not the mongodb store db_names in case of set to False
+        :param use_db_field: Set to True by default but enables the output of the json structure with the field names
+            and not the mongodb store db_names in case of set to False
         """
         use_db_field = kwargs.pop('use_db_field', True)
-        return json_util.dumps(self.to_mongo(use_db_field),  *args, **kwargs)
+        return json_util.dumps(self.to_mongo(use_db_field), *args, **kwargs)
 
     @classmethod
     def from_json(cls, json_data, created=False):
@@ -570,15 +569,15 @@ def _get_changed_fields(self, inspected=None):
                 continue
             elif (isinstance(data, (EmbeddedDocument, DynamicEmbeddedDocument))
                   and db_field_name not in changed_fields):
-                 # Find all embedded fields that have been changed
+                # Find all embedded fields that have been changed
                 changed = data._get_changed_fields(inspected)
                 changed_fields += ["%s%s" % (key, k) for k in changed if k]
             elif (isinstance(data, (list, tuple, dict)) and
                     db_field_name not in changed_fields):
                 if (hasattr(field, 'field') and
                         isinstance(field.field, ReferenceField)):
                     continue
-                elif (isinstance(field, SortedListField) and field._ordering):
+                elif isinstance(field, SortedListField) and field._ordering:
                     # if ordering is affected whole list is changed
                     if any(map(lambda d: field._ordering in d._changed_fields, data)):
                         changed_fields.append(db_field_name)
@@ -621,18 +620,18 @@ def _delta(self):
         else:
             set_data = doc
             if '_id' in set_data:
-                del(set_data['_id'])
+                del set_data['_id']
 
         # Determine if any changed items were actually unset.
         for path, value in set_data.items():
             if value or isinstance(value, (numbers.Number, bool)):
                 continue
 
-            # If we've set a value that ain't the default value dont unset it.
+            # If we've set a value that ain't the default value don't unset it.
             default = None
             if (self._dynamic and len(parts) and parts[0] in
                     self._dynamic_fields):
-                del(set_data[path])
+                del set_data[path]
                 unset_data[path] = 1
                 continue
             elif path in self._fields:
@@ -666,7 +665,7 @@ def _delta(self):
             if default != value:
                 continue
 
-            del(set_data[path])
+            del set_data[path]
             unset_data[path] = 1
         return set_data, unset_data
 
@@ -821,7 +820,6 @@ def _build_index_spec(cls, spec):
             parts = key.split('.')
             if parts in (['pk'], ['id'], ['_id']):
                 key = '_id'
-                fields = []
             else:
                 fields = cls._lookup_field(parts)
                 parts = []
@@ -981,7 +979,7 @@ def _lookup_field(cls, parts):
                 if hasattr(getattr(field, 'field', None), 'lookup_member'):
                     new_field = field.field.lookup_member(field_name)
                 elif cls._dynamic and (isinstance(field, DynamicField) or
-                                           getattr(getattr(field, 'document_type'), '_dynamic')):
+                                       getattr(getattr(field, 'document_type'), '_dynamic')):
                     new_field = DynamicField(db_field=field_name)
                 else:
                     # Look up subfield on the previous field or raise