Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swappable models #389

Closed
wants to merge 22 commits into from
Closed

Swappable models #389

wants to merge 22 commits into from

Conversation

m-vdb
Copy link
Contributor

@m-vdb m-vdb commented Jan 10, 2018

Fixes #306

This PR aims at proposing a powerful replacement for base classes that were removed in #377.
⚠️ Do not merge for now, we need to test the version change on live projects

Changes summary:

  • split models between abstract models and actual models
  • define settings for swappable models + getters (à la get_user_model())
  • add Django migrations (❗️this point needs to be validated and tested with live projects)
  • add tests for the whole thing

@jdufresne
Copy link
Collaborator

Do you mind cleaning up the lint errors so the Travis tests will run?

Do we really need a to allow swapping all models? Or would handling just Event be sufficient? I'd prefer not to add additional complexity if it isn't handling a real use case.

@jdufresne jdufresne self-requested a review January 10, 2018 13:35
@m-vdb
Copy link
Contributor Author

m-vdb commented Jan 10, 2018

Lint should be good after 581d0c9 and 3debd40. About swapping all the models: in my use cases, I have custom models for Event and Occurrence, but not for Calendar. I would argue that it is pretty useful to have this functionality for the occurrences as well. The swappability for the Calendar is not a must have in my opinion, but in some cases developers could have several calendar per user (for instance) and would like to enrich the calendar model. Since it was the same amount of work I decided to do it, but I can easily revert that.

I still need to figure out how to fix tests + migrations, but what do you think about this PR?

@llazzaro
Copy link
Owner

@m-vdb don't upgrade to 0.8.7 or do it in test server. we just removed this feature in 0.8.7

@llazzaro
Copy link
Owner

@jdufresne Let review this change, we could be breaking other people code.
Let me know what you think about this PR.
I will try to make some time at night to review this PR

@jdufresne
Copy link
Collaborator

I'll try to find some time to give this a deeper review.

@keeth
Copy link
Contributor

keeth commented Jan 12, 2018

I'm very interested in this as I use base classes heavily, and I find it hard to maintain.

@jdufresne
Copy link
Collaborator

@m-vdb Can you rebase this PR so Travis runs against the latest mater? I'd like to see CI green before the review continues.

@m-vdb
Copy link
Contributor Author

m-vdb commented Jan 12, 2018

my branch starts from develop, isn't it the way to go? Like I said in a previous comment, I still need to figure out how to make my test pass and test that in a live project. I am not sure about the way I handled the migrations.

@jdufresne
Copy link
Collaborator

Oh you're right. Sorry, I misspoke. I intended to ask to rebase on develop. When you have a chance, can you do that and squash the commits? It would help keep the change tidy and aid reviewing.

Nice, I think that the general approach looks good! I think the use of swappable will be easier to maintain than the previous base classes.

I'd like to see more tests and have them passing before this is merge ready. The tests go a long way towards allowing me to analyze the change. Let me know when that is ready and I'll continue the review. It might help to look at how Django tests the swappable User model as guidance.

Nice work!

except ImportError:
raise RuntimeError(self.__doc__.strip())

@override_settings(SCHEDULER_CALENDAR_MODEL='schedule.Calendar', SCHEDULER_EVENT_MODEL='schedule.Event')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this overridden to not be the alternate model? Shouldn't the new tests use the CustomEvent class?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had an error with it but it might be related to my migration issue

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah I know why:

  • the tests are executed within the context of the tests project, using the tests.settings module
  • the migrations is meaningful only in the context of the tests.test_project, using the tests.test_project.settings
  • as a consequence, importing the migration file like I do in the tests results in an import error: the django.conf.settings.SCHEDULER_CALENDAR_MODEL isn't defined for tests.settings but it is defined for tests.test_project.settings

.travis.yml Outdated
@@ -27,6 +27,7 @@ before_script:
- isort --check-only --diff
- python -m django check --settings=tests.settings
- python -m django makemigrations --settings=tests.settings --no-input --dry-run --check
- python -m django makemigrations test_project --settings=tests.test_project.settings
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels a bit odd to me on the surface. Could this instead be a part of running the test? I'd like to avoid an extra test setup step if possible so that the tests are immediately runnable as before.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I'll see what I can do, I just wanted to avoid executing a script within the tests. but maybe I can just use call_command(), will try.

@m-vdb m-vdb force-pushed the mvdb/swappable-models branch 2 times, most recently from aa42c40 to 76aea1e Compare January 15, 2018 10:51
@jdufresne
Copy link
Collaborator

To resolve conflicts, do you mind rebasing instead of merging? It would make the commit history much cleaner and easier to review. Thanks.

@m-vdb m-vdb force-pushed the mvdb/swappable-models branch from c7d09d3 to 4002ffe Compare January 16, 2018 06:50
@m-vdb
Copy link
Contributor Author

m-vdb commented Jan 16, 2018

done, sorry I missed that earlier. Here is where I am stuck: you can see that one of the tests that I introduced isn't passing. It is because I still haven't figured out how to migrate from the current version of django-scheduler to this one. Let me explain: if a user is under django-scheduler==0.8.8 and uses BASE_CLASSES and wants to switch to this new version, he will need to swap the models. And here comes the migration issue (cf. backtraces on travis).

Also, I realized something interesting. Once you define the swapped models, there are two migrations that are generated: one in the schedule module, and one in the test_project module.

I've tried to search Django documentation and other projects GH issues and I've found similar issues, but I still don't know exactly how to fix that:

@m-vdb
Copy link
Contributor Author

m-vdb commented Jan 16, 2018

I've managed to make it somehow work, by following django-oauth-toolkit documentation. What users would have to do:

  • define a custom model without extending the abstract class
  • generate the migration
  • define the swapped settings and make the custom model inherit from the abstract class
  • generate a second migration: this one is a bit problematic because Django sees that you are adding new fields to the custom models, and some fields are defined non-nullable on the abstract classes, so it will prompt the user to provide one-off defaults. It's a small detail but in terms of UX it is a bit distressing.

Ideally, I think we would like to create the custom models as subclasses of the abstract models from the beginning, and generate a migration there, but this fails because of the following:

ERRORS:
schedule.Event.creator: (fields.E304) Reverse accessor for 'Event.creator' clashes with reverse accessor for 'CustomEvent.creator'.
	HINT: Add or change a related_name argument to the definition for 'Event.creator' or 'CustomEvent.creator'.
schedule.Event.creator: (fields.E305) Reverse query name for 'Event.creator' clashes with reverse query name for 'CustomEvent.creator'.
	HINT: Add or change a related_name argument to the definition for 'Event.creator' or 'CustomEvent.creator'.
test_project.CustomEvent.creator: (fields.E304) Reverse accessor for 'CustomEvent.creator' clashes with reverse accessor for 'Event.creator'.
	HINT: Add or change a related_name argument to the definition for 'CustomEvent.creator' or 'Event.creator'.
test_project.CustomEvent.creator: (fields.E305) Reverse query name for 'CustomEvent.creator' clashes with reverse query name for 'Event.creator'.
	HINT: Add or change a related_name argument to the definition for 'CustomEvent.creator' or 'Event.creator'.

It's easily changeable on the models but it might introduce backwards-incompatible changes? Also, I think that defining the related_name as "creator" is wrong, because related name defines the naming convention when we want to access to all events created by a user. By default it is User.events_set.all(), defining this as creator changes that to User.creator.all() which is just confusing. (reference). LMK what you think @jdufresne

@coveralls
Copy link

coveralls commented Jan 16, 2018

Coverage Status

Coverage increased (+0.4%) to 80.473% when pulling 3c0ffa8 on m-vdb:mvdb/swappable-models into 51d36cb on llazzaro:develop.

@coveralls
Copy link

coveralls commented Jan 16, 2018

Coverage Status

Coverage increased (+0.4%) to 80.473% when pulling 1391f25 on m-vdb:mvdb/swappable-models into 51d36cb on llazzaro:develop.

@m-vdb
Copy link
Contributor Author

m-vdb commented Jan 16, 2018

@jdufresne ok I think I'm good for all logical aspects, even for the migration. Tests are green now, can you take a look? Also, I think that this requires documentation and an extended migration guide to this new version; it shouldn't be too hard to write, I can do it before we merge this.

def setUp(self):
super(TestSwappableModels, self).setUp()
# cleanup any possible leftovers before running
self._cleanup()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to call _cleanup both before and after? Shouldn't one be sufficient?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if for some reason the test completely crashes, this makes sure the test "setup" isn't corrupted. I can easily remove, it is just a nice-to-have for the developers of this repo.

@@ -1,7 +1,7 @@
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os

BASE_DIR = os.path.dirname(os.path.dirname(__file__))
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this change required to make the tests work? Can you explain?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see 8c45306 : __file__ is only absolute in python3, so it was never defined right in python2. This had no consequences before because this settings wasn't used throughout the tests but now it is.

@@ -9,3 +9,5 @@
/dist/
*.egg-info/
/docs/_build/
# generated in tests, shouldn't be versionned
tests/test_project/migrations/0001_initial.py
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why this is in gitignore. Shouldn't this exist as a normal file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the whole purpose of the test I wrote is to generate this file on the fly and verify that it is well formatted. If for some reason in the future someone inadvertently alters the shape of the generated file by changing the code of swappable models, then we will know it and the unit tests will break. Versioning this file doesn't make sense because it is generated by the tests. To avoid confusion for this repo's collaborators I chose to add this file to gitignore. If the test ever fails and the cleanup phase of the test fails to execute too, this file won't be visible with git status and a collaborator won't risk versioning it.


BASE_DIR = os.path.dirname(os.path.dirname(__file__))

SECRET_KEY = '_nc9_+ntllx!ptq5tzh96nn^$a#81#auhrdw#8t#+n0#4ig(lz'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per 8343dde, please use an obviously fake secret for tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8fd293d

"""
popen = subprocess.Popen(
[
'python',
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably use sys.executable

Copy link
Contributor Author

@m-vdb m-vdb Jan 21, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1391f25

'--settings=tests.test_project.settings'
],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you're not using the input/output, you can use DEVNULL instead of PIPE.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in self._check_command_return() I do. In case the command exited with a non-zero return code, this method will display the stdout and stderr of the command, see:

https://github.com/m-vdb/django-scheduler/blob/8c45306aec5d2b8b1592f7c327cecfc6bb2558a7/tests/test_swapped_models.py#L52-L69

@m-vdb m-vdb force-pushed the mvdb/swappable-models branch from 9661a38 to 1391f25 Compare January 21, 2018 18:47
@thongly
Copy link

thongly commented Feb 2, 2018

@jdufresne Is this still awaiting anything before a merge?

The base class is key, in my opinion, to address the problem of extensibility while avoiding the problems #377

@thongly
Copy link

thongly commented Feb 3, 2018

UPDATE - This goes away with specifying the settings variables for the SCHEDULER_CALENDAR_CLASS. Perhaps a loud warning is enough to avoid this.

@m-vdb - I am testing a few things with this branch and tried inheriting from the various AbstractBaseClasses.

I noticed that I am getting a reverse accessor name clash. However, if the special related_name syntax is used, this goes away - as per this section in the Django docs

calendar = models.ForeignKey( SCHEDULER_CALENDAR_MODEL, on_delete=models.CASCADE, verbose_name=_("calendar"), related_name='%(class)s' )

@thongly
Copy link

thongly commented Feb 3, 2018

Onto one of the final steps now, and I noticed that one of the migrations seems not to recognize the derived name of the Calendar class from settings.py.

Running migrations:
Applying schedule.0001_initial...Traceback (most recent call last):
File "manage.py", line 15, in
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.6/site-packages/django/core/management/init.py", line 371, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.6/site-packages/django/core/management/init.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 200, in handle
fake_initial=fake_initial,
File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/local/lib/python3.6/site-packages/django/db/migrations/migration.py", line 122, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/usr/local/lib/python3.6/site-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
schema_editor.create_model(model)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/schema.py", line 249, in create_model
definition, extra_params = self.column_sql(model, field)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/schema.py", line 141, in column_sql
db_params = field.db_parameters(connection=self.connection)
File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/related.py", line 966, in db_parameters
return {"type": self.db_type(connection), "check": self.db_check(connection)}
File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/related.py", line 963, in db_type
return self.target_field.rel_db_type(connection=connection)
File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/related.py", line 877, in target_field
return self.foreign_related_fields[0]
File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/related.py", line 634, in foreign_related_fields
return tuple(rhs_field for lhs_field, rhs_field in self.related_fields if rhs_field)
File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/related.py", line 621, in related_fields
self._related_fields = self.resolve_related_fields()
File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/related.py", line 606, in resolve_related_fields
raise ValueError('Related model %r cannot be resolved' % self.remote_field.model)
ValueError: Related model 'booking.Calendar' cannot be resolved

Upon inspection, this was due to the initial migrations for the booking app not having been created yet. The booking app is where the non-abstract Calendar lives.

However, trying to create the initial migrations for the app results in pointing back to the scheduler migrations.

File "/usr/local/lib/python3.6/site-packages/django/db/migrations/operations/fields.py", line 196, in state_forwards
state.models[app_label, self.model_name_lower].fields
KeyError: ('schedule', 'calendarrelation')

@thongly
Copy link

thongly commented Feb 4, 2018

Perhaps splitting up the app might help alleviate the migration issues - as in this PR for vera.

I'm not familiar enough with the swappable migrations, but I'll try to fix based on this -
https://github.com/wq/django-swappable-models#migration-scripts

@@ -16,7 +16,7 @@ def feed_title(self, obj):
return "Upcoming Events for %s" % obj.name

def get_object(self, request, calendar_id):
return Calendar.objects.get(pk=calendar_id)
return get_calendar_model().objects.get(pk=calendar_id)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of calling get_calendar_model() directly 2 times you can add Calendar = get_calendar_model().
General suggestion: I think the code will be cleaner if you use same approach for get_calendar_model() , get_event_model() and get_occurrence_model() for all your changes.


def test_get_calendar_model_swapped_bad_settings(self):
with override_local_settings(SCHEDULER_CALENDAR_MODEL='badsetting'):
with self.assertRaisesMessage(ImproperlyConfigured, "SCHEDULER_CALENDAR_MODEL must be of the form 'app_label.model_name'"):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line too long https://www.python.org/dev/peps/pep-0008/#maximum-line-length
Same for lines: 169, 178, 183, 192, 197

and then swap its models.
"""
settings_path = os.path.join(settings.BASE_DIR, 'tests', 'test_project', 'settings.py')
updated_settings_path = os.path.join(settings.BASE_DIR, 'tests', 'test_project', 'updated_settings.py.tpl')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line to long https://www.python.org/dev/peps/pep-0008/#maximum-line-length
Same for lines: 74, 104, 131, 143

@@ -16,7 +16,7 @@ def feed_title(self, obj):
return "Upcoming Events for %s" % obj.name

def get_object(self, request, calendar_id):
return Calendar.objects.get(pk=calendar_id)
return get_calendar_model().objects.get(pk=calendar_id)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of calling get_calendar_model() directly 2 times you can add Calendar = get_calendar_model().
General suggestion: I think the code will be cleaner if you use same approach for get_calendar_model() , get_event_model() and get_occurrence_model() for all your changes.

occurrence_id = get_kwarg_or_param(request, kwargs, 'occurrence_id')
return Occurrence.objects.filter(pk=occurrence_id).first() if occurrence_id else None
return get_occurrence_model().objects.filter(pk=occurrence_id).first() if occurrence_id else None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line too long https://www.python.org/dev/peps/pep-0008/#maximum-line-length
Same for lines: 117, 127, 224, 238, 243, 257

@@ -218,7 +228,7 @@ class CalendarRelation(models.Model):
may not scale well. If you use this, keep that in mind.
'''

calendar = models.ForeignKey(Calendar, on_delete=models.CASCADE, verbose_name=_("calendar"))
calendar = models.ForeignKey(SCHEDULER_CALENDAR_MODEL, on_delete=models.CASCADE, verbose_name=_("calendar"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


name: the name of the calendar
events: all the events contained within the calendar.
>>> calendar = Calendar(name = 'Test Calendar')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

calendar = Calendar(name='Test Calendar')
Don't use spaces around the = sign when used to indicate a keyword argument or a default parameter value. https://www.python.org/dev/peps/pep-0008/#other-recommendations

@@ -104,46 +104,18 @@ def get_calendars_for_object(self, obj, distinction=''):


@python_2_unicode_compatible
class Calendar(models.Model):
class AbstractCalendar(models.Model):
'''
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's not your line, but could you please replace ''' with """ in docstring?
https://www.python.org/dev/peps/pep-0008/#documentation-strings

@@ -47,10 +47,10 @@ def get_for_object(self, content_object, distinction='', inherit=True):


@python_2_unicode_compatible
class Event(models.Model):
class AbstractEvent(models.Model):
'''
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's not your line, but could you please replace ''' with """ in docstring?
https://www.python.org/dev/peps/pep-0008/#documentation-strings

return event


def get_calendar(event, request, **kwargs):
from schedule.models import Calendar
calendar = None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not your line, but could you please remove it? I think it's pointless.

@llazzaro
Copy link
Owner

llazzaro commented Jun 3, 2018

Thank you everyone for comments and reviews on this pull request.
I will soon try to review this PR.
@jdufresne are you ok with merging this changes? I didn't have time to review them, but if it's ok and you too... I will merge it soon. If not I will comeback here to add more comments

@jdufresne
Copy link
Collaborator

Sorry, @llazzaro. I won't have time in the near future to review this. If it looks good to you, please proceed.

@lggwettmann
Copy link

Is this going to merge any time soon? (Thanks for the great work, everyone!)

@lggwettmann
Copy link

lggwettmann commented Apr 3, 2020

What can I do to move this forward? I need to customize the occurrence model and would prefer to base that on the common project code. :) @llazzaro

@pickyuptruck
Copy link

any updates on this?

@m-vdb
Copy link
Contributor Author

m-vdb commented Apr 29, 2020

I am no longer using django-scheduler, so I think it might be best if someone took over :)

@m-vdb m-vdb closed this Oct 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

review migrations in virtualenv
9 participants