Audit: all models are being tracked by PG history (#14)

* install pghistory

* add created/modified/is_removed default at database level by SQL

* attendee model now have history tracked by db triggers

* add table comment to remove pk/id column which is pgh_obj_id

* id still required for updating model instance

* revert back to models.UUIDField on model itself

* revert history table name back to event to match pghistory's method name

* resolve conflics

* fix partial date saving error from Django Admin

* update django address and use bigint on it

* add table checking step for manage task update_content_types

* draft of user history

* add user_groups and user_permissions event tables

* add a test to limit model name length for pg-trigger names

* add ccontext for celery task and access csv importer

* add my own django-db-comments for labeling database tables

* upgrade version of db-comment

* user history diff in admin successfully shows

* add history view for attendee, style buttons and show user id in history list view without click

* add auth.Group in pg history

* tracking group_permissions by pg history

* track schedule Event model by PG history

* update django pghistory version, tack event relation in History name

* change event's event to event's history

* rename migration file to match model name

* rename GroupPermissionsEvent to GroupPermissionsHistory

* rename GroupEvent to GroupHistory

* rename UserPermissionEvent to UserPermissionHistory

* rename GroupPermissionsHistory to GroupPermissionHistory

* rename 0007_grouphistory to 0007_group_history

* rename UserGroupEvent to UserGroupHistory

* change UserEvent to UserHistory

* rename AttendeesEvent to AttendeeHistory for consistancy

* rename AttendeeHistory back to AttendeesHistory to match attendees table name

* rename migration file to make it easier to read

* add table comment seems overwritten by django-db-comments

* rename UserGroupHistory to UserGroupsHistory for consistancy

* UserHistory cannot be UsersHistory due to original table was created as users_user in Django Cookiecutter

* track category by pghistory now

* now track Note by pghistory

* track Past with pg history now

* track Past with pg history now with Admin

* fix past data issues, may break again along adding pg history models

* now tracking schedule's Calendar wth p history

* schedule calendar relation model now tracked by pghistory too

* track account's email address by pghistory

* track email confirmation by pg history. Currently there's no records due to sendgrid, but email service may switch in the future

* divert to upgrade django address

* tracked address country by pghistory

* now tracked state by pghistory

* tracked locality history by pghistory now

* tracked address by pghistory, noticed that django address id in history are still IntegerField instead of BigIntegerField

* tracked menu by pghistory

* track menu_auth_group by django pghistory now

* now tracking organization by django pghistory

* tracking division by django pghistory

* add campus to be tracked by pghistory

* track property with PG history now

* track SuitesHistory by PG history now

* track room by PG history now

* fix room migration

* track place by PG history

* make room object id to uuid compatible, also applied permissions on most of occasions models

* make room object id to uuid compatible, also applied permissions on most of occasions models

* tracked MessageTemplate by PG history now

* changing primary id to BigIntegerField in history model

* remove Assembly model column 'need_age' and track its history by PG history

* tracked Price model by PG history now

* tracked character model by PG history now

* tracked meet model with PG history

* track model gathering with PG history

* making pgh_id BigAutoField instead of AutoField

* now tracked Team model by PG history

* tracking attendance with PG history now

* tracked Relations by PG history, also make pk index at history db level (Past)

* move index on pk to the migration files instead of raw SQL unless necessary

* tracked Folk model by PG history, also make attenee uuid indexed

* tracking registration by PG history and change its assembly to non-null

* add price to Attending model and tracked by PG history

* add price to Attending model and tracked by PG history

* track AttendingMeet by PG history now

* tracked FolkAttendeesHistory by PG history finally

* update seed data to match content type# after PG history

* fix add more contact button position

* update PR info

* clean up some model def

README.md

@@ -231,8 +231,10 @@ DJANGO_DEFAULT_FROM_EMAIL=fake@email.com
 </details>
 
 ## [How to start dev env on Linux](https://cookiecutter-django.readthedocs.io/en/latest/developing-locally-docker.html)
+
 <details>
   <summary>Click to expand all</summary>
+
 * double check if the dev port 8008 is open on fire wall
 * add server's public ip to ALLOWED_HOSTS in settings
 * install docker and docker-compose, such as `sudo apt  install docker docker-compose`
@@ -252,6 +254,7 @@ DJANGO_DEFAULT_FROM_EMAIL=fake@email.com
 * create 2 superusers by `docker-compose -f local.yml run django python manage.py createsuperuser`
 * import the seed data by `docker-compose -f local.yml run django python manage.py loaddata fixtures/db_seed`
 * go to Django admin to add the first organization and all groups to the first user (superuser) at http://<<your domain name>>:8008/admin/users/user/
+
 </details>
 
 ## [How to start dev env on Windows](https://cookiecutter-django.readthedocs.io/en/latest/developing-locally-docker.html)
@@ -305,7 +308,7 @@ DJANGO_DEFAULT_FROM_EMAIL=fake@email.com
 * upadte content types after migration by `docker-compose -f local.yml run django python manage.py update_content_types`
 * create 2 superusers by `docker-compose -f local.yml run --rm django python manage.py createsuperuser`
 * import the seed data by `docker-compose -f local.yml run django python manage.py loaddata fixtures/db_seed`
-  (data were created by `docker-compose -f local.yml run django python manage.py dumpdata --exclude users.user --exclude admin.logentry --exclude sessions.session --exclude contenttypes.contenttype --exclude sites.site --exclude account.emailaddress --exclude account.emailconfirmation --exclude socialaccount.socialtoken --exclude auth.permission --indent 2 > fixtures/db_seed2.json`)
+  (data were created by `docker-compose -f local.yml run django python manage.py dumpdata --exclude users.user --exclude admin.logentry --exclude sessions.session --exclude contenttypes.contenttype --exclude sites.site --exclude account.emailaddress --exclude account.emailconfirmation --exclude socialaccount.socialtoken --exclude auth.permission --exclude pghistory.context --exclude pghistory.aggregateevent --indent 2 > fixtures/db_seed2.json`)
 * go to Django admin to add the first organization and all groups to the first user (superuser) at http://192.168.99.100:8008/admin123/users/user/
 * use browser to open http://192.168.99.100:8008/ and http://192.168.99.100:8025/
 * Enter postgres db console by `docker-compose -f local.yml exec postgres psql --username=YBIJMKerEaNYKqzfvMxOlBAesdyiahxk attendees_development`
@@ -388,9 +391,9 @@ All libraries are included to facilitate offline development, it will take port
 - [ ] Division specific menu links, such as including selected meets in the search params
   - [ ] Junior
   - [ ] Data
-- [ ] Audit log/history/vision of data
-  - [ ] find library and install: maybe django-pghistory with AggregateEvent
-  - [ ] each model level version
+- [x] [2PR#14](https://github.com/xjlin0/attendees32/pull/14) Audit log/history/vision of data
+  - [x] find library and install: maybe django-pghistory with AggregateEvent
+  - [x] each model level version
   - [ ] document aggregation level version
 - [x] [New repo] upgrade to Django 3.2LTS for support of DEFAULT_AUTO_FIELD
    -[x] accept partial date on all attending/past, etc django-date-extensions or django_partial_date. Also use Javascript solution to make yearless date back to 1800, so birthday of "1999 August" will be 08-01-1999 and "May 24th" will be 05-24-1800

attendees/occasions/admin.py

@@ -1,9 +1,11 @@
-
+from django.contrib import messages
 from django.contrib import admin
 from django.contrib.postgres import fields
 
 from django_json_widget.widgets import JSONEditorWidget
 from attendees.occasions.models import Attendance, MessageTemplate, Assembly, Price, Character, Meet, Gathering, Team
+from attendees.persons.models import PgHistoryPage
+from attendees.whereabouts.models import Organization, Division
 
 
 # Register your models here.
@@ -19,7 +21,7 @@
 #     extra = 0
 
 
-class MessageTemplateAdmin(admin.ModelAdmin):
+class MessageTemplateAdmin(PgHistoryPage, admin.ModelAdmin):
     formfield_overrides = {
         fields.JSONField: {"widget": JSONEditorWidget},
     }
@@ -29,8 +31,25 @@ class MessageTemplateAdmin(admin.ModelAdmin):
     def truncate_template(self, obj):
         return list(obj.templates.values())[0][:100] + "..."
 
-
-class AssemblyAdmin(admin.ModelAdmin):
+    def formfield_for_foreignkey(self, db_field, request, **kwargs):
+        if db_field.name == "organization":
+            kwargs["queryset"] = Organization.objects.all() if request.user.is_superuser else Organization.objects.filter(pk=request.user.organization_pk())
+        return super().formfield_for_foreignkey(db_field, request, **kwargs)
+
+    def get_queryset(self, request):
+        qs = super().get_queryset(request)
+        if request.user.is_superuser:
+            return qs
+        else:
+            if request.resolver_match.func.__name__ == "changelist_view":
+                messages.warning(
+                    request,
+                    "Not all, but only those records accessible to you will be listed here.",
+                )
+            return qs.filter(organization=request.user.organization)
+
+
+class AssemblyAdmin(PgHistoryPage, admin.ModelAdmin):
     formfield_overrides = {
         fields.JSONField: {"widget": JSONEditorWidget},
     }
@@ -40,12 +59,46 @@ class AssemblyAdmin(admin.ModelAdmin):
     list_display = ("id", "division", "display_name", "slug", "get_addresses")
     readonly_fields = ["id", "created", "modified"]
 
-
-class PriceAdmin(admin.ModelAdmin):
+    def formfield_for_foreignkey(self, db_field, request, **kwargs):
+        if db_field.name == "division":
+            kwargs["queryset"] = Division.objects.all() if request.user.is_superuser else Division.objects.filter(organization=request.user.organization)
+        return super().formfield_for_foreignkey(db_field, request, **kwargs)
+
+    def get_queryset(self, request):
+        qs = super().get_queryset(request)
+        if request.user.is_superuser:
+            return qs
+        else:
+            if request.resolver_match.func.__name__ == "changelist_view":
+                messages.warning(
+                    request,
+                    "Not all, but only those records accessible to you will be listed here.",
+                )
+            return qs.filter(division__organization=request.user.organization)
+
+
+class PriceAdmin(PgHistoryPage, admin.ModelAdmin):
     list_display = ("display_name", "price_type", "start", "price_value", "modified")
 
-
-class AttendanceAdmin(admin.ModelAdmin):
+    def formfield_for_foreignkey(self, db_field, request, **kwargs):
+        if db_field.name == "assembly":
+            kwargs["queryset"] = Assembly.objects.all() if request.user.is_superuser else Assembly.objects.filter(division__organization=request.user.organization)
+        return super().formfield_for_foreignkey(db_field, request, **kwargs)
+
+    def get_queryset(self, request):
+        qs = super().get_queryset(request)
+        if request.user.is_superuser:
+            return qs
+        else:
+            if request.resolver_match.func.__name__ == "changelist_view":
+                messages.warning(
+                    request,
+                    "Not all, but only those records accessible to you will be listed here.",
+                )
+            return qs.filter(assembly__division__organization=request.user.organization)
+
+
+class AttendanceAdmin(PgHistoryPage, admin.ModelAdmin):
     list_display_links = ("get_attendee",)
     list_filter = (
         ("gathering", admin.RelatedOnlyFieldListFilter),
@@ -77,6 +130,23 @@ class AttendanceAdmin(admin.ModelAdmin):
         ),
     )
 
+    def formfield_for_foreignkey(self, db_field, request, **kwargs):
+        if db_field.name == "gathering":
+            kwargs["queryset"] = Gathering.objects.all() if request.user.is_superuser else Gathering.objects.filter(meet__assembly__division__organization=request.user.organization)
+        return super().formfield_for_foreignkey(db_field, request, **kwargs)
+
+    def get_queryset(self, request):
+        qs = super().get_queryset(request)
+        if request.user.is_superuser:
+            return qs
+        else:
+            if request.resolver_match.func.__name__ == "changelist_view":
+                messages.warning(
+                    request,
+                    "Not all, but only those records accessible to you will be listed here.",
+                )
+            return qs.filter(gathering__meet__assembly__division__organization=request.user.organization)
+
     def get_attendee(self, obj):
         return obj.attending.attendee
 
@@ -109,26 +179,60 @@ class AttendanceInline(admin.StackedInline):
     #     return qs[:10]  # does not work
 
 
-class CharacterAdmin(admin.ModelAdmin):
+class CharacterAdmin(PgHistoryPage, admin.ModelAdmin):
     prepopulated_fields = {"slug": ("display_name",)}
     list_filter = ("assembly", "type")
     readonly_fields = ["id", "created", "modified"]
     list_display_links = ("display_name",)
     list_display = ("id", "assembly", "display_name", "slug", "display_order", "type")
 
+    def formfield_for_foreignkey(self, db_field, request, **kwargs):
+        if db_field.name == "assembly":
+            kwargs["queryset"] = Assembly.objects.all() if request.user.is_superuser else Assembly.objects.filter(division__organization=request.user.organization)
+        return super().formfield_for_foreignkey(db_field, request, **kwargs)
+
+    def get_queryset(self, request):
+        qs = super().get_queryset(request)
+        if request.user.is_superuser:
+            return qs
+        else:
+            if request.resolver_match.func.__name__ == "changelist_view":
+                messages.warning(
+                    request,
+                    "Not all, but only those records accessible to you will be listed here.",
+                )
+            return qs.filter(assembly__division__organization=request.user.organization)
+
     class Media:
         css = {"all": ("css/admin.css",)}
         js = ["js/admin/list_filter_collapse.js"]
 
 
-class TeamAdmin(admin.ModelAdmin):
+class TeamAdmin(PgHistoryPage, admin.ModelAdmin):
     prepopulated_fields = {"slug": ("display_name",)}
     readonly_fields = ["id", "created", "modified"]
     list_display_links = ("display_name",)
     list_display = ("id", "display_name", "slug", "meet", "display_order", "modified")
 
-
-class MeetAdmin(admin.ModelAdmin):
+    def formfield_for_foreignkey(self, db_field, request, **kwargs):
+        if db_field.name == "meet":
+            kwargs["queryset"] = Meet.objects.all() if request.user.is_superuser else Meet.objects.filter(assembly__division__organization=request.user.organization)
+        return super().formfield_for_foreignkey(db_field, request, **kwargs)
+
+    def get_queryset(self, request):
+        qs = super().get_queryset(request)
+        if request.user.is_superuser:
+            return qs
+        else:
+            if request.resolver_match.func.__name__ == "changelist_view":
+                messages.warning(
+                    request,
+                    "Not all, but only those records accessible to you will be listed here.",
+                )
+            return qs.filter(meet__assembly__division__organization=request.user.organization)
+
+
+class MeetAdmin(PgHistoryPage, admin.ModelAdmin):
     formfield_overrides = {
         fields.JSONField: {"widget": JSONEditorWidget},
     }
@@ -161,12 +265,29 @@ class MeetAdmin(admin.ModelAdmin):
         ),
     )
 
+    def formfield_for_foreignkey(self, db_field, request, **kwargs):
+        if db_field.name == "assembly":
+            kwargs["queryset"] = Assembly.objects.all() if request.user.is_superuser else Assembly.objects.filter(division__organization=request.user.organization)
+        return super().formfield_for_foreignkey(db_field, request, **kwargs)
+
+    def get_queryset(self, request):
+        qs = super().get_queryset(request)
+        if request.user.is_superuser:
+            return qs
+        else:
+            if request.resolver_match.func.__name__ == "changelist_view":
+                messages.warning(
+                    request,
+                    "Not all, but only those records accessible to you will be listed here.",
+                )
+            return qs.filter(assembly__division__organization=request.user.organization)
+
     class Media:
         css = {"all": ("css/admin.css",)}
         js = ["js/admin/list_filter_collapse.js"]
 
 
-class GatheringAdmin(admin.ModelAdmin):
+class GatheringAdmin(PgHistoryPage, admin.ModelAdmin):
     formfield_overrides = {
         fields.JSONField: {"widget": JSONEditorWidget},
     }
@@ -190,6 +311,23 @@ class GatheringAdmin(admin.ModelAdmin):
         ),
     )
 
+    def formfield_for_foreignkey(self, db_field, request, **kwargs):
+        if db_field.name == "meet":
+            kwargs["queryset"] = Meet.objects.all() if request.user.is_superuser else Meet.objects.filter(assembly__division__organization=request.user.organization)
+        return super().formfield_for_foreignkey(db_field, request, **kwargs)
+
+    def get_queryset(self, request):
+        qs = super().get_queryset(request)
+        if request.user.is_superuser:
+            return qs
+        else:
+            if request.resolver_match.func.__name__ == "changelist_view":
+                messages.warning(
+                    request,
+                    "Not all, but only those records accessible to you will be listed here.",
+                )
+            return qs.filter(meet__assembly__division__organization=request.user.organization)
+
     class Media:
         css = {"all": ("css/admin.css",)}
         js = ["js/admin/list_filter_collapse.js"]

attendees/occasions/apps.py

@@ -1,5 +1,41 @@
+import pghistory
 from django.apps import AppConfig
+from django.apps import apps as django_apps
 
 
 class OccasionsConfig(AppConfig):
     name = "attendees.occasions"
+
+    def ready(self):
+        schedule_calendar_model = django_apps.get_model("schedule.Calendar", require_ready=False)
+        schedule_calendarrelation_model = django_apps.get_model("schedule.CalendarRelation", require_ready=False)
+        schedule_event_model = django_apps.get_model("schedule.Event", require_ready=False)
+        schedule_eventrelation_model = django_apps.get_model("schedule.EventRelation", require_ready=False)
+
+        pghistory.track(
+            pghistory.Snapshot('calendar.snapshot'),
+            related_name='history',
+            model_name='CalendarHistory',
+            app_label='occasions'
+        )(schedule_calendar_model)
+
+        pghistory.track(
+            pghistory.Snapshot('calendarrelation.snapshot'),
+            related_name='history',
+            model_name='CalendarRelationHistory',
+            app_label='occasions',
+        )(schedule_calendarrelation_model)
+
+        pghistory.track(
+            pghistory.Snapshot('event.snapshot'),
+            related_name='history',
+            model_name='EventHistory',
+            app_label='occasions'
+        )(schedule_event_model)
+
+        pghistory.track(
+            pghistory.Snapshot('eventrelation.snapshot'),
+            related_name='history',
+            model_name='EventRelationHistory',
+            app_label='occasions',
+        )(schedule_eventrelation_model)

attendees/occasions/migrations/0001_message_template.py

@@ -1,6 +1,6 @@
 # Generated by Django 3.0.14 on 2021-08-28 17:55
 
-# from django.contrib.postgres.fields.jsonb import JSONField
+from attendees.persons.models.utility import Utility
 from django.db import migrations, models
 import django.utils.timezone
 import model_utils.fields
@@ -31,8 +31,31 @@ class Migration(migrations.Migration):
                 'db_table': 'occasions_message_templates',
             },
         ),
+        migrations.RunSQL(Utility.default_sql('occasions_message_templates')),
         migrations.AddConstraint(
             model_name='messagetemplate',
             constraint=models.UniqueConstraint(condition=models.Q(is_removed=False), fields=('organization', 'type'), name='organization_type'),
         ),
+        migrations.CreateModel(
+            name='MessageTemplatesHistory',
+            fields=[
+                ('pgh_id', models.BigAutoField(primary_key=True, serialize=False)),
+                ('pgh_created_at', models.DateTimeField(auto_now_add=True)),
+                ('pgh_label', models.TextField(help_text='The event label.')),
+                ('pgh_obj', models.ForeignKey(db_constraint=False, on_delete=models.deletion.DO_NOTHING, related_name='history', to='occasions.messagetemplate')),
+                ('id', models.BigIntegerField(db_index=True)),
+                ('created', model_utils.fields.AutoCreatedField(default=django.utils.timezone.now, editable=False, verbose_name='created')),
+                ('modified', model_utils.fields.AutoLastModifiedField(default=django.utils.timezone.now, editable=False, verbose_name='modified')),
+                ('is_removed', models.BooleanField(default=False)),
+                ('organization', models.ForeignKey(db_constraint=False, on_delete=models.deletion.DO_NOTHING, related_name='+', related_query_name='+', to='whereabouts.organization')),
+                ('templates', models.JSONField(blank=True, default=dict, help_text='Example: {"body": "Dear {name}: Hello!"}. Whatever in curly braces will be interpolated by variables, Please keep {} here even no data', null=True)),
+                ('defaults', models.JSONField(blank=True, default=dict, help_text='Example: {"name": "John", "Date": "08/31/2020"}. Please keep {} here even no data', null=True)),
+                ('type', models.SlugField(db_index=False, help_text='format: Organization_slug-prefix-message-type-name')),
+                ('pgh_context', models.ForeignKey(db_constraint=False, null=True, on_delete=models.deletion.DO_NOTHING, related_name='+', to='pghistory.context')),
+            ],
+            options={
+                'db_table': 'occasions_message_templateshistory',
+            },
+        ),
+        migrations.RunSQL(Utility.pgh_default_sql('occasions_message_templateshistory', original_model_table='occasions_message_templates')),
     ]