Skip to content

Commit

Permalink
Merge branch 'update-dependencies-jt'
Browse files Browse the repository at this point in the history
This will be release 1.4.

Addresses issues #4, #16, #27, #29, #30, and #40.
  • Loading branch information
jthomale committed Jun 11, 2019
2 parents 03e5b09 + 755dfa9 commit 06d6caa
Show file tree
Hide file tree
Showing 99 changed files with 802,670 additions and 558,823 deletions.
62 changes: 36 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1046,51 +1046,61 @@ file is in `.gitignore` for that reason. There is a
Early in development we implemented a series of tests using the built-in Django
test runner to do some simple sanity-checking to make sure the Django ORM
models for Sierra match the structures actually in the production database.
You can still run these tests after you first install the catalog-api to ensure
the models match your Sierra setup. Systems may differ from institution to
We have since converted these to run via pytest: see
`django/sierra/base/tests/test_database.py`.

When you run the full test suite, as [described below](#unit-tests), these run
against the test Sierra database—which is useful. But, there are times
that you'll want to run these tests against your live database to make sure the
models are accurate. For instance, systems may differ from institution to
institution based on what products you have, so you may end up needing to fork
this project and update the models so they work with your own setup. It may
also be worth running these tests after Sierra upgrades so that you can make
sure there were no changes made to the database that break the models.

If using Docker, run:
If using Docker, run _only_ the database tests using the following:

./docker-compose.sh run --rm live-db-test

./docker-compose.sh run manage-dev test base
If not using Docker, you can use the below command, instead. If applicable,
replace the value of the `--ds` option with whatever your DEV settings file is.

If not using Docker, you can run:
cd <project_root>
pytest --ds=sierra.settings.dev django/sierra/base/tests/test_database.py

cd <project_root>/django/sierra
manage.py test base
Note: Some of these tests may fail simply because the models are generally more
restrictive than the live Sierra database. We are forcing ForeignKey-type
relationships on a lot of fields that don't seem to have actual
database-enforced keys in Sierra. E.g., from what I can gather, `id` fields are
usually proper keys, while `code` fields may not be&mdash;but `code` fields are
frequently used in a foreign-key-like capacity. I think this leads to a lot of
the invalid codes you have in Sierra, where you have a code in a record that
_should_ point to some entry in an administrative table (like a location), but
it doesn't because the administrative table entry was deleted and the record
was never updated. And there are other cases, as well. E.g., a code might use
the string `none` instead of a null value, but there is no corresponding entry
for `none` in the related table. Bib locations use the string `multi` to
indicate that they have multiple locations, but there is no corresponding
`multi` record in the location table. Etc.

Note: If any `*_maps_to_database` tests fail, it indicates that there are
fields on the model that aren't present in the database. These are more
serious (but often easier to fix) than `*_sanity_check` tests, which test
to ensure that relationship fields work properly. In either case, you can
check the models against the SierraDNA documentation and your own database
to see where the problems lie and decide if they're worth trying to fix in
the models. If tests fail on models that are central, like `RecordMetadata`
or `BibRecord`, then it's a problem. If tests fail on models that are more
peripheral, like `LocationChange`, then finding and fixing those problems
may be less of a priority, especially if you never intend to use those
models in your API.
Ultimately, even though these `code` relationships aren't database-enforced
keys, we do still want the ORM to handle the relationships for us in the
general case where you _can_ match a code with the entry that describes it.
Otherwise we'd have to do the matching manually, which would somewhat reduce
the utility of the ORM.

Presumably because the Sierra data that customers have access to is
implemented in views instead of tables, proper data integrity is lacking in
a few cases. (E.g., the views sometimes don't implement proper primary key
/ foreign key relationships.) This can cause `*_sanity_check` tests to fail
in practice on live data when they should work in theory.

### <a name="unit-tests"></a>Running Unit(ish) Tests

More recently we've been working on adding unit (and integration) tests that
run with py.test. We recommend running these tests via Docker, but it *is*
run with pytest. We recommend running these tests via Docker, but it *is*
possible to run them outside of Docker if you're motivated enough.

If you followed the [Docker setup](#installation-docker),

./docker-compose.sh run --rm test

will run all available py.test tests.
will run all available pytest tests.

If you didn't follow the Docker setup, then you should still be able to create
a comparable test environment:
Expand All @@ -1108,7 +1118,7 @@ relevant connection details.

Spin up all of the needed test databases, and then run

py.test
pytest


### <a name="testing-exports"></a>Testing Sierra Exports Manually
Expand Down
164 changes: 0 additions & 164 deletions django/sierra/api/__init__.py
Original file line number Diff line number Diff line change
@@ -1,164 +0,0 @@
import csv
import re

from .models import APIUser
from django.contrib.auth.models import User


class APIUserManagerError(Exception):
pass


class UserExists(APIUserManagerError):
pass


class UserDoesNotExist(APIUserManagerError):
pass


def get_api_user(username):
api_user = None
try:
api_user = APIUser.objects.get(user__username=username)
except APIUser.DoesNotExist:
raise APIUserManagerError('APIUser {} does not exist.'
''.format(username))

return api_user


def create_api_user(username, secret, permissions=None, default=False,
email='', django_password=None, first_name='',
last_name=''):
user = None
api_user = None

try:
api_user = get_api_user(username)
except APIUserManagerError:
pass
else:
raise UserExists('APIUser {} already exists.'.format(username))

try:
user = User.objects.get(username=username)
except User.DoesNotExist:
if not django_password:
raise UserDoesNotExist('Django User {} does not exist and '
'must be created. You must supply a '
'django_password argument.'
''.format(username))
user = User.objects.create_user(username, email, django_password)
user.first_name = first_name
user.last_name = last_name
user.save()

api_user = APIUser(user=user)
api_user.set_secret(secret)
api_user.set_permissions(permissions=permissions, default=default)
return api_user


def update_api_user(username, secret=None, permissions=None, default=False,
email='', django_password=None, first_name='',
last_name=''):
try:
api_user = get_api_user(username)
except APIUserManagerError:
raise UserDoesNotExist('API User {} does not exist.'.format(username))

if (secret):
api_user.set_secret(secret)
if (permissions):
api_user.set_permissions(permissions=permissions, default=default)

django_user_changed = False
if (email and email != api_user.user.email):
api_user.user.email = email
django_user_changed = True
if (first_name and first_name != api_user.user.first_name):
api_user.user.first_name = first_name
django_user_changed = True
if (last_name and last_name != api_user.user.last_name):
api_user.user.last_name = last_name
django_user_changed = True
if (django_password and not api_user.user.check_password(django_password)):
api_user.user.set_password(django_password)
django_user_changed = True

if django_user_changed:
api_user.user.save()
return api_user


def set_api_user_secret(username, secret):
api_user = get_api_user(username)
api_user.set_secret(secret)
return api_user


def set_api_user_permissions(username, permissions, default=False):
api_user = get_api_user(username)
api_user.set_permissions(permissions=permissions, default=default)
return api_user


def batch_create_update_api_users(filepath, default=None):
"""
This will open a csv file provided by filepath and batch create API
users based on its contents. Column names should be provided that
match the args to create_api_user (except default). If the API user
already exists, then this will attempt to update the secret and the
permissions for that user.
"""
data = []
with open(filepath, 'r') as csvfile:
permreader = csv.DictReader(csvfile)
for row in permreader:
row_data = {}
try:
row_data['username'] = row['username']
except KeyError:
raise APIUserManagerError('Your csv file must include a '
'"username" column.')
row_data['secret'] = row.get('secret', None)
row_data['django_password'] = row.get('django_password', None)
row_data['first_name'] = row.get('first_name', None)
row_data['last_name'] = row.get('last_name', None)
row_data['email'] = row.get('email', None)

permissions = {}

for key, val in row.iteritems():
if re.match(r'^permission_', key):
if val in ('True', 'true', 'TRUE', 'T', 't', '1'):
val = True
else:
val = False
permissions[re.sub(r'^permission_', '', key)] = val

if not permissions and default is not None:
permissions = default

row_data['permissions'] = permissions
data.append(row_data)

for row in data:
try:
create_api_user(row['username'], row['secret'],
permissions=row['permissions'],
email=row['email'],
django_password=row['django_password'],
first_name=row['first_name'],
last_name=row['last_name'], default=default)
except UserExists:
update_api_user(row['username'], row['secret'],
permissions=row['permissions'],
email=row['email'],
django_password=row['django_password'],
first_name=row['first_name'],
last_name=row['last_name'], default=default)
except UserDoesNotExist:
print ('User {} does not exist and no django_password was '
'provided. Skipping.'.format(row['username']))
21 changes: 12 additions & 9 deletions django/sierra/api/exceptions.py
Original file line number Diff line number Diff line change
@@ -1,31 +1,34 @@
'''
"""
Implements REST API exceptions.
'''
"""
from rest_framework import exceptions
from rest_framework import views

def sierra_exception_handler(exc):
'''

def sierra_exception_handler(exc, context):
"""
Custom exception handler. Defines what gets returned in the
response when clients encounter errors.
'''
response = views.exception_handler(exc)
"""
response = views.exception_handler(exc, context)
if response is not None:
response.data['status'] = response.status_code

return response


class BadQuery(exceptions.APIException):
status_code = 400
default_detail = ('One of the query parameters was not correctly '
'formatted.')
'formatted.')


class BadUpdate(exceptions.APIException):
status_code = 400
default_detail = ('The requested resource could not be updated because '
'the content received was invalid.')


class ReadOnlyEdit(exceptions.APIException):
status_code = 400
default_detail = ('The requested resource could not be updated because '
'the request attempted to update read-only content.')
'the request attempted to update read-only content.')
Loading

0 comments on commit 06d6caa

Please sign in to comment.