Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test coverage for PostgreSQL #28

Merged
merged 33 commits into from
Mar 6, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
edcd854
Use Ubuntu Xenial and Python 3.7
juliotrigo Feb 21, 2019
43cd615
Add PostgreSQL v10 to Travis
juliotrigo Feb 21, 2019
b759b11
Remove Travis addons: GitHub check not going green
juliotrigo Feb 21, 2019
cbcfa2a
Travis CI: use addons to install postgresql
juliotrigo Feb 21, 2019
343080e
Install psycopg2 as part of the tests
juliotrigo Feb 21, 2019
c2e5577
Run tests also using PostgreSQL
juliotrigo Feb 21, 2019
c1c4de3
Only return DB engine options for PostgreSQL
juliotrigo Feb 21, 2019
e1682c2
Exclude NULL sorting and non explicit sorting from tests
juliotrigo Feb 23, 2019
79e991b
Add test fixture to know whether test are running on PostgreSQL
juliotrigo Feb 23, 2019
29eac10
Add Makefile targets to run RDMS docker containers for testing
juliotrigo Feb 23, 2019
4072fd0
Improve documentation
juliotrigo Feb 23, 2019
9ce72b4
Improve documentation
juliotrigo Feb 23, 2019
231d17c
Improve documentation
juliotrigo Feb 23, 2019
26f7081
Comments to keep DB versions synchronized
juliotrigo Feb 23, 2019
3387430
Do not use Docker patch versions to be in sync with Travis CI
juliotrigo Feb 23, 2019
a365970
Update documentation
juliotrigo Feb 23, 2019
69ac461
Improve documentation
juliotrigo Feb 24, 2019
ee99669
Fix README inline literals and make line sizes consistent
juliotrigo Feb 24, 2019
cb3a4aa
Make README line sizes consistent
juliotrigo Feb 24, 2019
7523dff
Amend docstring
juliotrigo Feb 24, 2019
462b102
Amend typo in README
juliotrigo Feb 27, 2019
bd2c7a1
Remove unnecessary constant
juliotrigo Feb 27, 2019
a6bf6d5
Use Docker containers for tests in Travis CI
juliotrigo Feb 27, 2019
aa3cf86
Use Travis addons to update Docker via apt
juliotrigo Feb 27, 2019
7c97fe6
Travis CI apt update
juliotrigo Feb 27, 2019
cdffb25
Remove Travis apt addons: it has no effect and doubles run time
juliotrigo Feb 27, 2019
767f7dd
Improve tests documentation
juliotrigo Feb 27, 2019
56a034d
Rename variables in tests
juliotrigo Feb 27, 2019
6af450c
Fix CHANGELOG syntax
juliotrigo Feb 27, 2019
e45d260
Improve test assertions and make variable name consistent
juliotrigo Feb 28, 2019
3c4bde4
Amend MySQL test container name
juliotrigo Feb 28, 2019
ce1ca02
Automatically clean up test docker containers
juliotrigo Feb 28, 2019
e9155eb
Improve assertions in sorting tests
juliotrigo Mar 5, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,16 @@
sudo: false

language: python
python: 3.7

dist: xenial

services:
- mysql
- docker

before_install:
- make docker-mysql-run
- make docker-postgres-run

install:
- pip install tox
Expand All @@ -28,7 +36,6 @@ matrix:
- stage: test
python: 3.7
env: TOX_ENV=py37
dist: xenial

- stage: deploy
script: skip
Expand Down
5 changes: 3 additions & 2 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Version 0.8.0

Released 2018-06-25

* Adds support for `ilike` (case-insensitive) string comparison.
* Adds support for ``ilike`` (case-insensitive) string comparison.


Version 0.7.0
Expand Down Expand Up @@ -51,7 +51,8 @@ Released 2017-05-22
* Adds support for boolean functions within filters
* Adds the possibility of supplying a single dictionary as filters when
only one filter is provided
* Makes the `op` filter attribute optional: `==` is the default operator
* Makes the ``op`` filter attribute optional: ``==`` is the default
operator

Version 0.2.0
-------------
Expand Down
19 changes: 19 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
.PHONY: test

POSTGRES_VERSION?=9.6
MYSQL_VERSION?=5.7


rst-lint:
rst-lint README.rst
Expand All @@ -14,3 +17,19 @@ test: flake8
coverage: flake8 rst-lint
coverage run --source sqlalchemy_filters -m pytest test $(ARGS)
coverage report -m --fail-under 100


# Docker test containers

docker-mysql-run:
docker run -d --rm --name mysql-sqlalchemy-filters -p 3306:3306 \
-e MYSQL_ALLOW_EMPTY_PASSWORD=yes \
mysql:$(MYSQL_VERSION)

docker-postgres-run:
docker run -d --rm --name postgres-sqlalchemy-filters -p 5432:5432 \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD= \
-e POSTGRES_DB=test_sqlalchemy_filters \
-e POSTGRES_INITDB_ARGS="--encoding=UTF8 --lc-collate=en_US.utf8 --lc-ctype=en_US.utf8" \
postgres:$(POSTGRES_VERSION)
124 changes: 93 additions & 31 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ SQLAlchemy-filters
Filtering
---------

Assuming that we have a SQLAlchemy `query` object:
Assuming that we have a SQLAlchemy ``query`` object:

.. code-block:: python

Expand Down Expand Up @@ -61,7 +61,8 @@ Then we can apply filters to that ``query`` object (multiple times):

result = filtered_query.all()

It is also possible to filter queries that contain multiple models, including joins:
It is also possible to filter queries that contain multiple models,
including joins:

.. code-block:: python

Expand All @@ -84,7 +85,10 @@ It is also possible to filter queries that contain multiple models, including jo
result = filtered_query.all()


`apply_filters` will attempt to automatically join models to `query` if they're not already present and a model-specific filter is supplied. For example, the value of `filtered_query` in the following two code blocks is identical:
``apply_filters`` will attempt to automatically join models to ``query``
if they're not already present and a model-specific filter is supplied.
For example, the value of ``filtered_query`` in the following two code
blocks is identical:

.. code-block:: python

Expand All @@ -106,13 +110,20 @@ It is also possible to filter queries that contain multiple models, including jo
]
filtered_query = apply_filters(query, filter_spec)

The automatic join is only possible if sqlalchemy can implictly determine the condition for the join, for example because of a foreign key relationship.
The automatic join is only possible if sqlalchemy can implictly
determine the condition for the join, for example because of a foreign
key relationship.

Automatic joins allow flexibility for clients to filter and sort by related objects without specifying all possible joins on the server beforehand.
Automatic joins allow flexibility for clients to filter and sort by
related objects without specifying all possible joins on the server
beforehand.

Note that first filter of the second block does not specify a model. It is implictly applied to the `Foo` model because that is the only model in the original query passed to `apply_filters`.
Note that first filter of the second block does not specify a model.
It is implictly applied to the ``Foo`` model because that is the only
model in the original query passed to ``apply_filters``.

It is also possible to apply filters to queries defined by fields or functions:
It is also possible to apply filters to queries defined by fields or
functions:

.. code-block:: python

Expand All @@ -123,8 +134,8 @@ It is also possible to apply filters to queries defined by fields or functions:
Restricted Loads
----------------

You can restrict the fields that SQLAlchemy loads from the database by using
the `apply_loads` function:
You can restrict the fields that SQLAlchemy loads from the database by
using the ``apply_loads`` function:

.. code-block:: python

Expand All @@ -136,13 +147,18 @@ the `apply_loads` function:
query = apply_loads(query, load_spec) # will load only Foo.name and Bar.count


The effect of the `apply_loads` function is to _defer_ the load of any other fields to when/if they're accessed, rather than loading them when the query is executed. It only applies to fields that would be loaded during normal query execution.
The effect of the ``apply_loads`` function is to ``_defer_`` the load
of any other fields to when/if they're accessed, rather than loading
them when the query is executed. It only applies to fields that would be
loaded during normal query execution.


Effect on joined queries
^^^^^^^^^^^^^^^^^^^^^^^^

The default SQLAlchemy join is lazy, meaning that columns from the joined table are loaded only when required. Therefore `apply_loads` has limited effect in the following scenario:
The default SQLAlchemy join is lazy, meaning that columns from the
joined table are loaded only when required. Therefore ``apply_loads``
has limited effect in the following scenario:

.. code-block:: python

Expand All @@ -154,9 +170,14 @@ The default SQLAlchemy join is lazy, meaning that columns from the joined table
query = apply_loads(query, load_spec) # will load only Foo.name


`apply_loads` cannot be applied to columns that are loaded as `joined eager loads <http://docs.sqlalchemy.org/en/latest/orm/loading_relationships.html#joined-eager-loading>`_. This is because a joined eager load does not add the joined model to the original query, as explained `here <http://docs.sqlalchemy.org/en/latest/orm/loading_relationships.html#the-zen-of-joined-eager-loading>`_
``apply_loads`` cannot be applied to columns that are loaded as
`joined eager loads <http://docs.sqlalchemy.org/en/latest/orm/loading_relationships.html#joined-eager-loading>`_.
This is because a joined eager load does not add the joined model to the
original query, as explained
`here <http://docs.sqlalchemy.org/en/latest/orm/loading_relationships.html#the-zen-of-joined-eager-loading>`_

The following would not prevent all columns from Bar being eagerly loaded:
The following would not prevent all columns from Bar being eagerly
loaded:

.. code-block:: python

Expand All @@ -169,10 +190,14 @@ The following would not prevent all columns from Bar being eagerly loaded:

.. sidebar:: Automatic Join

In fact, what happens here is that `Bar` is automatically joined to `query`, because it is determined that `Bar` is not part of the original query. The `load_spec` therefore has no effect because the automatic join
results in lazy evaluation.
In fact, what happens here is that ``Bar`` is automatically joined
to ``query``, because it is determined that ``Bar`` is not part of
the original query. The ``load_spec`` therefore has no effect
because the automatic join results in lazy evaluation.

If you wish to perform a joined load with restricted columns, you must specify the columns as part of the joined load, rather than with `apply_loads`:
If you wish to perform a joined load with restricted columns, you must
specify the columns as part of the joined load, rather than with
``apply_loads``:

.. code-block:: python

Expand Down Expand Up @@ -201,9 +226,12 @@ Sort
result = sorted_query.all()


`apply_sort` will attempt to automatically join models to `query` if they're not already present and a model-specific sort is supplied. The behaviour is the same as in `apply_filters`.
``apply_sort`` will attempt to automatically join models to ``query`` if
they're not already present and a model-specific sort is supplied.
The behaviour is the same as in ``apply_filters``.

This allows flexibility for clients to sort by fields on related objects without specifying all possible joins on the server beforehand.
This allows flexibility for clients to sort by fields on related objects
without specifying all possible joins on the server beforehand.


Pagination
Expand Down Expand Up @@ -240,7 +268,8 @@ following format:
# ...
]

The `model` key is optional if the original query being filtered only applies to one model.
The ``model`` key is optional if the original query being filtered only
applies to one model.

If there is only one filter, the containing list may be omitted:

Expand All @@ -249,7 +278,7 @@ If there is only one filter, the containing list may be omitted:
filter_spec = {'field': 'field_name', 'op': '==', 'value': 'field_value'}

Where ``field`` is the name of the field that will be filtered using the
operator provided in ``op`` (optional, defaults to `==`) and the
operator provided in ``op`` (optional, defaults to ``==``) and the
provided ``value`` (optional, depending on the operator).

This is the list of operators that can be used:
Expand All @@ -269,7 +298,8 @@ This is the list of operators that can be used:

Boolean Functions
^^^^^^^^^^^^^^^^^
``and``, ``or``, and ``not`` functions can be used and nested within the filter specification:
``and``, ``or``, and ``not`` functions can be used and nested within the
filter specification:

.. code-block:: python

Expand All @@ -292,7 +322,8 @@ Boolean Functions
]


Note: ``or`` and ``and`` must reference a list of at least one element. ``not`` must reference a list of exactly one element.
Note: ``or`` and ``and`` must reference a list of at least one element.
``not`` must reference a list of exactly one element.

Sort format
-----------
Expand All @@ -311,25 +342,44 @@ applied sequentially:
Where ``field`` is the name of the field that will be sorted using the
provided ``direction``.

The `model` key is optional if the original query being sorted only applies to one model.
The ``model`` key is optional if the original query being sorted only
applies to one model.


Running tests
-------------

There are some Makefile targets that can be used to run the tests. A
test database will be created, used during the tests and destroyed
afterwards.

The default configuration uses both SQLite and MySQL (if the driver is
installed) to run the tests, with the following URIs:
The default configuration uses **SQLite**, **MySQL** (if the driver is
installed, which is the case when ``tox`` is used) and **PostgreSQL**
(if the driver is installed, which is the case when ``tox`` is used) to
run the tests, with the following URIs:

.. code-block:: shell

sqlite+pysqlite:///test_sqlalchemy_filters.db
mysql+mysqlconnector://root:@localhost:3306/test_sqlalchemy_filters
postgresql+psycopg2://postgres:@localhost:5432/test_sqlalchemy_filters?client_encoding=utf8'

A test database will be created, used during the tests and destroyed
afterwards for each RDBMS configured.

There are Makefile targets to run docker containers locally for both
**MySQL** and **PostgreSQL**, using the default ports and configuration:

.. code-block:: shell

$ make docker-mysql-run
$ make docker-postgres-run

Example of usage:
To run the tests locally:

.. code-block:: shell

$ # Create/activate a virtual environment
$ pip install tox
$ tox

There are some other Makefile targets that can be used to run the tests:

.. code-block:: shell

Expand All @@ -345,10 +395,22 @@ Example of usage:
$ ARGS='--sqlite-test-db-uri sqlite+pysqlite:///test_sqlalchemy_filters.db' make coverage



Database management systems
---------------------------

The following RDBMS are supported (tested):

- SQLite
- MySQL
- PostgreSQL


Python 2
--------

There is no active support for python 2, however it is compatiable as of February 2019, if you install funcsigs.
There is no active support for python 2, however it is compatible as of
February 2019, if you install ``funcsigs``.


License
Expand Down
3 changes: 3 additions & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@
'mysql': [
'mysql-connector-python-rf==2.2.2',
],
'postgresql': [
'psycopg2==2.7.7'
],
'python2': [
"funcsigs>=1.0.2"
]
Expand Down
Loading