Testing sprout 🌱 (#84)

* test: add trivial test * feat: add integration workflow * feat: add install deps and test to int workflow * Draft test of get_oeis_values Needs server to be running already. * Improve test of get_oeis_values Use test client instead of already-running server. * Speed up background work By choosing a sequence with fewer references and smaller values * Drop dummy test * Enable test discovery * Update test comments to reflect current views.py * Run each test in a fresh database * Start documenting test system * Abstract endpoint tests * Skip known failing test Also, only show mid-test messages in verbose mode * Streamline endpoint test attribute assertions * Fail test config gracefully when POSTGRES_TEST_DB is unset or empty This is really convoluted. We should find a better way to handle it. * Document disposable database * Add disposable database setup to testing docs To clarify a reference to the PostgreSQL installation docs, I changed what looked like a sectioning error. * Spruce up testing docs * Add a few more details about writing tests * Cut needless import * Reference issue #77 for skipped test; fix typos * Add test command to manage.py * Remove continuous integration workflow Continuous integration will be in a different pull request. --------- Co-authored-by: Liam Mulhall <liammulh@gmail.com> Co-authored-by: Aaron Fenyes <fenyes@ihes.fr>
numberscope · Oct 21, 2023 · d1df281 · d1df281
1 parent 210cb85
commit d1df281
Show file tree

Hide file tree

Showing 10 changed files with 340 additions and 12 deletions.
diff --git a/README.md b/README.md
@@ -51,6 +51,7 @@ If you plan on developing backscope, you might find some of these useful:
 - [Understanding our requirements files](doc/requirements.md)
 - [API endpoints](doc/api_endpoints.md)
 - [Directory descriptions](doc/directory_descriptions.md)
+- [Writing and running tests](doc/tests.md)
 
 If you are a maintainer, you might find some of these useful:
 

diff --git a/doc/install-postgres.md b/doc/install-postgres.md
@@ -31,7 +31,7 @@ of commands that will configure PostgreSQL properly.
 Once the database and user are created and authentication and permissions are
 configured properly, you can proceed to the next step.
 
-### Set up your environment
+## Set up your environment
 
 This project uses python-dotenv. In order to detect your database
 username / password, you must create a file called `.env` in the root
@@ -46,10 +46,18 @@ APP_ENVIRONMENT="development"
 SECRET_KEY="Uneccessary for development"
 ```
 
+To run tests, the `.env` file must also include:
+
+```
+POSTGRES_DISPOSABLE_DB="<disposable database name>"
+```
+
+:warning: **Beware:** running tests will clear the database `POSTGRES_DISPOSABLE_DB`. Other actions may also clear this database.
+
 You can see other configuration options inside
 [the config file](./flaskr/config.py).
 
-### Configure the database
+## Configure the database
 
 Note that in the following guide, commands that you would be
 entering/executing are preceded by a `>` character (representing a generic

diff --git a/doc/tests.md b/doc/tests.md
@@ -0,0 +1,118 @@
+# Write and run tests
+
+## Write tests
+
+### Basics
+
+Backscope uses the [`unittest`](https://docs.python.org/3/library/unittest.html) framework for testing.
+
+Tests are kept in the [`flaskr/nscope/test`](../flaskr/nscope/test) directory. The test routine opens all the files with names matching `test*.py`, pulls out all the classes that descend from `unittest.TestCase`, and runs all the tests those classes describe. You can use the `@unittest.skip()` decorator to skip tests.
+
+### Examples
+
+The file [`trivial_test.py`](../flaskr/nscope/test/trivial_test.py) contains a minimal example of a test: the test that always passes. This test isn't worth running, so we gave it a file name that the test routine will ignore.
+
+The file [`abstract_endpoint_test.py`](../flaskr/nscope/test/abstract_endpoint_test.py) contains an abstract test&mdash;a class that describes a whole family of tests. The concrete tests in [`test_get_oeis_values.py`](../flaskr/nscope/test/test_get_oeis_values.py) descend from it. The abstract test can't be run, so we gave it a file name that the test routine will ignore.
+
+## Set up for testing
+
+### Create a disposable database
+
+Before you can run tests, you need to create a disposable database and give `<backscope database user>` all permissions on it. We'll use `<disposable database name>` to stand for whatever you name it.
+
+For guidance, consult the generic instructions on how to [create a database](install-postgres.md#create-a-database), or the Ubuntu-specific instructions on how to create a database when you [install and configure PostgreSQL](install-ubuntu.md#install-and-configure-postgresql) under Ubuntu.
+
+Unlike the main database, the disposable database doesn't need to be configured.
+
+### Specify the disposable database in your environment
+
+Add the line
+
+```
+POSTGRES_DISPOSABLE_DB="<disposable database name>"
+```
+
+to your `.env` file.
+
+:warning: **Beware:** running tests will clear the database `POSTGRES_DISPOSABLE_DB`. Other actions may also clear this database.
+
+For guidance, consult the basic instructions on how to [set up your environment](install-postgres.md#set-up-your-environment). If you're indecisive, put the line that specifies the disposable database just after the one that specifies the main database.
+
+## Run tests
+
+### Call the test routine
+
+1. Go into the top-level directory of the Backscope repository.
+2. Activate the Backscope virtual environment.
+   + If you're using [`venv`](https://docs.python.org/3/library/venv.html) to manage virtual environments, and you've put Backscope's virtual environment in a directory called `.venv`, use the command `source .venv/bin/activate` to activate.
+3. Call `python manage.py test`.
+   + In quiet mode, you'll see a string of characters representing passed (`.`), failed (`F`), and skipped (`s`) tests.
+   + To see the tests' names as well as their outcomes, call `python manage.py test -v` or `python manage.py test --verbose`.
+   + This command is a wrapper for `python -m unittest [-v]` and `python -m unitpytest discover [-v] [-s START] [-p PATTERN] [-t TOP]`. For more options, call `unittest` or `unittest discover` directly.
+
+### Look at the test output
+
+Here are some examples of what test results can look like.
+
+:arrow_down: The `..` below tells us that both tests passed.
+
+```
+> python manage.py test
+..
+----------------------------------------------------------------------
+Ran 2 tests in 1.650s
+
+OK
+```
+
+:arrow_down: The `s.` below tells us that one test was skipped and the other passed. Running in verbose mode may print an explanation of why we're skipping the test. This message is passed to the `@unittest.skip()` decorator.
+
+```
+> python manage.py test
+s.
+----------------------------------------------------------------------
+Ran 2 tests in 0.934s
+
+OK (skipped=1)
+```
+
+:arrow_down: The `F.` below tells us that one test failed and the other passed. A report from the failed test follows.
+
+```
+> python manage.py test
+F.
+======================================================================
+FAIL: test_endpoint (flaskr.nscope.test.test_get_oeis_values.TestGetOEISValues)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "/home/aaron/Documents/code/backscope/flaskr/nscope/test/abstract_endpoint_test.py", line 65, in test_endpoint
+    self.assertDictEqual(response.json, self.expected_response_json)
+AssertionError: {'id'[61 chars]': {'0': '1', '1': '2', '2': '4', '3': '8', '4[83 chars]32'}} != {'id'[61 chars]': {'1': '1', '2': '2', '3': '4', '4': '8', '5[84 chars]32'}}
+Diff is 1082 characters long. Set self.maxDiff to None to see it.
+
+----------------------------------------------------------------------
+Ran 2 tests in 1.631s
+
+FAILED (failures=1)
+```
+
+:arrow_down: Testing in verbose mode, like below, shows the tests' names as well as their outcomes.
+
+```
+> python manage.py test -v
+test_endpoint (flaskr.nscope.test.test_get_oeis_values.TestGetOEISValues) ... 
+  Testing response
+  Waiting for background work
+  Background work done
+ok
+test_endpoint (flaskr.nscope.test.test_get_oeis_values.TestGetOEISValuesWithoutShift) ... 
+  Testing response
+  Waiting for background work
+  Background work done
+ok
+
+----------------------------------------------------------------------
+Ran 2 tests in 2.556s
+
+OK
+```
diff --git a/flaskr/__init__.py b/flaskr/__init__.py
@@ -16,23 +16,43 @@
 
 from .config import config
 
-# This statement loads all environment variables from .env
+
+"""Exception for when environment variables are bad or missing."""
+class EnvironmentException(Exception):
+  pass
+
+
+# Load all environment variables from .env
 load_dotenv()
 
 # Create a new sql alchemy database object
 db = SQLAlchemy()
 
-# default environment is development, otherwie specified by .env
-def create_app(environment='development'):
-
-    # Get app type from .env
-    environment = os.environ.get('APP_ENVIRONMENT', environment)
+# To choose the environment, we look for settings in the following order:
+#  (1) Function parameter
+#  (2) .env
+#  (3) Default to 'development'
+def create_app(environment=None):
+    if environment is None:
+      # Get app type from .env if provided. Otherwise, use 'development'
+      environment = os.environ.get('APP_ENVIRONMENT', 'development')
 
     # Initial app and configuration
     app = Flask(__name__, instance_relative_config=True)
 
     # Upload config from config.py
     if environment == 'development': CORS(app)
+    if config[environment].TESTING and config[environment].SQLALCHEMY_DATABASE_URI is None:
+      ## this is a really convoluted way of throwing an exception when you try
+      ## to run tests without specifying the test database, but allowing the
+      ## test database to be unspecified in other circumstances. we should clean
+      ## this up somehow
+      raise EnvironmentException(
+        'To create the Backscope app in testing mode, the '
+        'POSTGRES_DISPOSABLE_DB environment variable must be set to a non-empty '
+        'string. Beware: running tests will clear the database '
+        'POSTGRES_DISPOSABLE_DB. Other actions may also clear this database.'
+      )
     app.config.from_object(config[environment])
 
     # Logging

diff --git a/flaskr/config.py b/flaskr/config.py
@@ -15,14 +15,18 @@
     'port': os.getenv('POSTGRES_PORT', 5432),
 }
 
+# the key 'POSTGRES_DISPOSABLE_DB' is missing by default because tests and other
+# actions can and will clear whatever database it names!
 TEST_POSTGRES = {
     'user': os.getenv('POSTGRES_USER', 'postgres'),
     'pw': os.getenv('POSTGRES_PASSWORD', 'root'),
-    'db': os.getenv('POSTGRES_DB', 'postgres'),
     'host': os.getenv('POSTGRES_HOST', 'localhost'),
     'port': os.getenv('POSTGRES_PORT', 5432),
 }
-
+if 'POSTGRES_DISPOSABLE_DB' in os.environ:
+  _postgres_disposable_db = os.getenv('POSTGRES_DISPOSABLE_DB')
+  if not f'{_postgres_disposable_db}' == '':
+    TEST_POSTGRES['db'] = _postgres_disposable_db
 
 class Config:
     ERROR_404_HELP = False
@@ -34,20 +38,26 @@ class Config:
 
     DOC_USERNAME = 'api'
     DOC_PASSWORD = 'password'
+
+    TESTING = False
+    DEBUG = False
 
 
 class DevConfig(Config):
     DEBUG = True
 
 
 class TestConfig(Config):
-    SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{user}:{pw}@{host}:{port}/{db}'.format(**TEST_POSTGRES)
+    if 'db' in TEST_POSTGRES and not '{db}'.format(**TEST_POSTGRES) == '':
+      SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://{user}:{pw}@{host}:{port}/{db}'.format(**TEST_POSTGRES)
+    else:
+      SQLALCHEMY_DATABASE_URI = None
     TESTING = True
     DEBUG = True
 
 
 class ProdConfig(Config):
-    DEBUG = False
+    pass
 
 
 config = {

diff --git a/flaskr/nscope/test/__init__.py b/flaskr/nscope/test/__init__.py
@@ -0,0 +1,3 @@
+"""
+Init file for test (only needed for test discovery)
+"""
diff --git a/flaskr/nscope/test/abstract_endpoint_test.py b/flaskr/nscope/test/abstract_endpoint_test.py
@@ -0,0 +1,66 @@
+import unittest
+import sys
+from flaskr import create_app, db
+import flaskr.nscope.views as views
+
+
+# guidance on test database handling:
+#   https://stackoverflow.com/a/17818795
+#   https://flask-testing.readthedocs.io/en/v0.4/
+
+class AbstractEndpointTest(unittest.TestCase):
+  def assert_endpoint_test_attr(self, name):
+    assert hasattr(self, name), f"Can't construct endpoint test without '{name}' attribute"
+
+  def __init__(self, *args, **kwargs):
+    # make sure required attributes are present
+    self.assert_endpoint_test_attr('endpoint')
+    self.assert_endpoint_test_attr('expected_response_json')
+
+    # check whether unittest is running in verbose mode
+    # hat tip StackOverflow users Dimitris Fasarakis Hilliard and EquipDev...
+    #   https://stackoverflow.com/a/43002355
+    #   https://stackoverflow.com/questions/43001768/how-can-a-test-in-python-unittest-get-access-to-the-verbosity-level#comment73163492_43002355
+    # ... who provided this code under the MIT license
+    #   https://meta.stackexchange.com/q/271080
+    self.verbose = ('-v' in sys.argv) or ('--verbose' in sys.argv)
+
+    super().__init__(*args, *kwargs)
+
+  def setUp(self):
+    self.app = create_app('testing')
+    self.ctx = self.app.app_context()
+    with self.ctx:
+      db.create_all()
+
+    # put mid-test messages on a new line
+    if self.verbose:
+      print()
+
+  def tearDown(self):
+    # wait for background work to finish
+    if self.verbose:
+      print("  Waiting for background work")
+    views.executor.shutdown()
+    if self.verbose:
+      print("  Background work done")
+
+    # clear database
+    db.session.remove()
+    with self.ctx:
+      db.drop_all()
+
+  def test_endpoint(self):
+    # using test client is recommended in Flask testing how-to
+    #   https://flask.palletsprojects.com/en/2.3.x/testing/
+    # "The test client makes requests to the application without running a live
+    # server." the `with` block runs teardown
+    #   https://github.com/pallets/flask/issues/2949
+    with self.app.test_client() as client:
+      if self.verbose:
+        print("  Testing response")
+      response = client.get(self.endpoint)
+      self.assertEqual(response.status_code, 200)
+      self.assertDictEqual(response.json, self.expected_response_json)
+
+      # TO DO: test background work
diff --git a/flaskr/nscope/test/test_get_oeis_values.py b/flaskr/nscope/test/test_get_oeis_values.py
@@ -0,0 +1,67 @@
+import unittest
+import flaskr.nscope.test.abstract_endpoint_test as abstract_endpoint_test
+
+
+class TestGetOEISValuesWithoutShift(abstract_endpoint_test.AbstractEndpointTest):
+  endpoint = "http://127.0.0.1:5000/api/get_oeis_values/A153080/12"
+
+  # we choose A153080 because:
+  # - it has zero shift, so the test can pass even if the shift defaults to zero
+  # - it currently has small values and few references, which speeds up the
+  #   background work triggered by the request
+  expected_response_json = {
+    'id': 'A153080',
+    'name': 'A153080 [name not yet loaded]',
+    'values': {
+      '0': '2',
+      '1': '15',
+      '2': '28',
+      '3': '41',
+      '4': '54',
+      '5': '67',
+      '6': '80',
+      '7': '93',
+      '8': '106',
+      '9': '119',
+      '10': '132',
+      '11': '145'
+    }
+  }
+
+# this test is skipped because it's sensitive to issue #77. the skip decorator
+# should be removed when the issue is fixed.
+#   https://github.com/numberscope/backscope/issues/77
+# the issue is that `fetch_values` never sets the `shift` attribute of the
+# Sequence it returns, so we end up indexing from zero even if the shift should
+# be nonzero
+@unittest.skip("Shift attribute isn't being set yet")
+class TestGetOEISValues(abstract_endpoint_test.AbstractEndpointTest):
+  endpoint = "http://127.0.0.1:5000/api/get_oeis_values/A321580/12"
+
+  # we choose A321580 because:
+  # - it has a nonzero shift, so we can make sure the default value is getting
+  #   changed to the actual value
+  # - it currently has small values and few references, which speeds up the
+  #   background work triggered by the request
+  expected_response_json = {
+    'id': 'A321580',
+    'name': 'A321580 [name not yet loaded]',
+    'values': {
+      '1': '1',
+      '2': '2',
+      '3': '4',
+      '4': '8',
+      '5': '10',
+      '6': '12',
+      '7': '16',
+      '8': '18',
+      '9': '24',
+      '10': '26',
+      '11': '28',
+      '12': '32'
+    }
+  }
+
+
+if __name__ == "__main__":
+    unittest.main()