Merge pull request #317 from toonarmycaptain/development

Implement Database object/refactoring.


Refactor application to use `Database` object in `definitions.DATABASE` for all interactions with persistence.

### Added
- Implement `Database` object to handle persistence 
- Implement `JSONDatabase(Database)` (original/current database backend).
### Changed
- Refactor all code/calls dealing with persistence to `definitions.DATABASE`.
- `Student.avatar_filename` changed to `Student.avatar_id` for naming consistency between database backends. This is ***backwards incompatible***, but is a simple string replace operation in any current data files.
- `JSONDatabase`'s `Registry` now checks if on-disk version of registry is correct, only writing to it if incorrect or non-existent.
- Increased test coverage, more tests converted to Pytest style tests.
- When clicking 'x' instead of 'save as' when a chart is displayed, UI no longer freezes, nor pops up a 'save chart as' file dialogue. 
### Removed
- `class_registry_functions.py`, `test_class_registry_functions.py`: functionality moved to `Registry` object in `persistence/databases/json_registry.py`.
### Depreciated
- Python 3.6 support ends with this release.
- Python 3.7 support will soon be removed also, in next release following release of python 3.9 - plan is to only support 2 minor releases of python at one time.
- `JSONDatabase` might be removed at some point, or not support new features, although it, or the data format might be kept for utility of debugging and editing.
- `data_version_conversion.py` will not be supporting conversion from older formats than current to any future versions.

.appveyor.yml

@@ -9,6 +9,13 @@ environment:
         - PYTHON: "C:\\Python37-x64"
         - PYTHON: "C:\\Python38-x64"
 
+matrix:
+    allow_failures:
+        - PYTHON: "C:\\Python36"
+          PYTHON_VERSION: "3.6.x"
+        - PYTHON: "C:\\Python36-x64"
+          PYTHON_VERSION: "3.6.x"
+
 init:
   - SET PATH=%PYTHON%;%PATH%
 

.travis.yml

@@ -11,13 +11,15 @@ python:
     - "3.6"
     - "3.7"
     - "3.8"
+    - "nightly"
 
 
 matrix:
     fast_finish: true
     allow_failures:
         - os: windows  # allow failure on Win until Travis-Win supports python.
-
+        - python: "3.6"
+        - python: "nightly"
 
 install:
     - pip install -r requirements_dev.txt

CHANGELOG.md

@@ -4,12 +4,43 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.7.0-alpha] - 2020-06-27
+### Added
+- `Database` ABC, establishing API for database objects.  
+    - Subclasses cannot instantiate without implementing methods defined in `Database`.
+    - Class attr `required_attributes: List[str]` is a list of string attr names that subclasses must in implement (such as `default_avatar_path`).. 
+    - Uses `ABCMetaEnforcedAttrs` metaclass to enforce existence of attrs in `required_attributes` in `Database` subclasses. 
+- `JSONDatabase(Database)` class implementing this API, apart from `get_avatar_path` which is incompatible: this method is implemented as `get_avatar_path_class_filename`, since it needs the class' name as well as the avatar's filename. This is clearly documented, and only used in one instance inside `take_chart_data_UI.py`.
+    - `Registry` object managing `JSONDatabase`'s registry.
+- `ClassIdentifier` - NamedTuple with attrs `id`, `name` for a class - allowing the `id` to be anything the database backend needs to use, with the human-readable/string name of the class. This avoids difficulty with supporting existing JSON database, as well as a uniform API between backends.
+- `definitions.DATABASE` `Database` object initialised on app_start, prompt user to set if app is being run for the first time.
+    - Presently defaults to `JSONDatabase`, ie legacy backend.
+    - Call to `DATABASE.close()` to close out database (eg write registry to disk for `JSONDatabase`, close connections etc).
+- `settings_functions` to change database backend. Actual change machinery yet to be implemented, as only JSON backend currently implemented.
+### Changed
+- Refactor all code/calls dealing with persistence to `definitions.DATABASE`.
+    - `Database` object responsible for creating needed paths apart from `app_data/`/`temp/`, such paths are removed from `DataFolder`.
+- `Student.avatar_filename` changed to `Student.avatar_id` for naming consistency between database backends. This is backwards incompatible, but is a simple string replace operation in any current data files.
+- `JSONDatabase`'s `Registry` now checks if on-disk version of registry is correct, only writing to it if incorrect or non-existent.
+- Factor out functions asking user for yes/no input into `ask_user_bool` function, taking a `question` and optional `invalid_input_response` parameters.
+- Increased test coverage, more tests converted to Pytest style tests.
+- Stricter `Path` object passing and usage.
+- Instances of 'folder' changed to 'dir' or 'directory' in vars/docstrings, apart from data_folder.py/`DataFolder`.
+- When clicking 'x' instead of 'save as' when a chart is displayed, UI no longer freezes, nor pops up a 'save chart as' file dialogue. 
+### Removed
+- `class_registry_functions.py`, `test_class_registry_functions.py`: functionality moved to `Registry` object in `persistence/databases/json_registry.py`.
+### Depreciated
+- Python 3.6 support ends with this release.
+- Python 3.7 support will soon be removed also, in `dionysus` release following release of python 3.9 - plan is to only support 2 minor releases of python at one time.
+- `JSONDatabase` may be removed at some point, or not support new features, although it, or the data format might be kept for utility of debugging and editing.
+- `data_version_conversion.py` will not be supporting conversion from older formats than current to any future versions.
+
 ## [0.6.0-alpha] - 2020-04-01
 ### Added
 - Implemented `temp` directory created in `APP_DATA` by `data_folder_check` on app start and removed on app exit (if it contains files). 
 - `NewClass` subclass of `Class` using `temp` directory to hold files before writing to database.
     - Initially holds avatars as user enters during class creation. 
-- Add [AllContributors](https://allcontributors.org/) badge to `README.md`, recognising project contributors, `.all-contributorsrc` with contributer data. 
+- Add [AllContributors](https://allcontributors.org/) badge to `README.md`, recognising project contributors, `.all-contributorsrc` with contributor data. 
 ### Changed
 - Separate UI/logic/persistence concerns in `create_classlist`. 
     - New functions `move_avatars_to_class_data`, `move_avatar_to_class_data` utilise `NewClass` to move avatars from `temp` to database.
@@ -215,7 +246,4 @@ Initial alpha release! Dionysus will take class lists, and successfully produce
 
 ### Known bugs/non-functional features:
 - setup.py is boilerplate and untested.
-- User supplied avatar does not copy to app data folder and thus does not work in app.
-- No indication of chart save location and not saving in desired/intended location in app_data/image_data/
-- Need to cut and paste/type user supplied avatar location is too awkward.
-- Preview/display of created chart does not reflect generated image accurately.
+

README.md

@@ -7,7 +7,6 @@
 [![Build status](https://ci.appveyor.com/api/projects/status/yb33uwd13tkv7l79?svg=true)](https://ci.appveyor.com/project/toonarmycaptain/dionysus) 
 [![Coverage Status](https://coveralls.io/repos/github/toonarmycaptain/dionysus/badge.svg)](https://coveralls.io/github/toonarmycaptain/dionysus) 
 [![codecov](https://codecov.io/gh/toonarmycaptain/dionysus/branch/master/graph/badge.svg)](https://codecov.io/gh/toonarmycaptain/dionysus)
-[![BCH compliance](https://bettercodehub.com/edge/badge/toonarmycaptain/dionysus?branch=master)](https://bettercodehub.com/) 
 [![Codacy Badge](https://api.codacy.com/project/badge/Grade/d24e9508258849c2b40760fce3448c6b)](https://www.codacy.com/app/toonarmycaptain/dionysus?utm_source=github.com&utm_medium=referral&utm_content=toonarmycaptain/dionysus&utm_campaign=Badge_Grade) 
 [![codebeat badge](https://codebeat.co/badges/c7b02602-ed39-46ff-9513-d06217fdfab4)](https://codebeat.co/projects/github-com-toonarmycaptain-dionysus-master) 
 [![CodeFactor](https://www.codefactor.io/repository/github/toonarmycaptain/dionysus/badge/master)](https://www.codefactor.io/repository/github/toonarmycaptain/dionysus/overview/master) 
@@ -18,7 +17,8 @@
 
 **dionysus** is an open source CLI app primarily aimed at teachers that charts student results for display using avatars, nicknames, or student names. 
 
-Currently in alpha release, fuller features, proper install/packaging, database backend, and a full GUI are future goals.
+Currently in alpha release, fuller features, proper install/packaging, database backend, and a full GUI/webapp are future goals.
+Supports latest two python minor versions - currently 3.7 and 3.8.
 
 Excitedly welcoming contributors! 
 

app_main.py

@@ -6,20 +6,22 @@
 
 import definitions
 
-from dionysus_app.class_registry_functions import cache_class_registry, check_registry_on_exit
 from dionysus_app.initialise_app import app_init, clear_temp
-from dionysus_app.UI_menus.main_menu import run_main_menu
+from dionysus_app.persistence.database_functions import load_database
 from dionysus_app.settings_functions import load_chart_save_folder
+from dionysus_app.UI_menus.main_menu import run_main_menu
 
 
 def quit_app():
     """
-    Checks disk registry, rewrites if inconsistent with runtime registry
-    (eg if user has deleted files during runtime), quits application.
+    Quits application.
+
+    Perform graceful termination tasks, eg clear temp/, close/finalise
+    database (eg close connections, flush data).
 
     :return: None
     """
-    check_registry_on_exit()  # Dump cached registry to disk if different class_registry.index.
+    definitions.DATABASE.close()
     clear_temp()  # Clear temp files.
     sys.exit()
 
@@ -36,10 +38,10 @@ def run_app():
 
     app_init()
 
-    # load runtime variables
-    definitions.REGISTRY = cache_class_registry()
+    # Load runtime variables.
+    definitions.DEFAULT_CHART_SAVE_DIR = load_chart_save_folder()
 
-    definitions.DEFAULT_CHART_SAVE_FOLDER = load_chart_save_folder()
+    definitions.DATABASE = load_database()
 
     run_main_menu()  # Startup checks successful, enter UI.
 

data_version_conversion.py

@@ -4,18 +4,16 @@
 import time
 
 from pathlib import Path
+from typing import Optional
 
 from dionysus_app.class_ import Class
-from dionysus_app.class_functions import write_classlist_to_file
-from dionysus_app.data_folder import DataFolder, CLASSLIST_DATA_FILE_TYPE
 from dionysus_app.file_functions import load_from_json_file
+from dionysus_app.persistence.databases.json import JSONDatabase
 from dionysus_app.student import Student
 from dionysus_app.UI_menus.UI_functions import select_file_dialogue
 
-CLASSLIST_DATA_PATH = DataFolder.generate_rel_path(DataFolder.CLASS_DATA.value)
 
-
-def main():
+def main() -> None:
     """
     Process arguments, run script based on args.
 
@@ -26,7 +24,7 @@ def main():
     run_script(run_args)
 
 
-def parse_args(args: list):
+def parse_args(args: list) -> argparse.Namespace:
     """
     Takes list of args passed to script.
 
@@ -44,7 +42,7 @@ def parse_args(args: list):
     return parser.parse_args(args)
 
 
-def run_script(args: argparse.Namespace):
+def run_script(args: argparse.Namespace) -> None:
     """
     Use args passed to script to execute chosen mode.
     No args: run GUI to select and process single file.
@@ -59,40 +57,52 @@ def run_script(args: argparse.Namespace):
     :param args: argparse.Namespace
     :return: None
     """
+    json_database = JSONDatabase()
     print(f'args={args}')
     print(f'args.filepath={args.filepath}')
     if args.all_class_data_files:
-        transform_all_old_data_files()
+        transform_all_old_data_files(json_database)
 
     elif args.filepath:
-        transform_old_cld_file(Path(args.filepath))
+        transform_old_cld_file(json_database, Path(args.filepath))
 
     else:
-        file_from_gui_dialogue()
+        file_from_gui_dialogue(json_database)
 
 
-def transform_all_old_data_files():
+def transform_all_old_data_files(json_database: JSONDatabase = None) -> None:
     """
     Transform data files matching old format to new format.
+    Instantiates JSONDatabase with default args.
 
+    :param json_database: JSONDatabase object
     :return: None
     """
-    for old_class_data_file in CLASSLIST_DATA_PATH.glob('**/*.cld'):
-        transform_old_cld_file(Path(old_class_data_file))
+    if not json_database:
+        json_database = JSONDatabase()
+    for old_class_data_file in json_database.class_data_path.glob('**/*.cld'):
+        transform_old_cld_file(json_database, Path(old_class_data_file))
 
     # NB can add operation to call on chart data files too if this is desired.
 
 
-def transform_old_cld_file(filepath: Path):
+def transform_old_cld_file(json_database: JSONDatabase, filepath: Path) -> None:
     """
     Transform .cld file to new format.
 
+    Takes a database object.
+    If none is passed, instantiates JSONDatabase with default args.
+
     NB If file is in the wrong location, or not in class data at all, a folder
     with the class name will be created in class_data.
 
+    :param json_database: JSONDatabase object
     :param filepath: Path
     :return: None
     """
+    if not json_database:
+        json_database = JSONDatabase()
+
     if not filepath.exists():
         print(f'File {filepath} does not exist.')
         return
@@ -112,28 +122,27 @@ def transform_old_cld_file(filepath: Path):
     class_name = filepath.stem
     new_class = transform_data(class_name, old_class_data)
 
-    write_classlist_to_file(new_class)
+    # Write to file rather than modifying in-place database:
+    json_database._write_classlist_to_file(new_class)
 
-    new_filename = class_name + CLASSLIST_DATA_FILE_TYPE
-    new_class_data_path_name = CLASSLIST_DATA_PATH.joinpath(class_name, new_filename)
+    new_filename = class_name + json_database.class_data_file_type
+    new_class_data_path_name = json_database.class_data_path.joinpath(class_name, new_filename)
 
     print(f'Transformed {new_class.name} data file '
           f'to new data format in {new_class_data_path_name}')
 
 
-def data_is_new_format(old_class_data: dict):
+def data_is_new_format(old_class_data: dict) -> bool:
     """
     Test if json dict data is in current format to avoid mangling good data.
 
     :param old_class_data: dict
     :return: Bool
     """
-    if 'students' in old_class_data.keys() and 'name' in old_class_data.keys():
-        return True
-    return False
+    return 'students' in old_class_data and 'name' in old_class_data
 
 
-def transform_data(class_name: str, old_class_data: dict):
+def transform_data(class_name: str, old_class_data: dict) -> Class:
     """
     Take class name (eg from old style cld filename), and loaded json dict,
     transform into a new-style Class object.
@@ -148,35 +157,34 @@ def transform_data(class_name: str, old_class_data: dict):
             new_students.append(Student(name=student_name))
         else:
             new_students.append(Student(name=student_name,
-                                        avatar_filename=old_class_data[student_name][0]),
+                                        avatar_id=old_class_data[student_name][0]),
                                 )
-    new_class = Class(name=class_name, students=new_students)
-    return new_class
+    return Class(name=class_name, students=new_students)
 
 
-def file_from_gui_dialogue():
+def file_from_gui_dialogue(json_database) -> None:
     """
     Spawn a GUI file selection dialogue, and transform the selected file into
     the new data format.
 
     :return: None
     """
-    filepath = get_data_file()
+    filepath = get_data_file(json_database)
     if filepath:
-        transform_old_cld_file(filepath)
+        transform_old_cld_file(json_database, filepath)
     if not filepath:
         print('No file selected.')
 
 
-def get_data_file():
+def get_data_file(json_database: JSONDatabase) -> Optional[Path]:
     """
     Prompt user to select a file from a GUI file selection dialogue.
 
     :return: Path or None
     """
     selected_filename = select_file_dialogue(title_str='Select file to transform:',
                                              filetypes=[('.cld', '*.cld'), ("all files", "*.*")],
-                                             start_dir=CLASSLIST_DATA_PATH,
+                                             start_dir=json_database.class_data_path,
                                              )
 
     if selected_filename is None:

definitions.py

@@ -1,10 +1,30 @@
+"""
+definitions.py - source for vars used throughout application.None
+
+ROOT_DIR - path to directory containing app_main/definitions
+DEFAULT_DATABASE_BACKEND - the default database backend
+DATABASE - the database object.
+DEFAULT_CHART_SAVE_DIR - default user save folder for generated charts.
+
+For state-holding object eg DATABASE, must import definitions, then use
+dot access to use the object:
+    import definitions
+    definitions.DATABASE.do_stuff()
+"""
 import os
 
 from pathlib import Path
-from typing import List, Optional
+from typing import Optional, TYPE_CHECKING
+
+# Import to get around circular import caused by type checking. Type as string.
+if TYPE_CHECKING:
+    from dionysus_app.persistence.database import Database  # Line skipped from coverage.
 
 ROOT_DIR = os.path.dirname(os.path.abspath(__file__))  # Global root directory.
 
-REGISTRY: Optional[List] = None
+DEFAULT_DATABASE_BACKEND = 'JSON'
+# Ignore typehint error: DATABASE object needs to be initialised with a value
+DATABASE: 'Database' = None  # type: ignore
+
 
-DEFAULT_CHART_SAVE_FOLDER: Optional[Path] = None  # Path object.
+DEFAULT_CHART_SAVE_DIR: Optional[Path] = None  # Path object.