Hyper-parameter Tuning Support -- Optuna (#62)

* Common hyperparameter tuning interface #60 

* Added Optuna support

* Refactored backend to support split of fixed and tuneable parameters 

* Added black/isort

* Handles usage pattern of drop-in argparser replacement where no configs (from cmd line or as input into ConfigArgBuilder) are passed thus falling back on all defaults or definitions from the command line. fix-up of all cmdline usage pattern. there were certain edge cases that were not getting caught correctly if it wasn't overriding an existing payload from a yaml file. #61

* Unit tests

Signed-off-by: Nicholas Cilfone <nicholas.cilfone@fmr.com>

.github/pull_request_template.md

@@ -0,0 +1,18 @@
+---
+name: Pull request
+about: Create a pull request for merge
+
+---
+
+## What does this PR do?
+E.g. Describe the added feature or what issue it fixes #(issue)...
+
+## Checklist
+  - [ ] Did you adhere to [PEP-8](https://www.python.org/dev/peps/pep-0008/) standards?
+  - [ ] Did you run black and isort prior to submitting your PR? 
+  - [ ] Does your PR pass all existing unit tests?
+  - [ ] Did you add associated unit tests for any additional functionality?
+  - [ ] Did you provide documentation ([Numpy Docstring format](https://numpydoc.readthedocs.io/en/latest/format.html#style-guide)) whenever possible, even for simple functions or classes?
+
+## Review
+Request will go to reviewers to approve for merge.

.github/workflows/python-coverage.yaml

@@ -21,11 +21,19 @@ jobs:
       with:
         python-version: '3.8'
 
+    - uses: actions/cache@v2
+      with:
+        path: ${{ env.pythonLocation }}
+        key: ${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}-${{ hashFiles('REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/DEV_REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/S3_REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/TUNE_REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/TEST_EXTRAS_REQUIREMENTS_REQUIREMENTS.txt') }}
+
     - name: Install dependencies and dev dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install -r DEV_REQUIREMENTS.txt
-        pip install -r S3_REQUIREMENTS.txt
+        pip install -r REQUIREMENTS.txt
+        pip install -r ./requirements/DEV_REQUIREMENTS.txt
+        pip install -r ./requirements/S3_REQUIREMENTS.txt
+        pip install -r ./requirements/TUNE_REQUIREMENTS.txt
+        pip install -r ./requirements/TEST_EXTRAS_REQUIREMENTS.txt
 
     - name: Test with pytest
       run: |

.github/workflows/python-docs.yaml

@@ -18,12 +18,18 @@ jobs:
       uses: actions/setup-python@v2
       with:
         python-version: '3.8'
+
+    - uses: actions/cache@v2
+      with:
+        path: ${{ env.pythonLocation }}
+        key: ${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}-${{ hashFiles('REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/DEV_REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/S3_REQUIREMENTS.txt') }}
+
     - name: Install dependencies and dev dependencies
       run: |
         python -m pip install --upgrade pip
         pip install -e .[s3]
-        pip install -r DEV_REQUIREMENTS.txt
-        pip install -r S3_REQUIREMENTS.txt
+        pip install -r ./requirements/DEV_REQUIREMENTS.txt
+        pip install -r ./requirements/S3_REQUIREMENTS.txt
 
     - name: Build docs with Portray
       env:

.github/workflows/python-lint.yaml

@@ -0,0 +1,42 @@
+# This workflow will run isort and black linters on PRs
+
+name: lint
+
+# on: workflow_dispatch
+on:
+  pull_request:
+    branches: [master]
+  push:
+    branches: [master]
+
+jobs:
+  run_lint:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.8'
+
+    - uses: actions/cache@v2
+      with:
+        path: ${{ env.pythonLocation }}
+        key: ${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}-${{ hashFiles('REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/DEV_REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/S3_REQUIREMENTS.txt') }}
+
+    - name: Install dependencies and dev dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e .
+        pip install -r ./requirements/DEV_REQUIREMENTS.txt
+        pip install -r ./requirements/S3_REQUIREMENTS.txt
+
+    - name: Run isort linter
+      run: |
+        isort --check . --skip="debug" --skip="versioneer.py" --skip="tests" --skip="_version.py"
+
+    - name: Run black linter
+      run: |
+        black --check . --exclude="versioneer.py|_version.py|debug|tests"

.github/workflows/python-pytest-s3.yaml

@@ -23,16 +23,21 @@ jobs:
       with:
         python-version: ${{ matrix.python-version }}
 
+    - uses: actions/cache@v2
+      with:
+        path: ${{ env.pythonLocation }}
+        key: ${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}-${{ hashFiles('REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/DEV_REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/S3_REQUIREMENTS.txt') }}
+
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
         pip install -e .
-        pip install -r DEV_REQUIREMENTS.txt
-        pip install -r S3_REQUIREMENTS.txt
+        pip install -r ./requirements/DEV_REQUIREMENTS.txt
+        pip install -r ./requirements/S3_REQUIREMENTS.txt
 
     - name: Test with pytest
       run: |
-        pytest tests/s3 --cov=spock --cov-config=.coveragerc --junitxml=junit/test-results-${{ matrix.python-version }}.xml --cov-report=xml --cov-report=html
+        pytest tests/s3 --cov=spock --cov-config=.coveragerc --junitxml=junit/test-results-s3-${{ matrix.python-version }}.xml --cov-report=xml --cov-report=html
 
     - name: Upload pytest test results
       uses: actions/upload-artifact@v2

.github/workflows/python-pytest-tune.yaml

@@ -0,0 +1,50 @@
+# This workflow will install Python dependencies, run S3 tests with a variety of Python versions
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: pytest-tune
+
+on:
+  pull_request:
+    branches: [master]
+  push:
+    branches: [master]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7, 3.8, 3.9]
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - uses: actions/cache@v2
+      with:
+        path: ${{ env.pythonLocation }}
+        key: ${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}-${{ hashFiles('REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/DEV_REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/S3_REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/TUNE_REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/TEST_EXTRAS_REQUIREMENTS_REQUIREMENTS.txt') }}
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e .
+        pip install -r ./requirements/DEV_REQUIREMENTS.txt
+        pip install -r ./requirements/S3_REQUIREMENTS.txt
+        pip install -r ./requirements/TUNE_REQUIREMENTS.txt
+        pip install -r ./requirements/TEST_EXTRAS_REQUIREMENTS.txt
+
+    - name: Test with pytest
+      run: |
+        pytest tests/tune --cov=spock --cov-config=.coveragerc --junitxml=junit/test-results-tune-${{ matrix.python-version }}.xml --cov-report=xml --cov-report=html
+
+    - name: Upload pytest test results
+      uses: actions/upload-artifact@v2
+      with:
+        name: pytest-results-${{ matrix.python-version }}
+        path: junit/test-results-${{ matrix.python-version }}.xml
+      # Use always() to always run this step to publish test results when there are test failures
+      if: ${{ always() }}

.github/workflows/python-pytest.yml

@@ -23,11 +23,16 @@ jobs:
       with:
         python-version: ${{ matrix.python-version }}
 
+    - uses: actions/cache@v2
+      with:
+        path: ${{ env.pythonLocation }}
+        key: ${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}-${{ hashFiles('REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/DEV_REQUIREMENTS.txt') }}-${{ hashFiles('./requirements/S3_REQUIREMENTS.txt') }}
+
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
         pip install -e .
-        pip install -r DEV_REQUIREMENTS.txt
+        pip install -r ./requirements/DEV_REQUIREMENTS.txt
 
     - name: Test with pytest
       run: |

CONTRIBUTING.md

@@ -4,6 +4,7 @@ Requests in the public repository.
 
 ## Contribution Guidelines
 1. Adhere to [PEP-8](https://www.python.org/dev/peps/pep-0008/) standards.
-2. Any changes to core functionality must pass all existing unit tests.
-3. Additional functionality should have associated unit tests.
-4. Provide documentation (Google Docstring format) whenever possible, even for simple functions or classes.
+2. Run black and isort linters before creating a PR.
+3. Any changes to core functionality must pass all existing unit tests.
+4. Additional functionality should have associated unit tests.
+5. Provide documentation ([Numpy Docstring format](https://numpydoc.readthedocs.io/en/latest/format.html#style-guide)) whenever possible, even for simple functions or classes.

NOTICE.txt

@@ -4,13 +4,21 @@
 // ------------------------------------------------------------------
 
 Spock
-Copyright [2109-2020] FMR LLC
+Copyright [2019-2021] FMR LLC
 
 This product includes software developed at
 FMR LLC (https://www.fidelity.com/).
 
 This product relies on the following works (and the dependencies thereof), installed separately:
 - attrs | https://github.com/python-attrs/attrs | MIT License
 - GitPython | https://github.com/gitpython-developers/GitPython | BSD 3-Clause License
+- pytomlpp | https://github.com/bobfang1992/pytomlpp | MIT License
 - PyYAML | https://github.com/yaml/pyyaml | MIT License
-- toml | https://github.com/toml-lang/toml | MIT License
+
+
+Optional extensions rely on the following works (and the dependencies thereof), installed separately:
+- boto3 | https://github.com/boto/boto3 | Apache License 2.0
+- botocore | https://github.com/boto/botocore | Apache License 2.0
+- hurry.filesize | https://pypi.org/project/hurry.filesize/ | ZPL 2.1
+- optuna | https://optuna.org/ | MIT License
+- s3transfer | https://github.com/boto/s3transfer | Apache License 2.0

README.md

@@ -3,6 +3,7 @@
 
 [![License](https://img.shields.io/badge/License-Apache%202.0-9cf)](https://opensource.org/licenses/Apache-2.0)
 [![Python](https://img.shields.io/badge/python-3.6+-informational.svg)]()
+[![Style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![PyPI version](https://badge.fury.io/py/spock-config.svg)](https://badge.fury.io/py/spock-config)
 [![Coverage Status](https://coveralls.io/repos/github/fidelity/spock/badge.svg?branch=master)](https://coveralls.io/github/fidelity/spock?branch=master)
 ![Tests](https://github.com/fidelity/spock/workflows/pytest/badge.svg?branch=master)
@@ -45,6 +46,10 @@ recent features, bugfixes, and hotfixes.
 
 See [Releases](https://github.com/fidelity/spock/releases) for more information.
 
+#### July 21, 2021
+* Added hyper-parameter tuning support with `pip install spock-config[tune]`
+* Hyper-parameter tuning backend support for Optuna define-and-run API (WIP for Ax)
+
 #### May 6th, 2021
 * Added S3 support with `pip install spock-config[s3]`
 * S3 addon supports automatically handling loading/saving from paths defined with `s3://` URI(s) by passing in an
@@ -81,6 +86,8 @@ Example `spock` usage is located [here](https://github.com/fidelity/spock/blob/m
   parameter configuration to YAML, TOML, or JSON with a single chained command (with extra runtime info such as Git info, 
   Python version, machine FQDN, etc). The saved markdown file can be used as the configuration input to reproduce 
   prior runtime configurations.
+* [Hyper-Parameter Tuner Addon](https://fidelity.github.io/spock/docs/addons/tuner/About.html): Provides a unified
+  interface for hyper-parameter tuning that supports various backends (Optuna, WIP: Ax)
 * [S3 Addon](https://fidelity.github.io/spock/docs/addons/S3/): Automatically detects `s3://` URI(s) and handles loading 
   and saving `spock` configuration files when an active `boto3.Session` is passed in (plus any additional 
   `S3Transfer` configurations)

REQUIREMENTS.txt

@@ -1,4 +1,4 @@
 attrs
 GitPython
-pyYAML
-toml
+pytomlpp
+pyYAML

docs/Installation.md

@@ -3,7 +3,7 @@
 ### Requirements
 
 * Python: 3.6+
-* Dependencies: attrs, GitPython, PyYAML, toml
+* Base Dependencies: attrs, GitPython, PyYAML, toml
 * Tested OS: Unix (Ubuntu 16.04, Ubuntu 18.04), OSX (10.14.6)
 
 ### Install/Upgrade
@@ -14,10 +14,21 @@ pip install spock-config
 ```
 
 #### w/ S3 Extension
+
+Extra Dependencies: boto3, botocore, hurry.filesize, s3transfer
+
 ```bash
 pip install spock-config[s3]
 ```
 
+#### w/ Hyper-Parameter Tuner Extension
+
+Extra Dependencies: optuna
+
+```bash
+pip install spock-config[tune]
+```
+
 #### Pip From Source
 ```bash
 pip install git+https://github.com/fidelity/spock

docs/Motivation.md

@@ -87,6 +87,8 @@ set of parameters.
 * Tractability and Reproducibility: Save runtime parameter configuration to YAML, TOML, or JSON with a single chained 
   command (with extra runtime info such as Git info, Python version, machine FQDN, etc). The saved markdown file can be
   used as the configuration input to reproduce prior runtime configurations.
+* Hyper-Parameter Tuner Addon: Provides a unified interface for hyper-parameter tuning that supports various 
+  backends (Optuna, WIP: Ax)
 * S3 Addon: Automatically detects `s3://` URI(s) and handles loading and saving `spock` configuration files when an
   active `boto3.Session` is passed in (plus any additional `S3Transfer` configurations)
 

docs/Quick-Start.md

@@ -107,7 +107,7 @@ fancier_parameter: 64.64
 most_fancy_parameter: [768, 768, 512, 128]
 ```
 
-Finally, we would run our script and pass the path to the configuration file to the command line (-c or --config):
+Finally, we would run our script and pass the path to the configuration file to the command line (`-c` or `--config`):
 
 ```bash
 $ python simple.py -c simple.yaml
@@ -131,4 +131,16 @@ configuration(s):
     fancy_parameter         float        parameter that multiplies a value 
     fancier_parameter       float        parameter that gets added to product of val and fancy_parameter 
     most_fancy_parameter    List[int]    values to apply basic algebra to 
+```
+
+### Spock As a Drop In For Argparser
+
+`spock` can easily be used as a drop in for argparser. This means that all parameter definitions as required to come in 
+from the command line or from setting defaults within the `@spock` decorated classes. Simply do not pass a `-c` or 
+`--config` argument at the command line and instead pass in all of the automatically generated cmd-line arguments.
+
+
+```bash
+$ python simple.py --BasicConfig.parameter --BasicConfig.fancy_parameter 8.8 --BasicConfig.fancier_parameter 64.64 \
+  --BasicConfig.most_fancy_parameter [768, 768, 512, 128]
 ```

docs/addons/S3.md

@@ -46,11 +46,11 @@ session = boto3.Session(
 
 ### Using the S3Config Object
 
-As an example let's create a basic `@spock` decorated class, instantiate a `S3Config` object from `spock.addons` with
+As an example let's create a basic `@spock` decorated class, instantiate a `S3Config` object from `spock.addons.s3` with
 the `boto3.session.Session` we created above, and pass it to the `ConfigArgBuilder`.
 
 ```python
-from spock.addons import S3Config
+from spock.addons.s3 import S3Config
 from spock.builder import ConfigArgBuilder
 from spock.config import spock
 from typing import List
@@ -123,8 +123,8 @@ With a `S3Config` object passed into the `ConfigArgBuilder` the S3 URI will auto
 If you require any other settings for uploading or downloading files from S3 the `S3Config` class has two extra 
 attributes:
 
-`download_config` which takes a `S3DownloadConfig` object from `spock.addons` which supports all ExtraArgs from
+`download_config` which takes a `S3DownloadConfig` object from `spock.addons.s3` which supports all ExtraArgs from
 [S3Transfer.ALLOWED_DOWNLOAD_ARGS](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_DOWNLOAD_ARGS)
 
-`upload_config` which takes a `S3UploadConfig` object from `spock.addons` which supports all ExtraArgs from
+`upload_config` which takes a `S3UploadConfig` object from `spock.addons.s3` which supports all ExtraArgs from
 [S3Transfer.ALLOWED_UPLOAD_ARGS](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS)

docs/addons/tuner/About.md

@@ -0,0 +1,22 @@
+# Hyper-Parameter Tuning Support
+
+This series of docs will describe the basics of hyper-parameter support within `spock`. `spock` tries to be as hands-off 
+as possible with the underlying backends that support hyper-parameter tuning and only provide a common and simplified
+interface to define hyper-parameter tuning runs. The rest is left up to the user to define and handle, thus to not
+handcuff the user into too simplified functionality.
+
+All examples can be found [here](https://github.com/fidelity/spock/blob/master/examples).
+
+### Installing
+
+Install `spock` with the extra hyper-parameter tuning related dependencies.
+
+```bash
+pip install spock-config[tune]
+```
+
+### Supported Backends
+* [Optuna](https://optuna.readthedocs.io/en/stable/index.html)
+
+### WIP/Planned Backends
+* [Ax](https://ax.dev/)