Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster cloning #29

Merged
merged 10 commits into from
Dec 5, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions .github/workflows/docker_tests.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Run python unit tests

name: docker tests

on:
push:
schedule:
-
cron: "0 0 * * 1"
branches: [ $default-branch ]

jobs:
unittests:
runs-on: ubuntu-latest
steps:
-
name: Checkout code
uses: actions/checkout@v3
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
-
name: Login to DockerHub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Build and push
uses: docker/build-push-action@v3
with:
context: .
target: dev
push: false
tags: ${{ secrets.REPO_NAME }}:gha
cache-from: type=registry,ref=${{ secrets.DOCKERHUB_USERNAME }}/${{ secrets.REPO_NAME }}:cache
cache-to: type=registry,ref=${{ secrets.DOCKERHUB_USERNAME }}/${{ secrets.REPO_NAME }}:cache,mode=max
outputs: type=docker
-
name: Unit tests
run: docker run ${{ secrets.REPO_NAME }}:gha pytest
-
name: Style tests
run: docker run ${{ secrets.REPO_NAME }}:gha pre-commit run --all-files
35 changes: 16 additions & 19 deletions .github/workflows/tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,28 +17,25 @@ jobs:
name: Checkout code
uses: actions/checkout@v3
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
-
name: Login to DockerHub
uses: docker/login-action@v2
name: Set up Python
uses: actions/setup-python@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
python-version: "3.10"
-
name: Build and push
uses: docker/build-push-action@v3
with:
context: .
target: dev
push: false
tags: ${{ secrets.REPO_NAME }}:gha
cache-from: type=registry,ref=${{ secrets.DOCKERHUB_USERNAME }}/${{ secrets.REPO_NAME }}:cache
cache-to: type=registry,ref=${{ secrets.DOCKERHUB_USERNAME }}/${{ secrets.REPO_NAME }}:cache,mode=max
outputs: type=docker
name: Install requirements
run: |
python -m pip install --upgrade pip
python -m pip install --upgrade build
python -m pip install -r requirements/prod.txt
python -m pip install -r requirements/dev.txt
-
name: Build
run: |
python -m build
python -m pip install -e . --no-deps
-
name: Unit tests
run: docker run ${{ secrets.REPO_NAME }}:gha pytest
run: pytest
-
name: Style tests
run: docker run ${{ secrets.REPO_NAME }}:gha pre-commit run --all-files
run: pre-commit run --all-files
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -88,10 +88,10 @@ repos:
# - id: setup-cfg-fmt

- repo: https://github.com/pre-commit/mirrors-mypy
rev: v0.971
rev: v0.982
hooks:
- id: mypy
args: []
args: [--show-error-codes]
additional_dependencies:
- numpy

Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.10.6-slim-buster as base
FROM python:3.10.8-slim-buster as base

# Install things
RUN apt-get update
Expand Down
60 changes: 35 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,34 @@ This repository is a template for future repositories. Features:
- Can be packaged with `pip`
- Working `pytest` tests in `tests` directory
- Install with `requirements/prod.txt` and `requirements/dev.txt`
- environment installed inside a Docker Container
- Environment installed inside a Docker Container
- `README.md` file with repeatable instructions
- Style checks using `flake8`, `mypy`, and `black` bundled into a single `pre-commit` action
- GitHub Actions automates style and unit tests across matrixed Python versions
- Uses Python 3.10 because stable [TensorFlow](https://www.tensorflow.org/install/pip) doesn't yet support Python 3.11

## Develop in Docker
## Develop without Docker
1. Create a Python virtual environment in `3.10.8`
```
python -m venv <environment-name>
```
2. In this virtual environment, install necessary requirements files
```
python -m pip install -r requirements/dev.txt
python -m pip install -r requirements/prod.txt
```
3. Build and install
```
python -m build
python -m pip install -e . --no-deps
```
4. Test things
```
pre-commit run --all-files
pytest
```

## Develop with Docker
Code development is in a Docker image, use these steps to spin up the image
1. Download and install [Docker](https://docs.docker.com/engine/install/)
2. Clone this repository
Expand All @@ -31,32 +52,21 @@ Code development is in a Docker image, use these steps to spin up the image
exit
```

## Develop without Docker
1. Create a Python virtual environment
```
python3 -m venv <environment-name>
```
2. In this virtual environment, install necessary requirements files
```
python -m pip install -r requirements/dev.txt
python -m pip install -r requirements/prod.txt
```
3. Build and install
```
python -m build
python -m pip install -e . --no-deps
```
4. Test things
```
pre-commit run --all-files
pytest
```

## Updating requirements folder
## Updating requirements directory
1. Use pip-compile to build a new pinned requirements file.
```
pip-compile requirements/prod.in --output-file=requirements/prod.txt
pip-compile requirements/dev.in --output-file=requirements/dev.txt
```
2. If using Docker, rebuild the container using the commands above.
```
exit
```
2. Rebuild the container using the commands above.

## When cloning this template:
1. Change the python version across config files
2. Change the Docker image names
3. Docker GitHub Actions will fail until certain Secrets are uploaded
1. `REPO_NAME`: The name of the Docker repository
2. `DOCKERHUB_USERNAME`: The DockerHub username
3. `DOCKERHUB_TOKEN`: A token for access to the repository
2 changes: 1 addition & 1 deletion requirements/dev.in
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
black==22.6.0
identify>=2.5.3,<3
mypy>=0.971,<1
mypy>=0.982,<1
pip-tools>=6.8.0,<7
pre-commit>=2.20.0,<3
pytest>=7.1.2,<8
2 changes: 1 addition & 1 deletion requirements/dev.txt
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ identify==2.5.3
# pre-commit
iniconfig==1.1.1
# via pytest
mypy==0.971
mypy==0.982
# via -r requirements/dev.in
mypy-extensions==0.4.3
# via
Expand Down