Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build: Demo of no-setup Makefile (standardized virtualenv handling) #347

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

build: Demo of no-setup Makefile (standardized virtualenv handling) #347

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
245e7bf
build: Demo of no-setup Makefile (standardized virtualenv handling)
timmc-edx Sep 22, 2023
37c3a4f
Move inclusion to top; use lower case for internal variables
timmc-edx Sep 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
__pycache__
.pytest_cache

/.venv-3.8

Comment on lines +5 to +6
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Author's note: In most repositories, tools like pytest are already configured such that they don't try to operate in the files inside the virtualenv since we call them with explicit paths, e.g. pylint edx_django_utils tests test_utils manage.py setup.py.

A few repos would need to be adjusted to follow this pattern. These repos are already a problem for people who use in-repo virtualenvs.

There would also need to be some adjustment for code searching. Developers who are used to using grep -nr ... would need to instead write grep -nr ... *. Same for ack, although adding --ignore-dir=match:/^.?venv-[0-9.]+$/ to one's ~/.ackrc will prevent searching any such virtualenv. Those who use ripgrep would already be covered, as ripgrep ignores hidden dirs by default.

# C extensions
*.so

Expand Down
66 changes: 33 additions & 33 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
include repo-tools/base.mk

.PHONY: clean compile_translations coverage diff_cover docs dummy_translations \
extract_translations fake_translations help pull_translations push_translations \
quality requirements selfcheck test test-all upgrade validate
Expand All @@ -20,63 +22,61 @@ help: ## display this help message
@echo "Please use \`make <target>' where <target> is one of"
@perl -nle'print $& if m{^[a-zA-Z_-]+:.*?## .*$$}' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m %-25s\033[0m %s\n", $$1, $$2}'

clean: ## remove generated byte code, coverage reports, and build artifacts
clean: $(coverage) ## remove generated byte code, coverage reports, and build artifacts
find . -name '__pycache__' -exec rm -rf {} +
find . -name '*.pyc' -exec rm -f {} +
find . -name '*.pyo' -exec rm -f {} +
find . -name '*~' -exec rm -f {} +
coverage erase
$(coverage) erase
rm -fr build/
rm -fr dist/
rm -fr *.egg-info

coverage: clean ## generate and view HTML coverage report
pytest --cov-report html
coverage: clean $(pytest) ## generate and view HTML coverage report
$(pytest) --cov-report html
$(BROWSER) htmlcov/index.html

docs: ## generate Sphinx HTML documentation, including API docs
tox -e docs
docs: $(tox) ## generate Sphinx HTML documentation, including API docs
$(tox) -e docs
$(BROWSER) docs/_build/html/index.html

upgrade: export CUSTOM_COMPILE_COMMAND=make upgrade
upgrade: ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in
pip install -qr requirements/pip-tools.txt
upgrade: $(pip) $(pip-sync) ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in
$(pip) install -qr requirements/pip-tools.txt
# Make sure to compile files after any other files they include!
pip-compile --allow-unsafe --rebuild --upgrade -o requirements/pip.txt requirements/pip.in
pip-compile --no-emit-trusted-host --upgrade -o requirements/pip-tools.txt requirements/pip-tools.in
pip install -qr requirements/pip.txt
pip install -qr requirements/pip-tools.txt
pip-compile --no-emit-trusted-host --upgrade -o requirements/base.txt requirements/base.in
pip-compile --no-emit-trusted-host --upgrade -o requirements/test.txt requirements/test.in
pip-compile --no-emit-trusted-host --upgrade -o requirements/doc.txt requirements/doc.in
pip-compile --no-emit-trusted-host --upgrade -o requirements/quality.txt requirements/quality.in
pip-compile --no-emit-trusted-host --upgrade -o requirements/ci.txt requirements/ci.in
pip-compile --no-emit-trusted-host --upgrade -o requirements/dev.txt requirements/dev.in
$(pip-compile) --allow-unsafe --rebuild --upgrade -o requirements/pip.txt requirements/pip.in
$(pip-compile) --no-emit-trusted-host --upgrade -o requirements/pip-tools.txt requirements/pip-tools.in
$(pip) install -qr requirements/pip.txt
$(pip) install -qr requirements/pip-tools.txt
$(pip-compile) --no-emit-trusted-host --upgrade -o requirements/base.txt requirements/base.in
$(pip-compile) --no-emit-trusted-host --upgrade -o requirements/test.txt requirements/test.in
$(pip-compile) --no-emit-trusted-host --upgrade -o requirements/doc.txt requirements/doc.in
$(pip-compile) --no-emit-trusted-host --upgrade -o requirements/quality.txt requirements/quality.in
$(pip-compile) --no-emit-trusted-host --upgrade -o requirements/ci.txt requirements/ci.in
$(pip-compile) --no-emit-trusted-host --upgrade -o requirements/dev.txt requirements/dev.in
# Let tox control the Django version for tests
sed '/^[dD]jango==/d' requirements/test.txt > requirements/test.tmp
mv requirements/test.tmp requirements/test.txt

isort: ## fixes isort issues found during quality check
tox -e isort
isort: $(tox) ## fixes isort issues found during quality check
$(tox) -e isort

quality: ## check coding style with pycodestyle and pylint
tox -e quality
$(tox) -e quality

requirements: ## install development environment requirements
pip install -r requirements/pip.txt
pip install -qr requirements/pip-tools.txt
pip-sync requirements/dev.txt requirements/private.*
pip install . # CLI entry points
requirements: $(pip) $(pip-sync) ## install development environment requirements
$(pip-sync) requirements/dev.txt requirements/private.*
$(pip) install . # CLI entry points

test: clean ## run tests in the current virtualenv
pytest
test: clean $(pytest) ## run tests in the current virtualenv
$(pytest)

diff_cover: test ## find diff lines that need test coverage
diff-cover coverage.xml
diff_cover: test $(diff-cover) ## find diff lines that need test coverage
$(diff-cover) coverage.xml

test-all: ## run tests on every supported Python/Django combination
tox -e quality
tox
test-all: $(tox) ## run tests on every supported Python/Django combination
$(tox) -e quality
$(tox)

validate: quality test ## run tests and quality checks

Expand Down
38 changes: 38 additions & 0 deletions repo-tools/base.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Base Makefile components to build on.
#
# Ideally, this would be pulled in from another repo by a `git subtree`
# reference and would include most of the Makefile so that upgrading
# repos to the latest functionality (and keeping them standardized)
# would be fast and simple.

PYTHON_VER := 3.8

# These are required for bootstrapping a virtualenv to the point where
# requirements can be installed.

venv_name := .venv-$(PYTHON_VER)
pip := $(venv_name)/bin/pip
pip-compile := $(venv_name)/bin/pip-compile
pip-sync := $(venv_name)/bin/pip-sync

$(venv_name): ## Create the virtualenv, or recreate it if it already exists
python$(PYTHON_VER) -m venv $(venv_name) --clear

$(pip): $(venv_name)
pip install -r requirements/pip.txt

$(pip-compile) $(pip-sync): $(venv_name)
$(pip) install -r requirements/pip-tools.txt

# The rest of the Python tools are listed here, and all are fulfilled
# by the same target. By calling them from their explicit virtualenv
# paths, we ensure that the virtualenv will be used (including by
# other tools they call).

coverage := $(venv_name)/bin/coverage
diff-cover := $(venv_name)/bin/diff-cover
pytest := $(venv_name)/bin/pytest
tox := $(venv_name)/bin/tox

$(venv_name)/bin/%: $(venv_name)
make requirements
Loading