build: Demo of no-setup Makefile (standardized virtualenv handling)

[timmc-edx]

This allows a developer to run make test or other targets without having to manually set up a virtualenv or install requirements. The virtualenv is set up for them as a hidden subdirectory and requirements are automatically installed as needed.

The goal here is larger: The new base.mk would be pulled in via git subtree from some centralized repository. Over time, this standardized base Makefile would grow and the Makefiles in regular repositories would shrink. By centralizing Makefile logic, we would gain the following:

• Standardized names for make targets. Currently there is no universal way to install Python dependencies, to run all tests and quality checks, or to build docs. Developers have to learn what is used in each repo. We have standards for some of this, but they're not always followed. By composing Makefiles, we can ensure new repos always follow the standard, even as the standard changes.
• Faster onboarding of developers who have not yet set up their own personalized virtualenv toolchains. Experienced Python developers already have these tools set up, but newer developers stumble on virtualenv creation.
• A way to update repos to the latest Makefile improvements without having to manually update several hundreds of repos by hand. Automating the update of a git subtree reference still requires making and merging PRs but it does not require manual review of each PR, let alone manual PR creation.

[timmc-edx]

Author's note: In most repositories, tools like pytest are already configured such that they don't try to operate in the files inside the virtualenv since we call them with explicit paths, e.g. pylint edx_django_utils tests test_utils manage.py setup.py.

A few repos would need to be adjusted to follow this pattern. These repos are already a problem for people who use in-repo virtualenvs.

There would also need to be some adjustment for code searching. Developers who are used to using grep -nr ... would need to instead write grep -nr ... *. Same for ack, although adding --ignore-dir=match:/^.?venv-[0-9.]+$/ to one's ~/.ackrc will prevent searching any such virtualenv. Those who use ripgrep would already be covered, as ripgrep ignores hidden dirs by default.

[robrap]

@timmc-edx:

1. I think you also had an idea for how you could use a shared docker file if one were wanted. Would that be a separate PR?
2. I imagine getting the right Python version is still an issue for a new developer. Are there any ways to help there, or is that one advantage that a container would always have? Maybe that brings us back to point 1.

[timmc-edx]

I think the main use for a shared Docker image would be getting the right version of Python, yeah. That might be a separate PR, but it looks like we could add something like this to the Makefile:

SHELL := docker run --rm -v .:/edx/dev -w /edx/dev IMAGE_REF /bin/bash -c

with an image based on a Dockerfile that looks something like this:

FROM ubuntu:focal as minimal-system

RUN useradd -m --shell /bin/false app
WORKDIR /edx/dev
RUN chown app:app /edx/dev

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
    apt-get -y dist-upgrade && \
    apt-get -y install --no-install-recommends \
        # Base Python system
        python3.8 \
        python3.8-venv

USER app

Seems like it works based on some local testing -- make requirements runs on the host, but all of the actual commands run in the container. It could be made switchable so that the container is not required.

[timmc-edx]

As a branch: timmc/demo-venv...timmc/demo-make-container