Use pip-tools to handle Python deps

[matiasb]

pip-tools helps making builds deterministic, controlling deps (and indirect deps) upgrades (and versions consistency) avoiding unexpected (and potentially breaking) changes.

We keep our direct deps in requirements.in from which we generate the requirements.txt (where all deps are pinned). We also constrain dev (and enterprise) deps based on base requirements.

Check how to update deps.

[mderynck]

Is the change for pip also needed in drone.yml in (Lint + Unit Test Backend) and snyk.yml?

[matiasb]

Is the change for pip also needed in drone.yml in (Lint + Unit Test Backend) and snyk.yml?

Not needed as long as we don't update/change requirements.txt in any way (requirements.txt will now include every dep and subdeps pinned; pip install keeps working, main diff is pip-sync clears the env in the process too, ensuring only the listed deps are installed as a final result). The main change in our flow would be to add/change/update deps using pip tools (requirements.in + pip-compile) to ensure we keep everything pinned and no unexpected upgrades happen behind the scenes (I was expecting to discuss this a bit in our call today (we discussed introducing this in the past too), but there was no time; maybe next week we can sync, backend-side).

[matiasb]

@matiasb does this change how we install deps or pip install -r requirements.txt is still the recommended way?

pip install -r requirements.txt will keep working (the difference is that this should never bring additional dependencies or different versions not listed there), and when starting an env from scratch, it would be the same as running pip-sync. pip-sync on the other hand, will also ensure to clean up any deps not listed in the requirements file, keeping the env exactly as described in requirements.txt (no urgent need to switch to this, and since we need to handle OSS and private requirements when building images/deploying, it still makes sense to keep using pip install in those contexts).
What we would need to change is how we add/update dependencies to our environment, making the changes in the requirements.in file instead, and compiling it (pip-compile) before pushing.