Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not freeze requirements #120

Merged
merged 1 commit into from
Feb 22, 2021

Conversation

minrk
Copy link
Member

@minrk minrk commented Feb 22, 2021

reverts #116, #117

Freezing requirements is for images or applications, not for libraries.

Sorry for not catching this in review! It is not generally appropriate to use dependabot and frozen requirements with packages. This creates unsolvable dependencies, as each package would have strict dependencies, but they would be different for each package depending on when they ran

In general, for libraries you want to only apply pinnings when:

  • set lower bound for oldest supported version (i.e. introducing a new feature you use)
  • explicitly block known-incompatible versions (e.g. specific broken releases of dependencies)
  • only set an upper bound temporarily after discovering that a new release has/will break your package and you need more time to fix it.

Javascript-style semver-pinning doesn't make sense in Python libraries, where they rapidly create impossible-to-install environments and force a massive increase in maintenance and releases.

freezing requirements is for images, applications. Not for libraries
@minrk minrk force-pushed the unfreeze-requirements branch from ffd83f2 to 122c163 Compare February 22, 2021 09:50
@minrk minrk requested a review from GeorgianaElena February 22, 2021 09:51
@manics
Copy link
Member

manics commented Feb 22, 2021

An approach I've taken in some of my projects is to have general requirements in setup.py, and pinned dependencies in dev-requirements.txt which are bumped by dependabot. This gives the benefit of reproducible dev environments for testing and also alerts you to new releases of dependencies, without imposing restrictions on users.

@GeorgianaElena
Copy link
Member

Freezing requirements is for images or applications, not for libraries.

That's a bummer :( I so liked the idea of having new releases alerts and PRs from dependabot.
But I'm super happy you noticed this before me releasing a new version of traefik-proxy 😅 Thank you, @minrk <3

An approach I've taken in some of my projects is to have general requirements in setup.py, and pinned dependencies in dev-requirements.txt which are bumped by dependabot.

@manics, this sounds like a great idea for discovering possible incompatibilities with newer pkg version, before somebody bumps into it and opens a bug report issue.

But, to be honest, traefik-proxy doesn't have so many direct dependencies, and the ones that could cause big issues are not updated very often (the python clients for etcd and consul). But this also means that if one of their dependencies is updated and becomes incompatible, we should pin versions until the python client releases a new version (this is what happened with aiohttp and consul python client).

@minrk, do you think we should use this for traefik-proxy or it might be an overkill for this project?

@GeorgianaElena
Copy link
Member

I'll go ahead and merge this now. Thanks again @minrk.

@GeorgianaElena GeorgianaElena merged commit 4cc5c4c into jupyterhub:master Feb 22, 2021
@consideRatio
Copy link
Member

Ohhh yeah that does make sense =/ Thank you @minrk!! I'm sorry for sidetracking you @GeorgianaElena but I'm glad you ended up appreciating the general strategy!

@minrk minrk deleted the unfreeze-requirements branch February 22, 2021 12:02
@minrk
Copy link
Member Author

minrk commented Feb 22, 2021

For the reproducible test env strategy, I personally like to not pin dependencies, and in fact install the latest pre-releases of dependencies with pip install --upgrade --pre. This way you get notified of incompatibilities, ideally during the beta cycle of dependencies, before they reach any production environments. The downside, solved by @manics approach, is that these can occur in otherwise-unrelated PRs, which can be frustrating and confusing if they happen often.

You can combine these approaches, by having the 'always-latest' tests run in one matrix entry and dependabot-frozen reproducible envs run in a different matrix entry. I'm not sure if that's worth it here or not. Additionaly, the number of prereleases we are likely to depend on here is pretty low, so the dependabot approach is almost identical.

I checked, and couldn't see an obvious way to make dependabot include --pre results. Anyone know if that's possible (seems like no? If it is, dependabot checks would have a big benefit on a low-traffic repo like this one because we would be notified of incompatible dependency changes when they are published, rather than when we happen to submit a PR here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants