environment management and packaging tools

[return42]

About environment management and packaging tools I recommend to read An unbiased evaluation of environment management and packaging tools from Anna-Lena Popkes .. or better listen to to her talk at PyCon DE 2023.

Related to discussions about:

• pre-commit: Added linters and pre-commit hooks #36 (comment)
• and Makefile: Improvement in code quality before the 0.9.x branch was forked #39 (comment)

---

To demonstrate how it could look like using Hatch, I added a PR on my fork:

• [hatch] replace Makefile by Hatch, a project management tool return42/vobject#2

[rsb-23]

@da4089 What are the requirements and its scope for current phase?
Post that, it will be easy to finalize on the tools and also take each part in separate PRs.

• Compatibility support (already finalized. 2.7, 3.7+)
• Static code analysis
• Unittest and coverage
• Package build / publish
• Environment management

[rsb-23]

Currently, I am only looking into static code and unittest parts to be handled at pre-commit, push and pull-request (3) stages.

[return42]

I have two minor concerns about the tooling: introducing a dependency on asdf (I haven't even heard of this before today), and requiring make which might be difficult for contributors running Windows (and not familiar with/willing to use WSL).

• If we have just 2-3 scripts to run all the tools, it won't be a problem in introducing dependencies.

In my POC no tools are needed .. where scripting is needed, its implemented in the Hatch environments ...

To demonstrate how it could look like using Hatch, I added a PR on my fork:

• [hatch] replace Makefile by Hatch, a project management tool return42/vobject#2

• Docker seems a good way to handle OS compatibility concerns and run tests for all python versions locally. POC {check extras & script folder too}
• All configs should be in pyproject.toml (single file). [We can remove .pylintrc and wherever possible]

This is not possible as long the py2 compatibility is in the master branch .. when we split py2 compatibility into a branch 0.9.x everything becomes much easier / but ATM and in my POC we have py2 in the master branch.

• We should keep "lint checks" and "unit tests" separate in CI. lint action. Also, we can separate for main branches.

Yes / see github actions in my POC or in the implementation

• Since linting doesn't impact code working, we should run linters only for python 3.8 and suppress the conflicting changes. It won't be an issue until unit tests pass for all py versions.

At the moment I am implementing liniting in my POC which is backward compatible (because py2 & py3 are in the master) ... but this is very time consuming, once we have forked the py2 branch it will be much easier and the development in the master does not need to be py2 compatible anymore as all required changes have to be merged manually to py2.

This py2 compatibility makes everything very complicated, we should stick to @da4089 's plan. The plan was to make the mass changes in the master branch and then fork as quickly as possible.

• Compatibility support (already finalized. 2.7, 3.7+)
• Static code analysis
• Unittest and coverage
• Package build / publish
• Environment management

Note: all this is in my POC / return42#2 / the POC consists only of this singe commit : return42@6e256ea

The usage is very simple / see chapter "Development" in the README.md

[rsb-23]

Just to be on same page, I'll mention my context again.

1. I have shared my feedback based on assumption that we will be linting py27 manually.
2. black, flake8, pylint, make, hatch, docker etc. are referred as tools.
3. My idea was not to control too much of developers' config and keep things basic/ simple.
   (I know simple is very subjective and differs to each dev. Ex: for me, pylint is not maintainable due to too many checks)
4. In linting point, I meant we can run lint-checks just once with py38 and skip py37, py39, etc. as they are unnecessary.
5. All changes should be smooth enough for the owner to maintain for long.

@da4089
There is never one perfect solution in programming. We (me & @return42 ) have shared our perspectives. I hope, we were able to address all your queries.
Now you can decide on which trade-offs to take and action plan to execute; I'll raise new PRs as required.

[return42]

Ex: for me, pylint is not maintainable due to too many checks

OK, this was not clear to me / thanks for clarifying / Personally I prefer these many checks: Pylint is a static code analyser, this tool finds a lot of bugs a developer might not see / not reported by flake8 .. / but this is my personal opinion.

About the other points I mostly agree with you / may I have a different view in some details and what comes first, but mostly I agree.

[da4089]

First up, thank you both for all your work on this topic: it's great to have your expertise and time applied to this project!

I had originally hoped that it might be possible to have the same automation in the 2.x and 3.x branches, but I now think that's not worth the effort (and I think we all agree on that?)

So, I think I'd like to see a first PR that does:

• Format the code (with black)
  • Either use an older version of Black, or VERY CAREFULLY audit the diff to ensure that it doesn't break the 2.x Unicode/string differences
• Fix the basic pylint issues
• Merge this ASAP
  • I think we have candidate patches for this already, but perhaps mixed in with other stuff. Let's cherry pick some patches into a new PR, and leave out eg. pyproject.toml, etc.

Then, make a few other minor cleanups:

• Embed version number in __init__.py
• Remove PyICU (and update python-dateutil dependency)
• Fix project metadata at PyPI

Ship 0.9.8, and make the v0.9.x branch

Open issues:

• Leave unit testing as a manual task for v0.9.x branch?
• Don't bother with coverage?
• Continue to use setuptools, setup.py, and twine for building and publishing?

Then we can move forward with changes to master branch, adding automation, CI tools, etc. I'd like to get that stuff merged from the various PRs / discussions we have in progress too. I'm happy to have master be a little messy for a time while we get some experience with the new tools, and make decisions about how they're configured, or switch out for something else, etc.

How's that sound?

[da4089]

Refocusing on the tooling for master branch:

I wish I had some good intuition as to what will happen with Python packaging. I've had years of using setup.py (with distutils and now setuptools), but I accept that way is obsolete. What I haven't understood is the best way forward.

I've played a little with flit and hatch, not really with PDM, a little with poetry, and I've started using build.py. I've not used rye, nor anything other than vanilla pip + venv (and virtualenv before that).

I tend to do my development with a bunch of venv directories: venv27, venv37, venv312, and I will generally manually switch between them (deactivate/./bin/venvXX/activate) when I want to test something out. If we have GitHub Actions' CI stuff set up properly, I think it should be reasonable for people to work in the lowest targeted version venv (ie. 3.7 until June, then 3.8), and when submitting a PR the automation should flag any problems across different operating systems, different interpreter versions, run the static analysis, style checks, coverage reports, etc.

I want to have the ability to run those things locally, so that when the automation picks up a problem, it's easy for me to iterate locally (I don't want to have to send patches to the PR trying things randomly until it passes). I also want to support developers using any (reasonable) operating system (Win, Mac, Linux, more or less), so I'm keen to keep the tooling platform agnostic.

I am happy to experiment with different tools until we find something (or some combination) that works for us. That said, I'd rather pick something and stick with it unless there's something clearly better: I don't want to waste time and effort chasing the latest and greatest for minimal extra value.

I'll have a look at the Hatch stuff @return42, but maybe once we've got the branching done.

[return42]

Refocusing on the tooling for master branch:
I'll have a look at the Hatch stuff @return42, but maybe once we've got the branching done.

Agreed, let's branch out first 👍