Adhere to Python Naming Conventions

[kgizdov]

I've noticed a trend that changing the major version of the uproot library results in repo name changes, source tree name changes, packaging changes, hardcoded test string changes and others, which is unnecessary and makes it very difficult to work with the package.

I don't see why uproot doesn't just do what every other package, whether Python or not, does - major versions with breaking changes are listed, and people who rely on it limit their requirements.

This is also extensively covered in PEP rules.

On top of all that, it goes against the principles behind having a centralised version control system. Major version histories get disconnected, it's impossible to follow the changes, etc. But I think the biggest, in my opinion, is it discourages people from contributing because it's cumbersome.

Even complete rewrites across major versions do not necessitate new repositories if it's the same project. It would make life much easier for users and contributors if uproot remained uproot everywhere and only its version was changed. :)

[Moelf]

I second this just because I come from an ecosystem that uses semVer so major release is "just another release" on GitHub.

But otherwise I don't see it as a big deal

[jpivarski]

After the mistake with Uproot 3 → 4, the PyPI package and GitHub repo have not changed. Uproot 4 and Uproot 5 will use the same PyPI package and do use the same GitHub repo as each other; only the major version number changes. The GitHub repo name was changed from "uproot4" to "uproot5" with redirects (URLs don't have to change) because the name "uproot4" is misleading and "uproot" is unavailable, due to it being a redirect to "uproot3".

(Incidentally, there was reasoning behind the Uproot 3 → 4 mistake, though on the whole it was a bad thing to do. Having two PyPI names made it possible to load both major versions in the same Python session, a feature that I already needed and could not get with Uproot 4 → 5 because I'm following this best practice. Of course, giving up on that is the lesser of two evils. The git repo was changed in Uproot 3 → 4 because Uproots 1 → 2 → 3 had too many commits, some of them involving large files, and that made cloning the repo slow. That problem could have been better fixed by manipulating git history, which I generally try to avoid.)

But anyway, this is not a trend because Uproot 4 → 5 does not change PyPI names or git repos.

[kgizdov]

It does change the git repos:

==> Retrieving sources...
  -> Downloading uproot-4.3.5...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0    14    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
==> ERROR: Failure while downloading https://github.com/scikit-hep/uproot4/archive/v4.3.5.tar.gz
    Aborting...
==> ERROR: Failed to generate new checksums

Maybe people are unaware, but GitHub now has an entirely new repository and only an HTML 301 web redirect, nothing else. No repo mirroring is happening.

And I'm afraid this will have more yet unforeseen side effects. This v5 should have been a branch, not a new repo, a rename, or something else of this sort.

I raised the same issue when this project was in its infancy and again when the messy 2->3 and 3-4 versions happened, and at no point, the reasons or even the final goal justified this sort of thing. Git branches and package versioning exist for a reason.

[jpivarski]

Not just for Awkward/Uproot, but for other Scikit-HEP projects, it's common to move a git repo to a new URL. The primary use-case for that is when someone develops a project under their own account, github-user/awesome-project, and then wants to transfer it to Scikit-HEP: scikit-hep/awesome-project. This may be a new git repo (not two mirrored git repos), but it preserves the git history, all the GitHub metadata, and sets up redirect links from old to new.

I just tried downloading v4.3.5.tar.gz from uproot4 and uproot5, and indeed there's a redirect there:

% wget https://github.com/scikit-hep/uproot5/archive/refs/tags/v4.3.5.tar.gz              
--2022-09-23 09:38:35--  https://github.com/scikit-hep/uproot5/archive/refs/tags/v4.3.5.tar.gz
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/scikit-hep/uproot5/tar.gz/refs/tags/v4.3.5 [following]
--2022-09-23 09:38:36--  https://codeload.github.com/scikit-hep/uproot5/tar.gz/refs/tags/v4.3.5
Resolving codeload.github.com (codeload.github.com)... 140.82.113.10
Connecting to codeload.github.com (codeload.github.com)|140.82.113.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘v4.3.5.tar.gz’

v4.3.5.tar.gz                   [ <=>                                        ] 788.19K  4.48MB/s    in 0.2s    

2022-09-23 09:38:36 (4.48 MB/s) - ‘v4.3.5.tar.gz’ saved [807110]

and

% wget https://github.com/scikit-hep/uproot4/archive/refs/tags/v4.3.5.tar.gz 
--2022-09-23 09:38:54--  https://github.com/scikit-hep/uproot4/archive/refs/tags/v4.3.5.tar.gz
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/scikit-hep/uproot5/archive/refs/tags/v4.3.5.tar.gz [following]
--2022-09-23 09:38:54--  https://github.com/scikit-hep/uproot5/archive/refs/tags/v4.3.5.tar.gz
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/scikit-hep/uproot5/tar.gz/refs/tags/v4.3.5 [following]
--2022-09-23 09:38:54--  https://codeload.github.com/scikit-hep/uproot5/tar.gz/refs/tags/v4.3.5
Resolving codeload.github.com (codeload.github.com)... 140.82.113.10
Connecting to codeload.github.com (codeload.github.com)|140.82.113.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘v4.3.5.tar.gz’

v4.3.5.tar.gz                   [ <=>                                        ] 788.19K  4.38MB/s    in 0.2s    

2022-09-23 09:38:54 (4.38 MB/s) - ‘v4.3.5.tar.gz’ saved [807110]

As you point out, there's a HTTP 301: moved permanently in the logs. But that's what we want: HTTP clients know to follow redirects to a new URL, just like they have to follow the HTTP 302: found, which happens for both URLs.

One rule that we adhere to when we use GitHub's "rename repo" feature is that we don't move a repo into a URL that has ever been occupied by another repo, because then URLs to issue and PR numbers would change their meaning. Renames need to go to a URL that has never been used before. This is why "scikit-hep/uproot4" couldn't become just "scikit-hep/uproot", as I would have really, really preferred, and why "scikit-hep/awkward-1.0" couldn't become "scikit-hep/awkward-array", though the solution of using "scikit-hep/awkward" is a satisfying one (repo name is the same as the PyPI and conda-forge package name).

Versions and git-branches are best practices, and we use those. I had no reason to suspect (still don't) that GitHub's "rename repo" feature is something to be avoided, provided that we rename to a new URL that has never been used. I wouldn't expect it to make a mirror repo (which would probably be worse than a redirect, in terms of keeping distributed data consistent), I expect it to make HTTP redirects, which it does, and I don't see what's wrong with HTTP redirects.

If there is something wrong with HTTP redirects, then we would also need to rethink how we adopt github-user/awesome-project into scikit-hep/awesome-project when new projects join Scikit-HEP. Keeping them always under personal github-user accounts is not a good way for projects to evolve from single-author ideas into community-maintained resources.

I raised the same issue when this project was in its infancy and again when the messy 2->3 and 3-4 versions happened, and at no point, the reasons or even the final goal justified this sort of thing. Git branches and package versioning exist for a reason.

(Not 2 → 3; that was an ordinary version bump from 2.9.11 to 3.0.0b1, and it happened in August 2018.)

For reference, here is your message from February 2021, when you raised this issue before. I gave a long explanation about why we did that, which ended with

I think I wouldn't do it that way again.

And so we didn't. Uproot version 5 is right now a git branch (main versus the legacy main-v4) and it is being released as an rc series, starting with 5.0.0rc1 (I just released 5.0.0rc3 this morning). In early December, it will be released as 5.0.0. Independent of that, the git repo's URL was changed to not be misleading, and there is an HTTP redirect so that all old URLs continue to work as before.

We did not make a version 5 git repo without version 4 history (as we did with 4 and 3, for the reasons I described in that email thread), and we did not make new PyPI package names. I'm still glad we've buried the Uproot 1‒3 history in an archived repo, but the PyPI package names was definitely a problem that was not justified by the reasons we had for it. You could call it a failed experiment. If anybody's thinking of doing something similar, I can show them concrete reasons why they shouldn't. But in particular, we didn't do it again.

[kgizdov]

Firstly, I understand all that and am glad about the bits that didn't happen this time.

However, I feel like I am misrepresenting what I mean. I am not looking for an explanation. Forgive me for being blunt. I guess I must be in order to get my point across.

I am a member of Scikit-HEP with a package of my own, so I know how organisation adoption works. That's not the issue at the moment.

Secondly, I am not disagreeing with the methods employed to rename the repo.

My point is that this should not be done at all, and I can see no justification for why it is being done.

For example, when Scikit-HEP adds a repo, it changes the repo owner; it does not change the name from mypackage to mypackage2.0-best-version-ever_final_v2. Specifically, repo owner change is a proper justification for why the URL is being changed.

Thirdly, I do not particularly care about the URL itself. The issue is deeper and more conceptual.

Finally, the idea that uproot4 moving to uproot5 is somehow futureproofing uproot6 and so on is ridiculous. There should be a repo with a name that does not contain the major version number and represents all past, current and future work done on uproot. That is the proper way of doing things. And by the proper way, I mean literally, everyone else does this for numerous reasons, the most important of which is - they have tried and eventually failed to do the opposite. That is why we have now arrived with versioning policies spanning every language, packaging tool, software ecosystem, etc.

[jpivarski]

Okay, so it's just the choice of name, scikit-hep/uproot5, then. I get that: it would be best if the repo did not have a version number in it at all.

What had to happen was that it could not be named scikit-hep/uproot4 anymore. Then it would have a wrong version number in the name, and as bad as it is to have a version number in the name, it's worse to have a wrong version number in the name. Someone looking at that might think, "Surely I've come to the wrong site..."

It also could not be named scikit-hep/uproot because the Uproot 1‒3 repo once had that URL. I strongly considered doing that, on the grounds that "uproot" is what it should be called and the original URL had been changed for more than a year at that point; Google seemed to have reindexed. But in the end, clobbering a global namespace of URLs is the same type of error as clobbering a global namespace of PyPI names, and we know that's bad.

I asked about possible names on Gitter and Slack many months in advance of doing it (no response). I don't want to keep moving from uproot5 to uproot6¹: it's unaesthetic, though it doesn't do damage because HTTP redirects work and none of these URLs have been taken. If you can think of an alternative, satisfying the constraints:

• not a URL we've used before
• not including a version number in the URL, especially not a wrong one
• something easily guessed or not too weird for human eyes when they see it

then we should consider it.

Footnotes

1. I don't see any reason why there would be an Uproot 6 in the future: the codebase seems to be converging. The 4 → 5 changes wouldn't be significant enough to bump the major version number if it weren't for the scale of changes in the Awkward codebase, which is also converging, albeit later than Uproot did. ↩