Missing valid mac tags #578
Comments
|
Do we know if that tag makes sense? |
|
Can you clarify? The tag seems fine to me, just missing the minor version. |
|
The tag makes sense. Apple used to use 10.x (major dot minor), but recently starts to use only one single number to version the platform (11, 12 onward). 11.0 and 12.0 are arguably not valid, but we might want to keep them for compatibility since currently existing wheel generators may be using them now. |
|
That's great to know! I don't use mac so I have not kept up to date with the versioning scheme. And I agree that we probably want to keep this old tags valid. I will update the PR when I am back to the computer. The proposed change would be for 11 and 12 generate both version, with and without the minor version, and for 13 and onwards only generate tags with the major version. Does that sound good? |
|
Yeah, I guess that makes sense. Apple never tells anyone how they do things and it’s impossible to know if the version scheme is going to change again, but the strategy you outlined is likely the best we can do. Another issue not related to |
|
Well, the PEP does not specify anything about matching older versions, it just maps the platform tag to |
|
Also, I think it would make sense to update the PEP to say to use |
That's what my question was about; does a minor-version-free macOS version make sense?
What would happen is we list all the appropriate tags in preference order, and then tools can use that order however they choose. So in this instance we would probably list
We should probably migrate that entire PEP to packaging.python.org. But in this specific case I don't know what makes the most sense since the tag creation logic is rather complicated and not well-specified already, so I don't know if continuing to specify any part of it is important. |
In this specific case, I think we should probably list it earlier, and basically treat it as an alternative to
Agreed.
I don't know how you would define this field in that case. |
It does in the sense that macOS does return a minor-less |
This seems incorrect, the current macOS version is
These version numbers do not depend directly on macOS, they're generated by Python so it depends where you install Python from. On macOS 12.4 with Python 3.9.6 from conda-forge, I get: >>> distutils.util.get_platform()
'macosx-11.0-arm64'
>>> sysconfig.get_platform()
'macosx-11.0-arm64'The original bug report on https://github.com/FFY00/meson-python/issues/91 came from a build with Python 3.8 from Homebrew. The Python shipped by Apple ( >>> os.__file__
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py'
>>> import distutils.util
>>> distutils.util.get_platform()
'macosx-10.14-arm64'I suggest to simply add the missing tags and make no other changes. |
|
Another issue: https://github.com/FFY00/meson-python/issues/160 The minor versions are also not generated. Should I be generating |
I think so. I believe the last time we discussed how to do version number support for the "new" macOS versions the guidance this was what was suggested. We can discuss doing something different (e.g. all versions, no minor number), but I think the concern with the latter was whether anyone made weird assumptions about always having a minor number. But my memory may be faulty with all of this, so reading the old issues is the best way to double-check me. |
|
Yes I think from previous discussions, a wheel against 12.0 should be installable on 12.6. |
|
That is all correct - and yes, we should generate That said, there is no reason for |
|
Now I remember why we didn't want to do midyear updates in the tags; we don't know how to back-fill older macOS versions without hard-coding the final minor version number. For instance, while we can infer 12.6 to 12.0, what do we do for macOS 11? We don't know what the upper limit is for a minor version until Apple announces that version is EOL. So until that happens we can only guess what the upper bound is for macOS 11 (e.g. is it 11.5, 11.10, 11.100?). So either we have to hard-code the upper limit known at any moment in time and then push a new version of |
|
I don't really see why that is required - why wouldn't you just say that any |
Potentially, but then why do we care about Or put another way (to make sure I understand), if you're running macOS 13.2 and have a wheel that wants
without making every consumer of
That would be true if the glibc team told us they have no plans to ever change the major version number, but they said they didn't when that discussion came up. Otherwise we will have a similar problem. But glibc also bumps their major version number way less than macOS, so hard-coding whatever glibc 2 stops at is much less effort than trying to keep up with macOS' release cadence. |
Aside from possible incompatibilities - which could indeed occur - because it makes the tags easier to understand and specify. These issues came up to begin with because we implemented tag support in It's the same for
It's a major version number change - I think there are no watertight guarantees about 13.0 either way. But besides that, I don't think this argument holds. In corner cases 12.6 can be a breaking release, and so can 13.0. The latter is more likely, but both are possible. For 99.9x% of wheels there will be no problem (because the breakage will likely be for something obscure), so you don't have to change today's assumptions.
I may be missing something here. Is that actually being done anywhere? |
|
To make the case for an ability to set macOS deployment target more concrete, here is a real-world case:
So this has nothing to do with "will it run on 13.0 or not" (it will, unless Apple happens to break something unrelated in the future), but with "the older OS version has an unpatched bug that we need to avoid".
I don't understand this part of your question. You have to know less. All you have to do is use a You seem to be working under the assumption that the packaging tooling must know whether 12.6 was actually released by Apple or not. But why is this relevant at all? As far as I can tell, nothing will go wrong if it doesn't. |
Because I don't know where we would need to stop generating those tags. To make this concrete, if you are on macOS 13, what tags would you have us generate to cover macOS 12? |
We do not need an explicit list of all valid tags for either tag generation at build time or tag validation at install time (see the bullet points in my comment above). I think the only reason you are asking is because that is what Unless I'm missing a use case here, a better design would be to have a function that validates a given tag given the target interpreter, rather than "return a list of all valid tags for this platform" and then have
If you don't want to change what It also doesn't seem to matter for performance by the way. I checked what |
I ask because the library is designed to generate the list of valid tags, not to ask the library if the current system is valid for a specific tag.
The specs around tags are not structured to require asking a library if something is valid; it's meant to work against a static list of tags you were given. You're meant to be able to take a list of wheel tags and a list of platform tags and figure out what to use w/o using Another thing to point out is we have gone this long w/o finer granularity than the annual release version and basically been fine (i.e. all we never listed the micro releases from the 10.x version range). I think we're at enough of an impasse that bringing this up on discuss.python.org is appropriate. |
Yes agreed. I think we typically ignore the corner cases, and in practice it won't be so bad. Similar to Python releases, it's very common to deal with minor releases, but very rare to worry about a 3.x.0 release having a bug and therefore needing a wheel to only work on .1 and up. I guess when it does come up, folks just raise an exception at runtime with instructions to upgrade.
If you don't mind, I'll leave it at this. I've already spent more time on investigating this than I had available, and moving this to Discourse is unlikely to bring a quick resolution. So I'll leave it to the next person with a real need for this. |
|
Can we design the interface to provide something like |
We could have a convenience function for that which simply does the logic of searching the list of tags. But you will probably also want a utility function which takes a set of wheel tags (or file names) and then returns which tag matches "the best" (i.e. highest in the list of supported tags). This would also let you define "best" in either terms of fastest or most compatible/generic based on the order of the list.
Well, the API is based on generators so you don't have to generate the full list if you get an answer early enough. 😉 But yes, the reason for the regeneration has been that a platform effectively states what tags it supports; we just happens to be calculated constantly because CPython and other interpreters don't record these details externally in some JSON file (which is something I have thought about). |
|
IIUC, what @uranusjr and @rgommers are suggesting is that we avoid the need to generate all the tags to determine compatibility with the current platform in a new API -- have the ability to validate the tag without needing to generate the entire For example, instead of generating the entire sequence of tags in def has_compatible_mac_platform(tag):
parts = tag.platform.split("_")
if len(parts) != 4 or parts[0] != "maxosx":
return False
major, minor, binary_format = int(parts[1]), int(parts[2]), parts[3]
version_str, _, cpu_arch = platform.mac_ver()
version = cast("MacVersion", tuple(map(int, version_str.split(".")[:2])))
arch = _mac_arch(cpu_arch)
# Check version
if (major, minor) > version:
return False
# TODO: Handle 11.0 vs 10.6 style comparisons, and other complexities
# Check format
if binary_format not in _mac_binary_formats(version, arch):
return False
return True |
|
The other "fun" part of tags is they were designed to be opaque strings, so parsing them was never part of the design. And I personally wouldn't want to try and make that be forward/backwards-compatible as I feel like that's asking for trouble. I also bet generating the tag combinations isn't the expensive part, but the parts that query the OS which you have to do anyway to validate compatibility. If we were dealing with list lengths in the millions I would understand, but we are dealing with maybe a thousand at worst, and we have If people want a more ergonomic, high-level I'm happy to talk about that, but I don't want to get sucked into premature optimization below that API surface when I don't know how much computation we can actually avoid in the common case. Or put another way, |
|
For me it has nothing to do with cost of computation, that seems basically irrelevant for any packaging tools that create, install, or modify wheels. And I don't think anyone suggested otherwise? The API is cumbersome, not slow. |
I've opened #602 so we can discuss what sort of API people may want. |
|
So, back to the original intent of this issue 😅, do we want to bother with minor-version-free macOS versions? |
I'd say yes from first principles, it's clearer and communicates better what is happening. Plus it matches PEP 425 as @FFY00 pointed out. The hesitation I have is transition - given that old |
|
I'll see if @ronaldoussoren or @ned-deily have an opinion, otherwise I'm okay with adding minor-less tags starting from macOS 11 on, listing them at the lowest priority for that macOS version. |
|
Yes, that's the right approach, IMHO. Thanks! |
Adding minor-less tags is fine, especially as it is easily possible to get a python build that generates them. Personally I don't particularly like minor-less versions, especially here where IMO the tags with a minor version should be kept as well, there are minor versions of macOS 11 and later and those can introduce new APIs. It should be possible to tag a wheel for 12.4 when a C extension uses an API introduced in 12.4. |
I'm definitely not dropping any tags, just potentially adding some. |
|
There are wheels with tags using minor versions different from 0 cropping up on pypi (e.g. https://pypi.org/project/z3-solver/4.12.4.0/#files), I guess that developers should be made aware of the problem because for a user is difficult to understand why, e.g., the wheel |
+1. As a data point for this: macOS 13.3 added a new version of the Accelerate framework, including a LAPACK update from 3.2 to 3.9 and 64-bit support. This was about as large and important an update as new features get - it makes NumPy's macOS wheels ~3x smaller and improves performance of linear algebra APIs by up to 7x. We've had to avoid shipping these wheels for ~8 months now because of the lack of minor version support in |
|
Someone still needs to figure out either how to backfill minor versions for older macOS releases without requiring a packaging update, or someone needs to re-architect how wheel tag compatibility is verified and change the spec accordingly (and I'm no longer on macOS, so it won't be me). |
What's the largest minor version we could expect Apple to produce? If we can all agree on what that is (plus margin of error), we could backfill to that when it hasn't been hardcoded yet due to the install of |
|
The highest bug fix release was 10.4.11 (2007), in the post "macOS 10.x" era the highest has been 12.7. It should be safe to assume that the minor version for macOS 11 onwards will stay below 10. |
@ronaldoussoren is that w/ a margin of error? And can we safely just assume this w/o having to manually track the eventual, final minor version? |
That's with a small margin of error based on the final releases of macOS 11, 12 and 13. The max minor version of those was 7, backfilling to .9 gives a margin of two. To be extra save assume that the max version is 12, that would also work for the 10.4 release which had 10.4.11 as its final release. But TBH that shouldn't be necessary with the current yearly release cadence of major versions. The current release cadence at Apple appears to be slightly faster than every other month for minor releases. See https://en.wikipedia.org/wiki/MacOS_version_history for the detailed information. And take all of this with a pinch of salt, I have no insight in the internal workings of Apple. |
PEP 425 says (https://peps.python.org/pep-0425/#platform-tag)
On recent mac versions
However, pypa/packaging does not consider
macosx_12_arm64a valid tag, onlymacosx_12_0_arm64The text was updated successfully, but these errors were encountered: