Decide which Python versions we're supporting

[rhofour]

Forking this off of #6313 since it's really a different issue.

The big question is how long will we support Python 2. There are some nice Python 3 features like type annotations which we can't use while support Python 2.

With different flags (say --python2 and --python) it's not hard to support both Python 2 and Python 3 in the generated code, but this doesn't help for the non-generated Python.

A more general question is exactly which versions of Python do we want to support. Do we want to support all versions of Python 3 from 3.9 back to 3.0?

[aardappel]

I think for now we should go with what @rw said in #6313 but glad to hear other opinions.

[github-actions]

This issue is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days.

[judahrand]

@aardappel So was the conclusion to drop 2.x support? And let 2.x users just use older versions?

[aardappel]

@rw suggested not to drop support unless there were very strong reasons to do so.

I think it be reasonable to start with --python meaning Python 3, and add a new --python2 flag to support that while we still can. Then start adding type annotations to the generated code functions and see where we get stuck, if anything.

Not a Python user myself, I'd be tempted to cut Python 2 entirely, but going with @rw's suggestion.

I'm not familiar with how many people are still using lower 3.x versions.. I'd guess that once you're using 3.x that upgrading is not hard, but what do I know :)

[dbaileychess]

I don't think we should be explicitly supporting Python 2. Python 2 is EOL as of 2020-01-01. If some user needs to use python 2 for some reason, they can use an older version of flatbuffers. I don't want to burden us because of a minority user.

@rhofour @judahrand Any interest in making PRs for your python 3 features?

[github-actions]

This issue is stale because it has been open 6 months with no activity. Please comment or label not-stale, or this will be closed in 14 days.

[github-actions]

This issue was automatically closed due to no activity for 6 months plus the 14 day notice period.

[fliiiix]

I stumbled over the Python 2 support again with my mr here -> #8145

@dbaileychess Can i create a MR which removes Python 2 support from flatbuffers?
I trully hope people stopped using Python 2 https://devguide.python.org/versions/

[aardappel]

See my comment above.. can you list what simplifications we are gaining by removing Python2 support? How many if-then complexities do we get if we add a --python2 flag to allow 2 and 3 to diverge? If the answer is low, then there is no reason to drop support.

While an individual user would be crazy to to still use Python2 at this point, FlatBuffers is a "foundational" technology that underlies many projects. You'd be surprised how many projects still rely on Python2, lots of companies have internal tools that depend on it that they can't or won't upgrade, and just breaking them for a tiny bit of convenience doesn't seem smart to me.

[fliiiix]

Don't get me wrong im very aware about the companies that struggle to update things. I maintain radish a Python tool which still official supports Python 3.5 and probably runs on all Python 3 versions. Because i work in such an environment.

But supporting Python 2 in 2023 2024 is crazy to me as the userbase that will keep up with the latest flatbuffers release but never updates Python is probably non-existent at this point.

How many if-then complexities do we get

I do not think thats the relevant metric here, it burdens each contributor to think about Python 2 support, it will become increasingly more complex to test (since as long as you advertiser that it works, you should be sure it still does), and the biggest argument is right now it leaves users who just use the --python flag with worse defaults because there is this very hypothetical user-base that still uses Python 2 with the latest flatbuffer release

Because of that I am strongly in favor of removing the promise for Python 2 support and if new code breaks it thats fine.

[aardappel]

Well, lets first see that "burden" a bit.. introduce a flag, and see what it looks like. Once that gets complex, the case for dropping it will have actual evidence. Right now we're talking hypotheticals.

Again, saying its "crazy" holds for individual cases, not once you factor in long lived projects and relying on other people's stuff.

Of course, @dbaileychess can make the final call, just my opinion.

[akb825]

@aardappel It's worth noting that the latest release generates code that isn't compatible with Python 2 due to a bug with the recently added --python-typing feature. I have a fix for this in PR #7983, but unfortunately it's been a victim of the recent lack of activity.

Just to give a bit of context for my own usage, I use Python scripts to generate flatbuffer data, and it's set up so it should work with the default version of Python on nearly every OS that ships with some version of Python. For some older Linux distros, as well as older versions of macOS, that has often been Python 2. As older versions of various Linux LTS releases have gone EOL themselves, and the last 2 or 3 versions of macOS have shipped with Python 3 installed, this has luckily been less of an issue lately. However, given how long many legacy technologies are used well beyond the end of support, I agree that flatbuffer support for Python 2 should probably only happen once there's a compelling need to.

[fliiiix]

and it's set up so it should work with the default version of Python

I mean im doing the same with Debian buster which only has flatbuffers in backports https://packages.debian.org/buster-backports/flatbuffers-compiler and is still on 1.12
Dropping Python 2 support from master is not affecting all these legacy systems as they will use
very old flatbuffers version.

the latest release generates code that isn't compatible with Python 2

This for me is a very good indicator that you should stop advertising compatibility with something if you can break it and nobody really complains and its just broken for a long time.

But if you @aardappel & @akb825 feel so strongly about Python 2 support i would love to see good tests for it preventing people like me from accidentally breaking the support. :)

[akb825]

Dropping Python 2 support from master is not affecting all these legacy systems as they will use
very old flatbuffers version.

At least in my case I check the generated flatbuffers into the repository so they can be used by default if flatbuffers isn't present on the system. If flatbuffers is installed, though, it will re-generate the C++ headers and Python files using the system version.

In the case of Debian you could work around it by installing the system package. In the case of CentOS/RHEL 7, best I can tell they don't have a system flatbuffers package available, so you'd need to track down the last version with Python 2 support and build it. For older versions of macOS, you would need to either build flatbuffers yourself or use something like Homebrew to install it, though if you set up Homebrew you would likely use it to also install a modern version of Python anyway.

This for me is a very good indicator that you should stop advertising compatibility with something if you can break it and nobody really complains and its just broken for a long time.

In this case it wasn't so much broken for a long time without people complaining as the repository being largely inactive for the last 6 months.

The --type-checking option was added in the May 8 release, but the initial version leaked even more type checking code when the option wasn't enabled, to the point where the scripts failed to run even on Python 3. (it skipped importing the modules but still output the type annotations) There were a couple bug reports, including one that I opened, and it was mostly fixed for the May 26 release. It did have one remaining bug when serializing vectors that I submitted a fix for 3 days later, but activity on the repository pretty much stopped after that release.

i would love to see good tests for it preventing people like me from accidentally breaking the support. :)

I agree with this. The fact that Python 3 was also broken for a few weeks means that the Python tests are probably lacking across all versions.

[fliiiix]

One other data point is that for cpp the support for very much supported grpc versions was just dropped see -> #8172

Debian Buster is https://packages.debian.org/buster/libgrpc-dev 1.16.1 and broken
Debian Bullseye is https://packages.debian.org/bullseye/libgrpc-dev 1.30.2 and broken
Only Bookworm will work

Because of that i don't see any reason to support Python 2 any longer the arguments made here are very theoretical and do not match any reality 😅