chore: python version to 3.11 (while supporting 3.10)

[mistercrunch]

Bumping python to 3.11 as the default version, supporting 3.10, and deprecating 3.9 support

Currently in this PR I'm setting next to 3.11 and current to 3.11.11. This means when 3.11.12 shows up CI will run on two different versions within 3.11. Eventually we can make next point to 3.12 as lib support grows.

about 3.12

Seems 3.12 is not easy at the moment, hit some issues that pointed to upgrading dependencies, but hit some conflicting diamond dependencies as I was doing so. #31313 would help sort this out at a faster pace, without the bugs and slowness of pip-compile-multi

Closes #31516

[korbit-ai]

I've completed my review and didn't find any issues... but I did find this bear.

/  \.-"""-./  \
\    -   -    /
 |   o   o   |
 \  .-'''-.  /
  '-\__Y__/-'
     `---`

Need a new review? Comment /korbit-review on this PR and I'll review your latest changes.

Korbit Guide: Usage and Customization

Interacting with Korbit

• You can manually ask Korbit to review your PR using the /korbit-review command in a comment at the root of your PR.
• You can ask Korbit to generate a new PR description using the /korbit-generate-pr-description command in any comment on your PR.
• Too many Korbit comments? I can resolve all my comment threads if you use the /korbit-resolve command in any comment on your PR.
• Chat with Korbit on issues we post by tagging @korbit-ai in your reply.
• Help train Korbit to improve your reviews by giving a 👍 or 👎 on the comments Korbit posts.

Customizing Korbit

• Check out our docs on how you can make Korbit work best for you and your team.
• Customize Korbit for your organization through the Korbit Console.

Current Korbit Configuration

General Settings

​

Setting	Value
Review Schedule	Automatic excluding drafts
Max Issue Count	10
Automatic PR Descriptions	❌

Issue Categories

​

Category	Enabled
Naming	✅
Database Operations	✅
Documentation	✅
Logging	✅
Error Handling	✅
Systems and Environment	✅
Objects and Data Structures	✅
Readability and Maintainability	✅
Asynchronous Processing	✅
Design Patterns	✅
Third-Party Libraries	✅
Performance	✅
Security	✅
Functionality	✅

Feedback and Support

• Tell us what you think of Korbit
• Schedule a call with our team
• Email us @ support@korbit.ai

[villebro]

Should we first merge #31313 before proceeding with this? Also, should we not wait until 3.12 is supported before moving the window of supported versions forward?

[michael-s-molina]

deprecating 3.9 support

@mistercrunch There's a 5.0 proposal to drop support for 3.9. We should wait for the breaking window unless you remove the 3.9 deprecation of this PR's scope.

[mistercrunch]

@michael-s-molina yup I'm taking this on as part of the 5.0 effort.

Should we first merge #31313

@villebro yes, that will help sorting things out. I'll push on that one to go first (pre 5.0)

should we not wait until 3.12 is supported ?

I don't think that's required, my take is we should support previous and current, and next is a nice to have as a placeholder to start getting things working. The next construct is useful to support the process of getting ready for it, but not needed to deprecate a previous version. Given the state of CI, and the fact that only some checks run on previous and next, we should strongly recommend using the current version, the next/previous are useful if/when we want to support more or while we're in the process of establishing or deprecating support.

[mistercrunch]

Here -> #31740

[mistercrunch]

Mmmmh, yeah goes back to 2019 so my hopes aren't up on this one, I couldn't find a decent workaround.Maybe it's a good think to make optional as in #31740 though?

[mistercrunch]

Few options:

• leave this PR as is, and assume for now that next means the latest 3.11.x
• set 3.12 as next and live with the red, deal with confusion for contributors (I think that's not great)
• set 3.12 as next and force a green status, have to dig in logs to see if it's actually working
• set 3.12 as next and keep pushing on this PR until we get it working

[villebro]

@mistercrunch what was missing for 3.12 support? Numpy wheels? Let's give it one more go, and if it just doesn't work out let's settle for one of the compromises listed above..

[mistercrunch]

Ok, added a commit bringing in 3.12 in the mix, let's see what fails.

[mistercrunch]

Ok I pushed by bumping a bunch of libraries here, and ultimately got to a place where next hung on setup-backend while installing python deps. Turns out there were no wheels out for pandas/numpy for the version we had pinned. So I bumped it beyond the range that was specified in our pyproject.toml (there was no note around why it had an upper bound) and uncovered some other issue we have with this newer version pandas.

Certainly a good thing to unpin and push through, but might require some work or waiting for newer version of libs to move forward.

[mistercrunch]

The new pandas df.to_sql seem doesn't support sqlalchemy datatypes or at least not the same way -> pandas-dev/pandas#57049 .

Seems the issue is with pandas>=2.2.0 so applying an upper bound and retrying. Sometimes it's good to let things library bake a little, though worth trying to see how far we get.

[villebro]

@mistercrunch after letting this simmer for a while longer, would the following work:

• we run 3.11 on Dockerfile and refer to it as the currently supported version in the docs
• on the CI matrix, we replace current (3.11) and next (3.12) with previous (3.10) and current (3.11). After all, those are the only versions we officially support.

I don't necessarily see any reason to have next on CI if it just causes issues like this. So in the future, when we do want to start supporting 3.12, we open up a PR that bumps the versions on previous and next, and then iterate until both versions actually work.

[mistercrunch]

Yes I think that's effectively what it is in current PR, except for next being failing on 3.12 now. Maybe I just remove next from the matrixes for now and we turn it back on when we're ready to start working on supporting 3.12

[mistercrunch]

Ok I didn't want to loose the work I did on supporting 3.12 so started this placeholder PR here. I'll rollback 3.12 support work from this PR as it's breaking things, and remove next from our matrixes on this PR RN

[michael-s-molina]

@mistercrunch Could you please add the changes made in this PR to UPDATING.md?

[mistercrunch]

Done! I think this might be ready for merging

[mistercrunch]

@villebro @michael-s-molina anything else needed to get this merged?

[michael-s-molina]

@villebro @michael-s-molina anything else needed to get this merged?

@mistercrunch I didn't see a change in UPDATING.md. Did you push it? I believe it's the last required step.

[mistercrunch]

Oops, I forgot to git push my commit. I just pushed it

[michael-s-molina]

@mistercrunch I left a comment about the integration tests in the committers channel.

[villebro]

LGTM, but before merging, would you be able to add a note about the strict=False bit in the description? I assume we're mostly ok with not being strict about those zips, but In general it's probably usually better to be more strict than not..

[mistercrunch]

I think something changed in 3.11 or 3.12 about this particular method, and ruff caught it and required this change as lint. It'll enforce this rule moving forward and is upfront about the reason (it's easy to lookup the specific rule and why it's enforced)