Configure package in pyproject.toml

[tylerflex]

Replace all requirements / setup.py

[daquintero]

So I'm already working on this. Just going to keep writing my notes on these public issues/discussions rather than have them in, say, a personal notebook. (I don't know if this will be annoying and I should just create personal discussions on the main discussions tab and reference them in the issues, still debating myself)

One thing that is interesting about the current setup.py structure is the create_config_folder function. I assume this is what allows us to save the Tidy3D API_KEY when people (like I did) do the pipx install or the other config parameters installs. I am exploring how to implement a subroutine function so that this is still implemented, but run from the pyproject.toml which is more declarative. I'm working on a first draft of the transfer and working out how to setup the eg. dynamic versioning and guarantee equivalence.

Worth mentioning I also have had a look at the flit vs poetry dependency management question to properly evaluate the alternatives. It seems like Poetry is for very advanced and complex libraries/dependencies whilst flit has benefits for packaging smaller projects. A little review of the community is here. It makes sense we use Poetry in our case, and it was important to look into this because the pyproject.toml configuration changes accordingly depending on the management tool parameters.

• Another thing I just noticed is that the license in the setup.py file doesn't actually match the licence in the repo so will fix that on the pyproject.toml

Currently I'm thinking about the versioning thing. pyproject.tomls are declarative, not functional. Which means that I'm thinking how the version tracking would work... Looking again into bump2version which I use already but in this context. I'm thinking of adding a script hook we can trigger

Also I'm wondering about this:

PACKAGE_NAME = "tidy3d"
REPO_NAME = "tidy3d"
PIP_NAME = "tidy3d"

I'm curious, what is the particular reason for them?

Writing this down here for reproducibility from a new venv to the working poetry install, maybe I'll add it in a script later:

python -m pip install --user pipx
python -m -m pipx ensurepath
pipx install poetry

Status in https://github.com/daquintero/tidy3d/tree/docs_improvement_suggestions

[tylerflex]

One thing that is interesting about the current setup.py structure is the create_config_folder function. I assume this is what allows us to save the Tidy3D API_KEY when people (like I did) do the pipx install or the other config parameters installs.

Yea, the ~/.tidy3d directory contains the API key (even when using regular pip) and maybe some other information, but I forget.

I am exploring how to implement a subroutine function so that this is still implemented, but run from the pyproject.toml which is more declarative. I'm working on a first draft of the transfer and working out how to setup the eg. dynamic versioning and guarantee equivalence.

I think perhaps another alternative (maybe simpler) is to just do the directory creation in web/__init__.py because it's only needed for the webapi. In fact, this might be how we should have had it set up from the beginning. You should probably check with the team working on the webapi to clarify that this won't break anything though, we can discuss.

Worth mentioning I also have had a look at the flit vs poetry dependency management question to properly evaluate the alternatives. It seems like Poetry is for very advanced and complex libraries/dependencies whilst flit has benefits for packaging smaller projects. A little review of the community is here. It makes sense we use Poetry in our case, and it was important to look into this because the pyproject.toml configuration changes accordingly depending on the management tool parameters.

Interesting, I'd never heard of it. Thanks for looking into that.

• Another thing I just noticed is that the license in the setup.py file doesn't actually match the licence in the repo so will fix that on the pyproject.toml

Thanks!

Currently I'm thinking about the versioning thing. pyproject.tomls are declarative, not functional. Which means that I'm thinking how the version tracking would work... Looking again into bump2version which I use already but in this context. I'm thinking of adding a script hook we can trigger

Version tracking between frontand and backend? or what do you mean here? Note that we have some sort of version tracking in the form of a mapping from solver version to front end version, which is maintained in the backend. I think this stuff we can probably start looking into with the solver team, but the front end packaging is probably higher priority at least right now.

Also I'm wondering about this:

PACKAGE_NAME = "tidy3d"
REPO_NAME = "tidy3d"
PIP_NAME = "tidy3d"

I'm curious, what is the particular reason for them?

Yea a few years ago, we had a basic version of tidy3d on PYPI called tidy3d. This pydantic version was in development for a while and was uploaded as tidy3d-beta on PYPI. The git repository was also called Tidy3D-Beta or something, so I made these constants to just track these. When we released this officially, we unified everything but these constants just stayed in the setup.py. We can unify them if it makes it easier.

[daquintero]

Sounds good!

I think the web/__init__.py is a good way to go re the create_config_folder as I was just writing this when you posted, but you've already proposed the solution.

Going back to the create_config_folder this works fine because setup.py is run, but pyproject.toml is declared. I think this might be an unportable issue. Why don't we make that create_config_folder function to run whenever import tidy3d instead which achieves the same objective? Apparently it's a bit nontrivial to add post install hooks to the pyproject.toml install. According to these people and even the poetry people it's impossible to run a post-install script in any pyproject.toml

Sorry I meant pyproject.toml currently requires a version field, but in the setup.py this is extracted from the tidy3d/version.py. In any case we can sort this out later.

I've got a working pyproject.toml with poetry already then! And can begin unifying the documentation requirements install.

Worth mentioning details re poetry extras

[daquinteroflex]

Bugs I've found working with poetry:

• pip latest packages are not always the latest poetry repository packages. This causes several issues when using more later versions and reproducing the lockfile.

Relevant bugs are discussed here, this Github issue.

Actions to fix:

poetry self update
poetry cache clear pypi --all # Clear old cache, sometimes has to be combined with other commands
poetry update [the problematic package] # or
poetry update # for all packages, will take a bit of time

Note I've had to update jaxlib as they've only left on pypi 0.4.1 onwards really.

[tylerflex]

Think we can close this?