docs: document the option of installing with virtual environments

[nogjam]

pip_install_strictdoc_deps.py requires a manual pip install of the toml module before it works with a fresh virtual environment. We can achieve a working installation much easier with a virtual environment (which is better practice anyway) and pip. This begs the question of whether this script is needed at all. Delete it, and make the corresponding updates to the Dev Guide and tox.ini.

[nogjam]

I am aware that I have broken invoke server (at least) by deleting pip_install_strictdoc_deps.py. Re-evaluating, learning about the invoke check task...

[nogjam]

invoke check before changes:

********************
UNSUPPORTED: StrictDoc HTML2PDF integration tests :: features/html2pdf/06_system_chromedriver/test.itest (6 of 6)
********************
Failed Tests (5):
  StrictDoc HTML2PDF integration tests :: features/html2pdf/01_empty_document/test.itest
  StrictDoc HTML2PDF integration tests :: features/html2pdf/02_three_top_level_requirements/test.itest
  StrictDoc HTML2PDF integration tests :: features/html2pdf/03_three_documents_with_assets/test.itest
  StrictDoc HTML2PDF integration tests :: features/html2pdf/04_composable_document_with_assets/test.itest
  StrictDoc HTML2PDF integration tests :: features/html2pdf/05_generate_bundle_document/test.itest

invoke check after changes (no difference):

********************
UNSUPPORTED: StrictDoc HTML2PDF integration tests :: features/html2pdf/06_system_chromedriver/test.itest (6 of 6)
********************
Failed Tests (5):
  StrictDoc HTML2PDF integration tests :: features/html2pdf/01_empty_document/test.itest
  StrictDoc HTML2PDF integration tests :: features/html2pdf/02_three_top_level_requirements/test.itest
  StrictDoc HTML2PDF integration tests :: features/html2pdf/03_three_documents_with_assets/test.itest
  StrictDoc HTML2PDF integration tests :: features/html2pdf/04_composable_document_with_assets/test.itest
  StrictDoc HTML2PDF integration tests :: features/html2pdf/05_generate_bundle_document/test.itest

[nogjam]

I am aware that I have broken invoke server (at least) by deleting pip_install_strictdoc_deps.py. Re-evaluating, learning about the invoke check task...

I have re-evaluated this and learned more about using invoke check. This should be ready for review.

[nogjam]

Hmm, I am noticing one more thing: I mistakenly forgot to update the .rst docs in addition to the .sdoc docs. Is this solvable with an invoke docs?

EDIT: Just tried it. Although it does indeed update the .rst docs, this invocation also seems to generate a larger diff than I was expecting:

➜ git diff --stat
 docs/sphinx/source/_assets/StrictDoc_Workspace-Roadmap.drawio.png | Bin 845541 -> 859334 bytes
 docs/sphinx/source/strictdoc_01_user_guide.rst                    |  16 +++++++++++++---
 docs/sphinx/source/strictdoc_03_faq.rst                           |  34 ++++++++++++++++++++++++++++------
 docs/sphinx/source/strictdoc_04_release_notes.rst                 |  30 +++++++++++++++++++++++++++---
 docs/sphinx/source/strictdoc_11_developer_guide.rst               |  25 +++++++++++++++++--------
 5 files changed, 85 insertions(+), 20 deletions(-)

Is this desirable?

[stanislaw]

Hey @nogjam , thanks for reviewing this setup step and contributing a possible improvement.

Before implementing the current solution, I had definitely tried doing the setup your way, but I encountered a limitation of pip install ., and that is that it installs both the dependencies AND the project itself. This means that you end up having a sort of undefined behavior because from then on, imports from any Python file will likely work against the installed package, not against the source files of StrictDoc. I double-checked if there was an option for pip install to avoid that kind of behavior and couldn't find it at the time.

Could you confirm that this problem indeed exists and suggest a possible workaround?

Thanks for looking into this. I am all in for simpler solutions.

[nogjam]

Hmm, I thought only pip install --editable . installed the current project, not pip install . . I'll look into this.

This means that you end up having a sort of undefined behavior because from then on, imports from any Python file will likely work against the installed package, not against the source files of StrictDoc.

Could you please clarify and/or provide an example of this undesirable behavior?

[nogjam]

Here's some more information from https://setuptools.pypa.io/en/latest/userguide/development_mode.html:

An “editable installation” works very similarly to a regular install with pip install ., except that it only installs your package dependencies, metadata and wrappers for console and GUI scripts.

Please note that, by default an editable install will expose at least all the files that would be available in a regular installation. However, depending on the file and directory organization in your project, it might also expose as a side effect files that would not be normally available.

Is the kind of problem you're thinking of something that invoke check would find? I can confirm that running it both with and without the "strictdoc" package installed produces the same results: 5 failed tests, as shown above. I'm using Python 3.12.3, FWIW.

[stanislaw]

Here's some more information from https://setuptools.pypa.io/en/latest/userguide/development_mode.html:

An “editable installation” works very similarly to a regular install with pip install ., except that it only installs your package dependencies, metadata and wrappers for console and GUI scripts.

Please note that, by default an editable install will expose at least all the files that would be available in a regular installation. However, depending on the file and directory organization in your project, it might also expose as a side effect files that would not be normally available.

Is the kind of problem you're thinking of something that invoke check would find? I can confirm that running it both with and without the "strictdoc" package installed produces the same results: 5 failed tests, as shown above. I'm using Python 3.12.3, FWIW.

The kind of problem I am referring to is when the source files will be at least partially read from the installed strictdoc pip package, not the source code repository of StrictDoc. If you run a test from a clean Git tree without making any changes to the source of StrictDoc, you will probably not notice anything odd as long as the Pip-installed version in your environment and the source code version are the same.

However, if you start working with the source code of StrictDoc, you will quickly notice that some/all of the files will be taken from the installed Pip, and so your developer experience will not be as you want it to be.

An “editable installation” works very similarly to a regular install with pip install ., except that it only installs your package dependencies, metadata and wrappers for console and GUI scripts.

The custom script that your MR is trying to remove only installs StrictDoc's dependencies and nothing else whereas even in the "editable installation" mode pip install still behaves like a strictdoc installer. For example, if I test on CI, I want to test the source code by only installing the dependencies but not the Pip itself.

Just to make sure: is this pip_install_strictdoc_deps.py script getting that much in your way? Normally, you should not work with this script directly by using the invoke tasks instead.

I would be open to removing this script if there was a good replacement but pip install ... doesn't seem to be a good one unfortunately.

[nogjam]

The kind of problem I am referring to is when the source files will be at least partially read from the installed strictdoc pip package, not the source code repository of StrictDoc.

Ah thanks, I think I understand now. So we don't want to install the project as it exists on PyPI, but we do want to install the project as it currently exists in the source code.

In this case, both pip install . and pip install -e . do what you want.

They both install the project from source code; the difference is, --editable mode will pick up changes made as you work on the source code, whereas non-editable mode will not, requiring you to re-run pip install . every time you want to see your changes.

From https://setuptools.pypa.io/en/latest/userguide/development_mode.html:

To facilitate iterative exploration and experimentation, setuptools allows users to instruct the Python interpreter and its import machinery to load the code under development directly from the project folder without having to copy the files to a different location in the disk. This means that changes in the Python source code can immediately take place without requiring a new installation.

So pip install . works the same way as the custom script, installing only the dependencies of StrictDoc, not the package from PyPI. For CI builds, either way (-e or not) will work fine, but the -e is pointless for one-off builds. It's more for developers working on the source code who want to see their changes take effect immediately.

To answer you question about whether it's getting in my way:

I'm pretty new to StrictDoc and getting spun up on it at work for some specs/requirements we're working on. (We are impressed, btw. This is a very nice tool and will save us a ton of time. Thank you for your work on it!) We will likely bring more people up to speed in the near future, and I have that in mind as I'm getting started.

The custom install script was not intuitive to me, a bit of a sticking point since I'm used to pip and poetry. So I thought I would recommend a simplification to ease adoption and make the process smoother for others on our team and elsewhere in the open-source community who decide to use and contribute to this wonderful project. Despite all the bifurcation in Python's packaging ecosystem, most people are still familiar with the original pip.

[stanislaw]

First of all, a general comment: I do understand how pip install -e . works, and how it is different from pip install .. I use both versions sometimes but rarely, mostly when I test the compatibility between StrictDoc and its daughter ReqIF project. I also did use Poetry with StrictDoc several years ago but then removed it as a tool that didn't give me much added value on top of what one can do with pyproject.toml when it comes to releasing Pips. Also maybe now Poetry is doing better but back then I had some hangs or weird dependency resolution conflicts that I didn't have without it. Back then, I switched to Tox to be able to have the virtual environments, each per a given Python version. I have been pretty happy with that setup until now, and would prefer to change it only if there is a significant advantage offered (for example, uv is referred to as a much faster and modern installer).

The kind of problem I am referring to is when the source files will be at least partially read from the installed strictdoc pip package, not the source code repository of StrictDoc.

Ah thanks, I think I understand now. So we don't want to install the project as it exists on PyPI, but we do want to install the project as it currently exists in the source code.

Not exactly :) I don't want to install the project at all when it comes to development. In the development mode, I want to work directly with the strictdoc/cli/main.py as an explicit entry point to the program to avoid even thinking about any pip packaging, be it in a virtual environment or not. At the same time, there are also integration tests that do test against the installed version, they are just not the primary ones I use during the development.

In this case, both pip install . and pip install -e . do what you want.

They both install the project from source code; the difference is, --editable mode will pick up changes made as you work on the source code, whereas non-editable mode will not, requiring you to re-run pip install . every time you want to see your changes.

From https://setuptools.pypa.io/en/latest/userguide/development_mode.html:

To facilitate iterative exploration and experimentation, setuptools allows users to instruct the Python interpreter and its import machinery to load the code under development directly from the project folder without having to copy the files to a different location in the disk. This means that changes in the Python source code can immediately take place without requiring a new installation.

So pip install . works the same way as the custom script, installing only the dependencies of StrictDoc, not the package from PyPI. For CI builds, either way (-e or not) will work fine, but the -e is pointless for one-off builds. It's more for developers working on the source code who want to see their changes take effect immediately.

To answer you question about whether it's getting in my way:

I'm pretty new to StrictDoc and getting spun up on it at work for some specs/requirements we're working on. (We are impressed, btw. This is a very nice tool and will save us a ton of time. Thank you for your work on it!) We will likely bring more people up to speed in the near future, and I have that in mind as I'm getting started.

The custom install script was not intuitive to me, a bit of a sticking point since I'm used to pip and poetry. So I thought I would recommend a simplification to ease adoption and make the process smoother for others on our team and elsewhere in the open-source community who decide to use and contribute to this wonderful project. Despite all the bifurcation in Python's packaging ecosystem, most people are still familiar with the original pip.

If you are 100% sure, working with pip install . (with or without -e) would make your and others' life easier, we could simply extend the development instructions and suggest this workflow as an option. Your PR could simply add a small subsection such as Development within a virtual environment and describe the steps you would like to use there. With this approach, you don't need to modify anything except the documentation (i.e., no changes in the GitHub CI YAML files).

I personally don't care much about the virtual environments when I run strictdoc because I have a multitude of Python versions with pyenv. At the same time, a separate sections suggesting to always work with a venv is fine for me as an option. I prefer to have StrictDoc's development dependencies installed and then call strictdoc/cli/main.py for development, also I configure the PyCharm IDE to do the same.

Let me know if updating a doc with a dedicated section for virtual environments would make be a good solution.

---

Updated later:

Suggestion:

• Installing StrictDoc from GitHub (developer mode)

Add this and explain your preferred approach:

• Installing StrictDoc from GitHub (developer mode with a virtual environment)

[nogjam]

Thanks for your detailed explanation! Sounds like there may be deep waters here I do not yet fathom, but may understand better with time if I continue working with StrictDoc and other tools like pyenv (which I'm not familiar with).

I also appreciate the insight on StrictDoc's history with Poetry, presumably in response to my comment here. I too have been frustrated by some of Poetry's dependency issues in the past. Presently, I am a Poetry fan, it works well for me, and I share the concerns of others in the Python community around uv's sustainability outlook, esp it not being written in Python. But I'm not opposed to trying something new and would love to be wrong about my concerns.

I like your suggestion to simply extend the docs. I'd be happy to update this PR to that end!

[nogjam]

• Revert all the code changes and adjust changes to documentation.

[stanislaw]

Thanks for your detailed explanation! Sounds like there may be deep waters here I do not yet fathom, but may understand better with time if I continue working with StrictDoc and other tools like pyenv (which I'm not familiar with).

Sounds good. I will also have an eye on improving these setup steps if I see a good opportunity.

I also appreciate the insight on StrictDoc's history with Poetry, presumably in response to my comment here. I too have been frustrated by some of Poetry's dependency issues in the past. Presently, I am a Poetry fan, it works well for me, and I share the concerns of others in the Python community around uv's sustainability outlook, esp it not being written in Python. But I'm not opposed to trying something new and would love to be wrong about my concerns.

I like your suggestion to simply extend the docs. I'd be happy to update this PR to that end!

Yes, just updating the docs is good for this round.

Regarding your other comment regarding also updating the corresponding RST file. The Sphinx website used to be the default documentation website but not anymore. We have recently migrated to host the native SDoc docs on readthedocs, and the Sphinx website will probably be deprecated or removed very soon. Don't regenerate the docs with this MR, just the .sdoc change is enough. (I am not decided yet on whether to keep the Sphinx one just as a working example of the SDoc->RST export feature or delete it for good.)

[nogjam]

• Revert all the code changes and adjust changes to documentation.

We should be good to go!

(I'm a bit new to GitHub pull requests. Not sure if there's a better way to block merge based on pending tasks.)

[nogjam]

Can close #2058 when merged.

[stanislaw]

Thanks! I also renamed the PR title to better reflect the final change.