installing nornir branches requires poetry, which may be a problem

[dbarrosop]

Poetry may be harder to install in production systems while at the same time users may need to make use of development branches to satisfy their needs while the work is done upstream to integrate their changes.

A way of removing poetry as a dependency would be to combine poetry with a setup.py file instead of using the pyproject.toml file. This way poetry could still be used for our build pipeline (main use) while allowing users to install it directly with pip.

[dbarrosop]

pinging interested parties: @dmfigol @ktbyers @optiz0r

[dmfigol]

I don't really like this proposal, but I am biased because I really like poetry.

• Almost every project has some tooling required for development environment
• curl https://get-poetry.py | python is as dangerous as running pip install poetry because arbitrary code can be run in setup.py (by the way, the recommended way to install pip is also with curl: curl -L https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python get-pip.py )
• Alternatively, if one really wants to use pip to install poetry, it should be installed outside of the project virtual environment: python -m pip install poetry or python -m pip install --user poetry (to install in home directory)
• Pip 19+ adds support for PEP517/518 and this allows to install the project directly from git branch / pull request / locally without poetry based on the information in pyproject.toml:

-> % pip install git+https://github.com/nornir-automation/nornir.git@develop
Collecting git+https://github.com/nornir-automation/nornir.git@develop
  Cloning https://github.com/nornir-automation/nornir.git (to revision develop) to /private/var/folders/js/jzwkkbb53nd7ns_53g069nlr0000gn/T/pip-req-build-0b49axc5
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
...
Successfully installed MarkupSafe-1.1.1 asn1crypto-0.24.0 bcrypt-3.1.6 certifi-2018.11.29 cffi-1.12.1 chardet-3.0.4 colorama-0.4.1 cryptography-2.5 future-0.17.1 idna-2.8 jinja2-2.10 junos-eznc-2.2.0 lxml-4.3.1 mypy-extensions-0.4.1 napalm-2.4.0 ncclient-0.6.3 netaddr-0.7.19 netmiko-2.3.0 nornir-2.0.0 nxapi-plumbing-0.5.2 paramiko-2.4.2 pyIOSXR-0.53 pyasn1-0.4.5 pycparser-2.19 pydantic-0.17 pyeapi-0.8.2 pynacl-1.3.0 pyserial-3.4 pyyaml-3.13 requests-2.21.0 ruamel.yaml-0.15.88 scp-0.13.0 selectors2-2.0.1 six-1.12.0 textfsm-0.4.1 urllib3-1.24.1

or

-> % git clone https://github.com/nornir-automation/nornir
-> % cd nornir && git checkout develop
-> % python -m venv .venv && source .venv/bin/activate && pip install -U pip setuptools
-> % pip install .
Processing /Users/dmfigol/work/tmp/nornir
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
...
Successfully installed MarkupSafe-1.1.1 asn1crypto-0.24.0 bcrypt-3.1.6 certifi-2018.11.29 cffi-1.12.1 chardet-3.0.4 colorama-0.4.1 cryptography-2.5 dataclasses-0.6 future-0.17.1 idna-2.8 jinja2-2.10 junos-eznc-2.2.0 lxml-4.3.1 mypy-extensions-0.4.1 napalm-2.4.0 ncclient-0.6.3 netaddr-0.7.19 netmiko-2.3.0 nornir-2.0.0 nxapi-plumbing-0.5.2 paramiko-2.4.2 pyIOSXR-0.53 pyYAML-3.13 pyasn1-0.4.5 pycparser-2.19 pydantic-0.17 pyeapi-0.8.2 pynacl-1.3.0 pyserial-3.4 requests-2.21.0 ruamel.yaml-0.15.88 scp-0.13.0 selectors2-2.0.1 six-1.12.0 textfsm-0.4.1 urllib3-1.24.1

[dbarrosop]

Ok, I will test your pip examples, if that works I don't think there is much to discuss here. My understanding was that it didn't work. @optiz0r would you mind testing as well?

[ktbyers]

I don't see an issue with installing Poetry from downloading it and executing it. Poetry is the more fundamental tool so it probably should be outside of the relevant virtual environment.

Reading on this thread here, it looks like the Poetry should install Nornir as being editable. In other words, if we do 'poetry install'; a 'pip install -e' is executed.

python-poetry/poetry#34

I do see that when I test it:

$ poetry install
Installing dependencies from lock file

Nothing to install or update

  - Installing nornir (2.0.0)

(poetry_test2) [kbyers@ip-172-30-0-171 nornir]$ pip list | grep nornir
nornir              2.0.0      /home/kbyers/tmp_nornir/nornir     

Note, this was using Poetry that was installed via downloading and executing get-poetry.py

I did still run into AttributeError: module 'jsonschema' has no attribute 'Draft7Validator' error in doing this (using both poetry install and pip install .). Strangely, I tested it in two different virtual environments and one time I was not seeing the issue, and then a second situation I was (I don't know why that occurred).

This jsonschema problem did go away when I upgraded to jsonschema 3.0.0.

I suspect the problem is an issue in the interaction between poetry and pip. Pip is invoking Poetry as the build tool, but then tries to use the older version of jsonschema that it just installed (instead of jsonschema that Poetry requires).

But given that poetry install is supposed to use the local repository (i.e. pip editable). I expect across time these tool interaction issues will be reduced.

For now I say we just update the lock file to use jsonschema 3.0.0 (assuming that works for our dependencies which I think it probably does). Then maybe submit an issue to poetry if our pip-poetry-jsonschema interaction issue is actually a bug.

My reaction would be pretty strongly against adding a setup.py file (i.e. I would be more inclined to march forward and try to work out the issues we are seeing).

Note, I was testing using the following:

$ poetry --version
Poetry 0.12.11
(poetry_test2) [kbyers@ip-172-30-0-171 nornir]$ pip list | grep pip
pip                 19.0.3     
(poetry_test2) [kbyers@ip-172-30-0-171 nornir]$ pip list | grep setuptools
setuptools          40.8.0     

[optiz0r]

As also commented in #326 ,

python -m pip install --user poetry
poetry install

Is indeed doing the trick, and poetry continues working after the first run. This resolves this issue from my perspective.

[dbarrosop]

Awesome, glad to see it was easier than expected :)