Bump jsonschema to ^3

[optiz0r]

Previous requirement on 2.6.0 causes poetry commands to fail
with error:

[AttributeError]
module 'jsonschema' has no attribute 'Draft7Validator'

[dmfigol]

I don't think we use jsonschema anywhere, that's why it should not be in pyproject.toml, which describes direct dependencies.
Could you please clarify?

[optiz0r]

Can you clarify, this should be listed in poetry.lock only, but not pyproject.toml? Presently, poetry insists v2.6.0 be installed, which then renders poetry unusable. pip install --upgrade jsonschema resolves it temporarily, but poetry will revert back to 2.6.0 on any action. This commit is the result of poetry add --dev jsonschema=^3 (the --dev is only to minimise the diff against the current poetry.lock file, where this is listed as a dev dependency, though I think it's really a runtime dependency from pydantic?)

[dbarrosop]

Previous requirement on 2.6.0 causes poetry commands to fail

Would you mind clarifying that statement, please? Mostly trying to reproduce as I haven't seen that error before.

What @dmfigol is trying to explain is that we shouldn't have to deal in pyproject.toml with dependencies that are not directly to the project. (jsonschema is an indirect dependency and should be handled by whichever library is using it).

[optiz0r]

I totally get your point that this is an implicit dependency. Without this change, I can't actually use poetry to manage dependencies. It may not be "correct", but working is better than not-working?

A log, captured on Fedora:

cd ~/git/nornir
python -m venv .env3
source .env3/bin/activate
pip install poetry
poetry install

For reference, poetry install does the following:

[~/git/nornir] [.env3] develop+ ± poetry install
Installing dependencies from lock file


Package operations: 72 installs, 1 update, 0 removals

  - Installing pycparser (2.19)
  - Installing asn1crypto (0.24.0)
  - Installing cffi (1.12.1)
  - Installing bcrypt (3.1.6)
  - Installing cryptography (2.5)
  - Installing decorator (4.3.2)
  - Installing ipython-genutils (0.2.0)
  - Installing pyasn1 (0.4.5)
  - Installing pynacl (1.3.0)
  - Installing paramiko (2.4.2)
  - Installing parso (0.3.4)
  - Installing ptyprocess (0.6.0)
  - Installing traitlets (4.3.2)
  - Installing wcwidth (0.1.7)
  - Installing backcall (0.1.0)
  - Installing colorama (0.4.1)
  - Installing jedi (0.13.2)
  - Installing jupyter-core (4.4.0)
  - Installing lxml (4.3.1)
  - Installing markupsafe (1.1.0)
  - Installing pexpect (4.6.0)
  - Installing pickleshare (0.7.5)
  - Installing prompt-toolkit (2.0.8)
  - Installing pygments (2.3.1)
  - Installing pyserial (3.4)
  - Installing python-dateutil (2.8.0)
  - Installing pyyaml (3.13)
  - Installing pyzmq (17.1.2)
  - Installing scp (0.13.0)
  - Installing selectors2 (2.0.1)
  - Installing textfsm (0.4.1)
  - Installing tornado (5.1.1)
  - Installing atomicwrites (1.3.0)
  - Installing future (0.17.1)
  - Installing ipython (7.2.0)
  - Installing jinja2 (2.10)
  - Updating jsonschema (3.0.0 -> 2.6.0)
  - Installing jupyter-client (5.2.4)
  - Installing more-itertools (6.0.0)
  - Installing ncclient (0.6.3)
  - Installing netaddr (0.7.19)
  - Installing netmiko (2.3.1 9c0996d)
  - Installing pluggy (0.8.1)
  - Installing py (1.7.0)
  - Installing snowballstemmer (1.2.1)
  - Installing appdirs (1.4.3)
  - Installing click (7.0)
  - Installing coverage (4.5.2)
  - Installing dataclasses (0.6)
  - Installing ipykernel (5.1.0)
  - Installing junos-eznc (2.2.0)
  - Installing mccabe (0.6.1)
  - Installing mypy-extensions (0.4.1)
  - Installing nbformat (4.4.0)
  - Installing nxapi-plumbing (0.5.2)
  - Installing pycodestyle (2.5.0)
  - Installing pydocstyle (3.0.0)
  - Installing pyeapi (0.8.2)
  - Installing pyflakes (2.1.0)
  - Installing pyiosxr (0.53)
  - Installing pytest (4.2.1)
  - Installing toml (0.10.0)
  - Installing typed-ast (1.3.1)
  - Installing black (18.9b0)
  - Installing flake8-import-order (0.18)
  - Installing mypy (0.670)
  - Installing napalm (2.4.0)
  - Installing nbval (0.9.1)
  - Installing pydantic (0.17)
  - Installing pylama (7.6.6)
  - Installing pytest-cov (2.6.1)
  - Installing requests-mock (1.5.2)
  - Installing ruamel.yaml (0.15.88)
  - Installing nornir (2.0.0)

Now, any subsequent poetry commands yield the following:

[~/git/nornir] [.env3] develop+ ± poetry install

[AttributeError]
module 'jsonschema' has no attribute 'Draft7Validator'

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

[~/git/nornir] [.env3] develop+ 1 ± poetry show

[AttributeError]
module 'jsonschema' has no attribute 'Draft7Validator'

show [--no-dev] [-t|--tree] [-l|--latest] [-o|--outdated] [-a|--all] [--] [<package>]

[~/git/nornir] [.env3] develop+ 1 ± poetry add something

[AttributeError]
module 'jsonschema' has no attribute 'Draft7Validator'

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...

Upgrading jsonschema fixes this, but of course later running poetry install command downgrades jsonschema to match the lock file, which breaks it again.

pip install --upgrade jsonschema
[~/git/nornir] [.env3] develop+ ± poetry debug:info

Poetry
======

 * Version: 0.12.11
 * Python:  3.6.5


Virtualenv
==========

 * Python:         3.6.5
 * Implementation: CPython
 * Path:           /data/home/brobert/git/nornir/.env3
 * Valid:          True


System
======

 * Platform: linux
 * OS:       posix
 * Python:   /usr

[~/git/nornir] [.env3] develop+ ± poetry install
Installing dependencies from lock file


Package operations: 0 installs, 1 update, 0 removals

  - Updating jsonschema (3.0.0 -> 2.6.0)
  - Installing nornir (2.0.0)
[~/git/nornir] [.env3] develop+ ± poetry install

[AttributeError]
module 'jsonschema' has no attribute 'Draft7Validator'

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

poetry debug:resolve -vvv suggests the dependency might be coming from nbformat:
nbformat (4.4.0) depends on jsonschema (>=2.4,<2.5.0 || >2.5.0)

Seems related to python-poetry/poetry#532 (unresolved at time of writing)

So where should this be solved?
nbformat doesn't depend on 3.0.0. Poetry doesn't depend on jsonschema. The only place that uses both nbformat/jsonschema/poetry in combination is nornir, so nornir is the only place that can realistically set this dependency on ^3. Open to counter arguments here...

As an aside, is it right that poetry install seems to be trying to install nornir==2.0.0 every time it runs?

[ktbyers]

@optiz0r Do you run into the issue if you install Poetry without using pip?

curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python

The above is the canonical way for installing poetry.

You can get into conflicts between Poetry's dependencies and the dependencies of the libraries Poetry installs if you use Pip. I had this same 'Draft7Validator/jsonschema' error at one point.

[optiz0r]

Issue is not present if poetry is installed that way. That said, eww to the curl | interpreter pattern in 2019.

[ktbyers]

Okay, so we can close this issue correct?

Once Poetry is installed in the proper way (i.e. not using pip), then jsonschema 3.0.0 is no longer required.

[ktbyers]

For reference, here is what the Poetry readme says on this:

Using pip to install poetry is also possible.

pip install --user poetry

Be aware, however, that it will also install poetry's dependencies which 
might cause conflicts.

From https://github.com/sdispater/poetry#installation

[dbarrosop]

eww to the curl | interpreter pattern in 2019

well, it makes sense IMO in this case as you are installing a tool to manage python. You probably installed pip via apt or yum or similar (assuming a linux machine), poetry falls under the same category as pip.

[optiz0r]

I would still like to vote for inclusion of this change on the basis that:

• get-poetry.py is not suitable for installation on our production systems without a whole lot of extra effort
• jsonschema is already an implicit dependency, the issue is a combination of versions of software tools so a fix cannot be made anywhere but this project. If there's a time in the future when a nornir dependency implicitly depends on jsonschema 3+, it could be removed.
• It costs this project very little to include this change, but brings benefit, and the work has already been done

You probably installed pip via apt or yum or similar (assuming a linux machine), poetry falls under the same category as pip.

Yes, pip was installed by OS package manager, because it's a popular, mature product, made available by LTS linux vendors. Poetry is shiny and new and has no packages available as yet. I tried building them last night with FPM but hit a wall several levels of dependency deep. Packaging is hard. Please help me make things less hard.

[dbarrosop]

I'd rather fix the root cause of the problem instead of having to manage dependencies we don't care about directly. I have created #332 to discuss this. Let me know if I understood your scenario correctly and if you agree that's a better solution.

[optiz0r]

What you propose in #332 WFM. Feel free to leave this closed. I'll rebase this commit out of #329 so that can be merged independently.

[dbarrosop]

thanks!

[optiz0r]

For the sake of others finding this issue, get-poetry.py doesn't work on systems where python resolves to python2, and python3.6+ is a separate binary (see python-poetry/poetry#721, unresolved at time of writing). In order to use poetry on such systems, must install the following way:

curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python3

And then use it as follows:

python3 ~/.poetry/bin/poetry [commands]

Their suggested workaround is to install using pip, which doesn't work in this case because of the jsonschema conflict. Packaging is hard.

[dmfigol]

@optiz0r Another way is to use python -m pip install --user poetry outside of virtual environment. This will install poetry dependencies in your home, but not inside virtual environment.

[optiz0r]

@dmfigol I concede, pip3 install --user poetry is doing the job.