Cannot install requirements with python 3.8

[barredterra]

Information about bug

When trying to install erpnext's requirements with python 3.8, I get an error. ERPNext requires taxjar, which in turn requires jsonobject, which causes the error.

Module

other

Version

Frappe: develop
ERPNext: develop (4738367)

Installation method

manual install

Relevant log output / Stack trace / Full Error Message.

> bench --version                                                                
5.7.6

> bench version -f table
+---------+----------------+---------+---------+
| App     | Version        | Branch  | Commit  |
+---------+----------------+---------+---------+
| erpnext | 13.x.x-develop | develop | 4738367 |
| frappe  | 14.x.x-develop | develop | aeba8a2 |
+---------+----------------+---------+---------+

> env/bin/pip install -e apps/erpnext
Obtaining file:///home/frappe/frappe-bench/apps/erpnext
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pandas~=1.1.5 in ./env/lib/python3.8/site-packages (from erpnext==14.0.0.dev0) (1.1.5)
Requirement already satisfied: gocardless-pro~=1.22.0 in ./env/lib/python3.8/site-packages (from erpnext==14.0.0.dev0) (1.22.0)
Requirement already satisfied: PyGithub~=1.55 in ./env/lib/python3.8/site-packages (from erpnext==14.0.0.dev0) (1.55)
Requirement already satisfied: redisearch==2.0.0 in ./env/lib/python3.8/site-packages (from erpnext==14.0.0.dev0) (2.0.0)
Requirement already satisfied: googlemaps in ./env/lib/python3.8/site-packages (from erpnext==14.0.0.dev0) (4.4.7)
Collecting taxjar~=2.0.0
  Using cached taxjar-2.0.0-py3-none-any.whl
Requirement already satisfied: python-youtube~=0.8.0 in ./env/lib/python3.8/site-packages (from erpnext==14.0.0.dev0) (0.8.1)
Requirement already satisfied: pycountry~=20.7.3 in ./env/lib/python3.8/site-packages (from erpnext==14.0.0.dev0) (20.7.3)
Requirement already satisfied: Unidecode~=1.2.0 in ./env/lib/python3.8/site-packages (from erpnext==14.0.0.dev0) (1.2.0)
Requirement already satisfied: tweepy~=3.10.0 in ./env/lib/python3.8/site-packages (from erpnext==14.0.0.dev0) (3.10.0)
Requirement already satisfied: plaid-python~=7.2.1 in ./env/lib/python3.8/site-packages (from erpnext==14.0.0.dev0) (7.2.1)
Requirement already satisfied: python-stdnum~=1.16 in ./env/lib/python3.8/site-packages (from erpnext==14.0.0.dev0) (1.17)
Requirement already satisfied: redis>=2.10 in ./env/lib/python3.8/site-packages (from redisearch==2.0.0->erpnext==14.0.0.dev0) (3.5.3)
Requirement already satisfied: hiredis>=0.2.0 in ./env/lib/python3.8/site-packages (from redisearch==2.0.0->erpnext==14.0.0.dev0) (2.0.0)
Requirement already satisfied: rmtest>=0.2 in ./env/lib/python3.8/site-packages (from redisearch==2.0.0->erpnext==14.0.0.dev0) (0.7.0)
Requirement already satisfied: six>=1.10.0 in ./env/lib/python3.8/site-packages (from redisearch==2.0.0->erpnext==14.0.0.dev0) (1.16.0)
Requirement already satisfied: requests>=2.6 in ./env/lib/python3.8/site-packages (from gocardless-pro~=1.22.0->erpnext==14.0.0.dev0) (2.25.1)
Requirement already satisfied: python-dateutil>=2.7.3 in ./env/lib/python3.8/site-packages (from pandas~=1.1.5->erpnext==14.0.0.dev0) (2.8.2)
Requirement already satisfied: numpy>=1.15.4 in ./env/lib/python3.8/site-packages (from pandas~=1.1.5->erpnext==14.0.0.dev0) (1.22.2)
Requirement already satisfied: pytz>=2017.2 in ./env/lib/python3.8/site-packages (from pandas~=1.1.5->erpnext==14.0.0.dev0) (2021.1)
Requirement already satisfied: deprecated in ./env/lib/python3.8/site-packages (from PyGithub~=1.55->erpnext==14.0.0.dev0) (1.2.13)
Requirement already satisfied: pynacl>=1.4.0 in ./env/lib/python3.8/site-packages (from PyGithub~=1.55->erpnext==14.0.0.dev0) (1.5.0)
Requirement already satisfied: pyjwt>=2.0 in ./env/lib/python3.8/site-packages (from PyGithub~=1.55->erpnext==14.0.0.dev0) (2.0.1)
Requirement already satisfied: requests-oauthlib<2.0.0,>=1.3.0 in ./env/lib/python3.8/site-packages (from python-youtube~=0.8.0->erpnext==14.0.0.dev0) (1.3.1)
Requirement already satisfied: dataclasses-json<0.6.0,>=0.5.3 in ./env/lib/python3.8/site-packages (from python-youtube~=0.8.0->erpnext==14.0.0.dev0) (0.5.6)
Requirement already satisfied: isodate<0.7.0,>=0.6.0 in ./env/lib/python3.8/site-packages (from python-youtube~=0.8.0->erpnext==14.0.0.dev0) (0.6.1)
Collecting jsonobject>=0.9.10
  Using cached jsonobject-0.9.10.tar.gz (394 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: marshmallow<4.0.0,>=3.3.0 in ./env/lib/python3.8/site-packages (from dataclasses-json<0.6.0,>=0.5.3->python-youtube~=0.8.0->erpnext==14.0.0.dev0) (3.14.1)
Requirement already satisfied: typing-inspect>=0.4.0 in ./env/lib/python3.8/site-packages (from dataclasses-json<0.6.0,>=0.5.3->python-youtube~=0.8.0->erpnext==14.0.0.dev0) (0.7.1)
Requirement already satisfied: marshmallow-enum<2.0.0,>=1.5.1 in ./env/lib/python3.8/site-packages (from dataclasses-json<0.6.0,>=0.5.3->python-youtube~=0.8.0->erpnext==14.0.0.dev0) (1.5.1)
Requirement already satisfied: cffi>=1.4.1 in ./env/lib/python3.8/site-packages (from pynacl>=1.4.0->PyGithub~=1.55->erpnext==14.0.0.dev0) (1.15.0)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.8/site-packages (from requests>=2.6->gocardless-pro~=1.22.0->erpnext==14.0.0.dev0) (2021.10.8)
Requirement already satisfied: idna<3,>=2.5 in ./env/lib/python3.8/site-packages (from requests>=2.6->gocardless-pro~=1.22.0->erpnext==14.0.0.dev0) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./env/lib/python3.8/site-packages (from requests>=2.6->gocardless-pro~=1.22.0->erpnext==14.0.0.dev0) (1.26.8)
Requirement already satisfied: chardet<5,>=3.0.2 in ./env/lib/python3.8/site-packages (from requests>=2.6->gocardless-pro~=1.22.0->erpnext==14.0.0.dev0) (4.0.0)
Requirement already satisfied: oauthlib>=3.0.0 in ./env/lib/python3.8/site-packages (from requests-oauthlib<2.0.0,>=1.3.0->python-youtube~=0.8.0->erpnext==14.0.0.dev0) (3.1.1)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in ./env/lib/python3.8/site-packages (from requests>=2.6->gocardless-pro~=1.22.0->erpnext==14.0.0.dev0) (1.7.1)
Requirement already satisfied: wrapt<2,>=1.10 in ./env/lib/python3.8/site-packages (from deprecated->PyGithub~=1.55->erpnext==14.0.0.dev0) (1.12.1)
Requirement already satisfied: pycparser in ./env/lib/python3.8/site-packages (from cffi>=1.4.1->pynacl>=1.4.0->PyGithub~=1.55->erpnext==14.0.0.dev0) (2.21)
Requirement already satisfied: typing-extensions>=3.7.4 in ./env/lib/python3.8/site-packages (from typing-inspect>=0.4.0->dataclasses-json<0.6.0,>=0.5.3->python-youtube~=0.8.0->erpnext==14.0.0.dev0) (4.1.1)
Requirement already satisfied: mypy-extensions>=0.3.0 in ./env/lib/python3.8/site-packages (from typing-inspect>=0.4.0->dataclasses-json<0.6.0,>=0.5.3->python-youtube~=0.8.0->erpnext==14.0.0.dev0) (0.4.3)
Building wheels for collected packages: jsonobject
  Building wheel for jsonobject (setup.py): started
  Building wheel for jsonobject (setup.py): finished with status 'error'
  Running setup.py clean for jsonobject
Failed to build jsonobject
Installing collected packages: jsonobject, taxjar, erpnext
  Running setup.py install for jsonobject: started
  Running setup.py install for jsonobject: finished with status 'error'

env/bin/pip install -e apps/erpnext > install_erpnext.txt < install_erpnext.txt
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      Warning: 'classifiers' should be a list, got type 'tuple'
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.8
      creating build/lib.linux-x86_64-3.8/jsonobject
      copying jsonobject/__init__.py -> build/lib.linux-x86_64-3.8/jsonobject
      copying jsonobject/exceptions.py -> build/lib.linux-x86_64-3.8/jsonobject
      warning: build_py: byte-compiling is disabled, skipping.

      running build_ext
      building 'jsonobject.api' extension
      creating build/temp.linux-x86_64-3.8
      creating build/temp.linux-x86_64-3.8/jsonobject
      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/frappe/frappe-bench/env/include/python3.8 -c jsonobject/api.c -o build/temp.linux-x86_64-3.8/jsonobject/api.o
      jsonobject/api.c:16:10: fatal error: Python.h: No such file or directory
       #include "Python.h"
                ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for jsonobject
  error: subprocess-exited-with-error

  × Running setup.py install for jsonobject did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      Warning: 'classifiers' should be a list, got type 'tuple'
      running install
      /home/frappe/frappe-bench/env/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.8
      creating build/lib.linux-x86_64-3.8/jsonobject
      copying jsonobject/__init__.py -> build/lib.linux-x86_64-3.8/jsonobject
      copying jsonobject/exceptions.py -> build/lib.linux-x86_64-3.8/jsonobject
      warning: build_py: byte-compiling is disabled, skipping.

      running build_ext
      building 'jsonobject.api' extension
      creating build/temp.linux-x86_64-3.8
      creating build/temp.linux-x86_64-3.8/jsonobject
      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/frappe/frappe-bench/env/include/python3.8 -c jsonobject/api.c -o build/temp.linux-x86_64-3.8/jsonobject/api.o
      jsonobject/api.c:16:10: fatal error: Python.h: No such file or directory
       #include "Python.h"
                ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> jsonobject

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

[ankush]

python 3.8 too? I thought this was only a problem in 3.10.

We have been using 3.8 and 3.9 for quite some time now, even CI runs on 3.8

erpnext/.github/workflows/server-tests-mariadb.yml

Line 60 in 4738367

python-version: 3.8

ref: #29236

Looks like the fix is already there in upstream development branch, they just need to release it 😐 dimagi/jsonobject#183

[barredterra]

@ankush interesting, maybe it has to do with differences in the OS or migrating from an earlier python?

I installed python3.8 via apt on Ubuntu 18.04, then ran bench migrate-env python3.8. Previous python was 3.6.9.

[ankush]

I didn't read the full error message before. I am guessing you need devel version of python from Ubuntu repos.

Default ones only provide an interpreter, devel package comes with libs necessary to compile packages that have C code.

IIRC official docs do mention this somewhere.

This issue isn't related to that dependency being unsupported (which is only a problem on 3.10)

Closing this for now.

[barredterra]

Thanks @ankush, I was able to fix it. These were roughly the commands I ran:

sudo apt install python3.8 python3.8-dev
cd /home/frappe/frappe-bench
# move env to env-old so you can restore it just in case
mv env env-old
# create new env
virtualenv --python python3.8 env
# upgrade pip
env/bin/pip install -U pip
# install apps and requirements
env/bin/pip install -e apps/frappe -e apps/erpnext
# delete env-old (optional)
rm -rf env-old

You can add any custom apps to the last line: -e apps/my-app.