Unable to install kedro

[tharris72]

I have a mac with macOS Catalina Version 10.15.1 on my computer. I installed Conda and created a new conda environment using
conda create -n pydatala_kedro python=3.6
Then I activated the environment using
conda activate pydatala_kedro

Then i tried
pip install kedro

I got an error saying
Successfully built anyconfig PyYAML memory-profiler python-json-logger SQLAlchemy future
Failed to build psutil
ERROR: botocore 1.13.30 has requirement python-dateutil<2.8.1,>=2.1; python_version >= "2.7", but you'll have python-dateutil 2.8.1 which is incompatible.
Running setup.py install for psutil ... error
ERROR: Command errored out with exit status 1:

Then I ran
pip install python-dateutil==2.7
that command worked
Then I tried (again)
pip install kedro

Building wheel for psutil (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /opt/miniconda3/envs/pydatala_kedro/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/y3/wy9bly3d7yg83nqjy49rjx0r0000gn/T/pip-install-_qvx9_nv/psutil/setup.py'"'"'; file='"'"'/private/var/folders/y3/wy9bly3d7yg83nqjy49rjx0r0000gn/T/pip-install-_qvx9_nv/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/y3/wy9bly3d7yg83nqjy49rjx0r0000gn/T/pip-wheel-82ys5j35 --python-tag cp36

   cwd: /private/var/folders/y3/wy9bly3d7yg83nqjy49rjx0r0000gn/T/pip-install-_qvx9_nv/psutil/

Complete output (50 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.7-x86_64-3.6
creating build/lib.macosx-10.7-x86_64-3.6/psutil
copying psutil/_pswindows.py -> build/lib.macosx-10.7-x86_64-3.6/psutil
copying psutil/_common.py -> build/lib.macosx-10.7-x86_64-3.6/psutil
copying psutil/init.py -> build/lib.macosx-10.7-x86_64-3.6/psutil
copying psutil/_psosx.py -> build/lib.macosx-10.7-x86_64-3.6/psutil
copying psutil/_psbsd.py -> build/lib.macosx-10.7-x86_64-3.6/psutil
copying psutil/_psaix.py -> build/lib.macosx-10.7-x86_64-3.6/psutil
copying psutil/_pslinux.py -> build/lib.macosx-10.7-x86_64-3.6/psutil
copying psutil/_compat.py -> build/lib.macosx-10.7-x86_64-3.6/psutil
copying psutil/_psposix.py -> build/lib.macosx-10.7-x86_64-3.6/psutil
copying psutil/_pssunos.py -> build/lib.macosx-10.7-x86_64-3.6/psutil
creating build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_contracts.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_connections.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/runner.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_unicode.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_misc.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_posix.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_linux.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_sunos.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/init.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_aix.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_process.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_bsd.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_system.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_osx.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_memory_leaks.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/test_windows.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
copying psutil/tests/main.py -> build/lib.macosx-10.7-x86_64-3.6/psutil/tests
running build_ext
building 'psutil._psutil_osx' extension
creating build/temp.macosx-10.7-x86_64-3.6
creating build/temp.macosx-10.7-x86_64-3.6/psutil
creating build/temp.macosx-10.7-x86_64-3.6/psutil/arch
creating build/temp.macosx-10.7-x86_64-3.6/psutil/arch/osx
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/miniconda3/envs/pydatala_kedro/include -arch x86_64 -I/opt/miniconda3/envs/pydatala_kedro/include -arch x86_64 -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=567 -DPSUTIL_OSX=1 -I/opt/miniconda3/envs/pydatala_kedro/include/python3.6m -c psutil/_psutil_common.c -o build/temp.macosx-10.7-x86_64-3.6/psutil/_psutil_common.o
In file included from /usr/local/lib/gcc/x86_64-apple-darwin17.5.0/7.3.0/include-fixed/syslimits.h:7:0,
from /usr/local/lib/gcc/x86_64-apple-darwin17.5.0/7.3.0/include-fixed/limits.h:34,
from /opt/miniconda3/envs/pydatala_kedro/include/python3.6m/Python.h:11,
from psutil/_psutil_common.c:9:
/usr/local/lib/gcc/x86_64-apple-darwin17.5.0/7.3.0/include-fixed/limits.h:194:15: fatal error: limits.h: No such file or directory
#include_next <limits.h> /* recurse down to the real one */
^~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1

ERROR: Failed building wheel for psutil
Running setup.py clean for psutil
Failed to build psutil

It seems to be looking for a file named limits.h

/usr/local/lib/gcc/x86_64-apple-darwin17.5.0/7.3.0/include-fixed/limits.h:194:15: fatal error: limits.h: No such file or directory
#include_next <limits.h> /* recurse down to the real one */
^~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1

I'm not sure why the file is missing how to fix it but this seems to be a dependency issue trying to install psutil using the setup.py file

When I try to run
pip install psutil I get the same error as above

Any help could be greatly appreciated

[WillAyd]

I think this is caused by boto/botocore#1872 which seems to have caused issues downstream in a few other packages

[mzjp2]

I think this is caused by boto/botocore#1872 which seems to have caused issues downstream in a few other packages

Don't think so, it only prints an error, but doesn't break the installation. The error here seems to be based on a C-library dependency.

@tharris72 Do you have Xcode tools installed/updated on your system? (xcode-select --install)

If so, can you check what which gcc outputs? If it outputs the gcc that belongs to Conda, can you uninstall that and let python install it using your systme gcc/clang? I've seen issues before on Conda Python 3.6 with conda's gcc. I suspect if you used 3.7 it'd work 🤔

[tharris72]

@mzjp2 I do have xcode on my machine. I ran the code you showed (xcode-select --install) before I posted the original post. I read somewhere that apple moved some of the sdk kits with the latest mac os Catalina update or the one before 10.14. The thing that doesn't make sense to me is that the error I am getting is

/usr/local/lib/gcc/x86_64-apple-darwin17.5.0/7.3.0/include-fixed/limits.h:194:15: fatal error: limits.h: No such file or directory
#include_next <limits.h> /* recurse down to the real one */

but I checked and the limits.h file actually there

I just tried making a conda environment using python 3.7 and I am still getting the same error

[lorenabalan]

I read somewhere that apple moved some of the sdk kits with the latest mac os Catalina update or the one before 10.14.

You might've come across this already, but have you tried running something similar to this: https://programmer.help/blogs/how-to-resolve-pip-install-twisted-error-limits.h-no-such-file-or-directory.html ?

[tharris72]

@lorenabalan
The link is for mac os 10.14. I have 10.15. I tried using the open command on
open /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
but I still get the error

I think the problem is because some of the folders were moved around for mac os 10.15 and I am trying to create a virtual environment the compiler or something else is looking in the wrong place.
The link you gave me helped me to find that there is also a limits.h file in
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/limits.h

The error says the program is looking in
/usr/local/lib/gcc/x86_64-apple-darwin17.5.0/7.3.0/include-fixed/limits.h

I'm guessing that these are the same file. I'm not sure if the program is supposed to be looking in the new folder now that stuff has been moved around. I have tried a few things but they haven't worked. I think it is a path error but I'm just not sure how to fix it.

I also tried making a virtual environment in pyenv and pip installing kedro and am getting the same error.

[yetudada]

@tharris72 We think we might have resolved this issue in the upcoming release of kedro 0.15.5 because we have removed the memory-profiler from core library dependencies and this has removed one of its dependencies on psutils the library which requires you to have Xcode. Please let us know if you will be able install Kedro without any errors then.

[tharris72]

@yetudada No problem. As soon as it is released I can try to install it a gain and let you guys know what happens. Thanks for the help.

[lorenabalan]

Hi @tharris72 , Kedro 0.15.5 is out, let us know if you run into the same problem.

[tharris72]

It worked!
Thanks,
I will close out the issue.

[abhishekh-cpnet]

@lorenabalan @yetudada Hi, I am facing a similar issue with Kedro 0.16.1. I am using Mint Linux.

[mzjp2]

@lorenabalan @yetudada Hi, I am facing a similar issue with Kedro 0.16.1. I am using Mint Linux.

Can you post the output of pip install kedro?

[abhishekh-cpnet]

@mzjp2 I get the error while installing the dependencies.

pip install 'kedro[all]'

[mzjp2]

@mzjp2 I get the error while installing the dependencies.

Can you try the answers outlined here? giampaolo/psutil#1143

[abhishekh-cpnet]

@mzjp2 I get the error while installing the dependencies.

Can you try the answers outlined here? giampaolo/psutil#1143

Thanks @mzjp2, The problem got solved.

Issue solved by:
sudo apt install python3.6-dev
python3 -m pip install psutil
pip install "kedro[all]"

[mzjp2]

Awesome!

[abhishekh-cpnet]

@mzjp2 How do you solve the exception errors while installing kedro["group"]

Exception:
pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'['": Expected stringEnd

[mzjp2]

@mzjp2 How do you solve the exception errors while installing kedro["group"]

Exception:
pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'['": Expected stringEnd

pip install "kedro[group]" not pip install kedro["group"] -- look at the " -- on UNIX, a [ is a builtin command and so needs to be quoted if you want to use the string "[".

[abhishekh-cpnet]

Thanks. It worked