psutil 1.1.0 won't import on Centos 6.4

[giampaolo]

From arn...@infinidat.com on October 01, 2013 21:03:51

The newest version of psutils on pypi doesn't work for us.

>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/psutil/__init__.py", line 74, in <module>
  File "build/bdist.linux-x86_64/egg/psutil/_pslinux.py", line 21, in <module>
  File "build/bdist.linux-x86_64/egg/_psutil_linux.py", line 7, in <module>
  File "build/bdist.linux-x86_64/egg/_psutil_linux.py", line 6, in __bootstrap__
ImportError: 
/root/.python-eggs/psutil-1.1.0-py2.7-linux-x86_64.egg-tmp/_psutil_linux.so: 
undefined symbol: prlimit


>>> import platform
>>> platform.platform()
'Linux-2.6.32-358.el6.x86_64-x86_64-with-centos-6.4-Final'

Original issue: http://code.google.com/p/psutil/issues/detail?id=438

[giampaolo]

From m...@matt-good.net on October 01, 2013 14:12:49

We ran into this as well.  We're using RHEL with a 2.6.32 kernel, which is 
prior to the introduction of the prlimit API.

I've attached a patch that would conditionally include the code that depends on 
prlimit.  The change to psutil/__init__.py is a bit ugly, since it reproduces 
the list of some conditionally defined RLIMIT constants already present in 
psutil/_pslinux.py.  It's also updating locals() which may mess with some code 
inspection, so suggestions are welcome.

For now we've rolled back to the prior release of psutil, but I thought I'd 
take a stab at a patch.

Attachment: prlimit.diff

[giampaolo]

From g.rodola on October 03, 2013 12:32:29

Fixed in revision 372bbd8245db . Could you please try latest HG version, run 
tests and report back?

Status: Started
Labels: -Priority-Medium Priority-High OpSys-Linux Milestone-1.2.0
Compatibility

[giampaolo]

From m...@matt-good.net on October 03, 2013 12:51:36

That almost fixed it, but you're using "#ifdef HAS_PRLIMIT" which is always 
true, since on older kernels that's defined, but "false".  I changed it to "#if 
HAS_PRLIMIT" and it's working for us.


diff -r 2192b1ed7a01 psutil/_psutil_linux.c
--- a/psutil/_psutil_linux.c    Thu Oct 03 21:31:17 2013 +0200
+++ b/psutil/_psutil_linux.c    Thu Oct 03 12:49:30 2013 -0700
@@ -99,7 +99,7 @@
 #endif


-#ifdef HAS_PRLIMIT
+#if HAS_PRLIMIT
 /*
  * A wrapper around prlimit(2); sets process resource limits.
  * This can be used for both get and set, in which case extra
@@ -331,7 +331,7 @@
      {"ioprio_set", linux_ioprio_set, METH_VARARGS,
         "Set process I/O priority"},
 #endif
-#ifdef HAS_PRLIMIT
+#if HAS_PRLIMIT
      {"prlimit", linux_prlimit, METH_VARARGS,
         "Get or set process resource limits."},
 #endif
@@ -408,7 +408,7 @@
 #endif


-#ifdef HAS_PRLIMIT
+#if HAS_PRLIMIT
     PyModule_AddIntConstant(module, "RLIM_INFINITY", RLIM_INFINITY);
     PyModule_AddIntConstant(module, "RLIMIT_AS", RLIMIT_AS);
     PyModule_AddIntConstant(module, "RLIMIT_CORE", RLIMIT_CORE);

[giampaolo]

From g.rodola on October 03, 2013 13:09:35

Mmm that doesn't work for me on Ubuntu 12.04 (meaning it thinks prlimit() is 
not available).

[giampaolo]

From m...@matt-good.net on October 03, 2013 14:33:47

Yeah, I guess this really needs an autoconf-style check to attempt compiling a 
short program that uses prlimit.  Looks like there's a patch proposed for 
Python 3.4 that would add prlimit, using autoconf to define HAVE_PRLIMIT: 
http://bugs.python.org/file30685/prlimit2.patch Unfortunately, I'm not sure 
what would be recommended to do that with distutils extension modules.

[giampaolo]

From g.rodola on October 03, 2013 14:53:45

I'm not an expert of C related tools but I think we might determine the kernel 
version in setup.py in order to use a macro similar to this:

Extension(
   ...
   define_macros=[('HAVE_PRLIMIT', 1 if get_kernel_ver() > (2, 6, 36) else 0)]
   ...
)

I'm not particularly thrilled at this perspective though because 
get_kernel_ver() has to be rock solid.

[giampaolo]

From g.rodola on October 03, 2013 14:59:17

A quick research on Google shows that CentOS is not the only distro who got 
bitten by this: 
https://www.google.com/search?q=python+prlimit&oq=python+prlimit&aqs=chrome..69i57j69i60l2.1625j0&sourceid=chrome&ie=UTF-8#q=psutil+prlimit
 I guess we need to fix this quickly.

[giampaolo]

From g.rodola on October 03, 2013 16:46:03

Please check out revision 055e8d480462 .
After Googling around I noticed some people who uses #ifdef __NR_prlimit64.
I took a look at latest Linux source code, grepped for "prlimit" (case 
insensitive search) and that looks like the only reasonable definition at hand 
we could use.

man pages also states:

> The  prlimit() system call is available since Linux 2.6.36.  
> Library support is available since glibc 2.13.

I'm not sure how and whether we should check the glibc requirement (my best 
guess is we don't). 
AFAICT latest HG version works fine on my Ubuntu 64-bit.
Please report back how it works for you on CentOS.

[giampaolo]

From arn...@infinidat.com on October 04, 2013 01:25:28

I'm afraid it still looks the same on our CentOS, with revision 055e8d480462 .

btw the glibc version on CentOS 6.4 is 2.12. If I understand correctly, you 
won't use this system call if the kernel version is as old...?

Please let me know if I can help from our end.

[giampaolo]

From g.rodola on October 04, 2013 12:49:30

Still looks the same means you keep getting "undefined symbol: prlimit"?

> btw the glibc version on CentOS 6.4 is 2.12. If I understand 
> correctly, you won't use this system call if the kernel version 
> is as old...?

Currently there are no glib version checks and I don't know if there should be.
In summary I'm not sure what to do. =)
I guess now I'll install CentOS so that at least I can reproduce the problem.
Will report back in a while.

[giampaolo]

From g.rodola on October 04, 2013 13:38:08

Ok, tried just now with your same configuration:

[root@centos psutil]# uname -a
Linux centos 2.6.32-358.el6.i686 #1 SMP Thu Feb 21 21:50:49 UTC 2013 i686 i686 
i386 GNU/Linux

...and I'm able to compile and run tests.
prlimit() related functions are not executed as expected.
Can you provide more info on what does not work exactly?

[giampaolo]

From arn...@infinidat.com on October 04, 2013 15:39:14

Still getting "undefined symbol: prlimit" on import.
Attaching full output.

Attachment: output.txt

[giampaolo]

From g.rodola on October 04, 2013 17:30:45

> /root/python/include/python2.7/pyconfig.h:1151:1: warning: "_GNU_SOURCE" redefined

I have fixed this warning in a revision which is older than the one you're 
supposed to be using. 
Are you sure this isn't something else?
Please try:

python setup.py clean
python setup.py install
python

...and then in the interpreter:

>>> import os, sys
>>> sys.path.remove(os.getcwd())
>>> import psutil

[giampaolo]

From arn...@infinidat.com on October 05, 2013 11:32:12

I'm using the latest revision (I added 'hg identify' to the output).
That warning doesn't comes up when I use the system Python (2.6.6. I was using 
Python 2.7.5 before). The system python fails to import too - same message...
I'm adding the output again, this time using the system python.

sys.path.remove(os.getcwd()) doesn't work, because sys.path doesn't contain the 
current directory.

Attachment: output.txt

[giampaolo]

From g.rodola on October 05, 2013 11:47:40

If you remove HAVE_PRLIMIT definition, like this, does it still crash?

diff --git a/psutil/_psutil_linux.c b/psutil/_psutil_linux.c
--- a/psutil/_psutil_linux.c
+++ b/psutil/_psutil_linux.c
@@ -27,10 +27,6 @@

  // Linux >= 2.6.13
 #define HAVE_IOPRIO defined(__NR_ioprio_get) && defined(__NR_ioprio_set)
- // Linux >= 2.6.36 kernels, glibc >= 2.13
-#ifdef __NR_prlimit64
-    #define HAVE_PRLIMIT
-#endif


My doubt is that for some reason your python interpreter is using an open 
version of _psutil_linux.so.

[giampaolo]

From g.rodola on October 05, 2013 11:52:25

s/open/old

[giampaolo]

From arn...@infinidat.com on October 05, 2013 12:19:27

This resolves the problem, import now succeeds.
I tried to compile a simple program with ifdef, looks like __NR_prlimit64 is defined...

I also see this here:

[root@host-ci08 ~]# grep -R prlimit /usr/include/ | grep define
/usr/include/bits/syscall.h:#define SYS_prlimit64 __NR_prlimit64
/usr/include/asm-generic/unistd.h:#define __NR_prlimit64 261
/usr/include/asm/unistd_64.h:#define __NR_prlimit64             302
/usr/include/asm/unistd_32.h:/* #define __NR_prlimit64      340 */

[giampaolo]

From g.rodola on October 05, 2013 12:21:34

So latest HG version works for you, is that right?

[giampaolo]

From arn...@infinidat.com on October 05, 2013 12:32:32

Negative. The workaround you suggested works; the latest revision on hg does not.

[giampaolo]

From g.rodola on October 05, 2013 12:48:36

I suggested that just to understand whether your python interpreter detected 
the change, it was not a workaround.

OK, I still cannot reproduce the issue and ran out of no ideas. Do you think 
perhaps you can give me SSH access to that box so that I can try it myself?

[giampaolo]

From mariogiov on October 07, 2013 02:53:26

I'm getting the same error on Scientific Linux 6.4 (kernel 
v2.6.32-358.18.1.el6.x86_64) with Python 2.7.5 (built locally). Attaching 
output (apologies that some of it is in Swedish but you can probably make out 
the meaning nonetheless). Haven't had time to try the new revision you suggest 
yet but I will probably give it a whirl later today.

Attachment: psutil-err_scilinux-6.4.out

[giampaolo]

From g.rodola on October 07, 2013 03:00:22

Yes please, use latest HG version (hg clone 
https://g.rodola@code.google.com/p/psutil/ in which I already attempted to 
provide a fix.

[giampaolo]

From mariogiov on October 07, 2013 03:23:15

Same import trouble (using commit 1623:b1e6e9da489e). Attaching output for that 
as well. I also tried with Python v2.7.1 but that made no difference. Hmm.

Attachment: psutil-err_scilinux-6.4_hg-release.out

[giampaolo]

From g.rodola on October 07, 2013 13:02:57

Ok, I should have made it. Please try revision c41b9ce21b04 .

[giampaolo]

From arn...@infinidat.com on October 07, 2013 14:01:11

Looks good. Revision c41b9ce21b04 builds and imports successfully on our CentOS machine.

[giampaolo]

From g.rodola on October 07, 2013 16:06:11

I released 1.1.1 version including this just now. Closing.

Status: Fixed
Labels: -Milestone-1.2.0 Milestone-1.1.1

[giampaolo]

From g...@rzn.co.il on October 08, 2013 05:06:02

verified:

[root@host-ci08 ~]# easy_install -i http://pypi.python.org/simple psutil
Searching for psutil
Reading http://pypi.python.org/simple/psutil/ Reading 
https://code.google.com/p/psutil/ Best match: psutil 1.1.1
Downloading 
https://pypi.python.org/packages/source/p/psutil/psutil-1.1.1.tar.gz#md5=24430ee6486be2f1a960d9ce4dc87ad0
 Processing psutil-1.1.1.tar.gz
Writing /tmp/easy_install-TA3D6H/psutil-1.1.1/setup.cfg
Running psutil-1.1.1/setup.py -q bdist_egg --dist-dir 
/tmp/easy_install-TA3D6H/psutil-1.1.1/egg-dist-tmp-dSVuFW
zip_safe flag not set; analyzing archive contents...
Adding psutil 1.1.1 to easy-install.pth file

Installed 
/root/python/lib64/python2.7/site-packages/psutil-1.1.1-py2.7-linux-x86_64.egg
Processing dependencies for psutil
Finished processing dependencies for psutil
[root@host-ci08 ~]# ipython
impoer Python 2.7.5 (default, Sep 30 2013, 17:45:26)
Type "copyright", "credits" or "license" for more information.

IPython 1.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import psutil

In [2]: psutil.__
psutil.__all__           psutil.__dict__          psutil.__getattribute__  
psutil.__name__          psutil.__reduce__        psutil.__sizeof__
psutil.__builtins__      psutil.__doc__           psutil.__hash__          
psutil.__new__           psutil.__reduce_ex__     psutil.__str__
psutil.__class__         psutil.__file__          psutil.__init__          
psutil.__package__       psutil.__repr__          psutil.__subclasshook__
psutil.__delattr__       psutil.__format__        psutil.__loader__        
psutil.__path__          psutil.__setattr__       psutil.__version__

In [2]: psutil.__version__
Out[2]: '1.1.1'

[giampaolo]

From sv24i...@gmail.com on October 19, 2013 14:40:23

Sorry but it seems I get the same issue on Oracle Enterprise Linux 6.4
(oraclelinux-release-6Server-4.0.4.x86_64)

Python 2.6.6 and psutil 1.1.1

>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File 
"/usr/lib64/python2.6/site-packages/psutil-1.1.1-py2.6-linux-x86_64.egg/psutil/__init__.py",
 line 89, in <module>
    import psutil._pslinux as _psplatform
  File 
"/usr/lib64/python2.6/site-packages/psutil-1.1.1-py2.6-linux-x86_64.egg/psutil/_pslinux.py",
 line 21, in <module>
    import _psutil_linux
ImportError: 
/usr/lib64/python2.6/site-packages/psutil-1.1.1-py2.6-linux-x86_64.egg/_psutil_linux.so:
 undefined symbol: prlimit

Any idea ?

Thanks in advance

[giampaolo]

From g.rodola on October 20, 2013 04:05:55

Please try revision 5bdae7b71188 .

[giampaolo]

From sv24i...@gmail.com on October 20, 2013 07:55:10

Hi,

The revision 5bdae7b71188 works !

Thanks