Fails to build on Python 3.11 RC2: fatal error: longintrepr.h: No such file or directory #41
Comments
|
I haven't had time to try to add Python 3.11 compatibility yet. I'll look into this. |
|
This one should be solvable by a simple cython bump |
|
@zhuyifei1999 do you have rough ETA when you'll be able to have a look on the issue? Thanks for any info! |
guppy doesn't use cython. It's direct C code using CPython API.
Let me try this weekend. |
|
The frame optimization in Python 3.11 in python/cpython#88756 made it quite a bit more difficult to support it. Guppy needs the ability to read all locals and globals of all stack frames for traversing purposes, and knowing names of variables (the "relate" part of guppy). Previously this was easy: Every frame will have a Python-visible frame object, so we can traverse the rootstate for frame objects: Lines 309 to 322 in 4cb9fcb and then let frame object traversal do its thing: Lines 199 to 270 in 4cb9fcb But now I need to do frame attributes as "rootstate" attributes since all the intermediate structs are not Python-visible objects. And It's not just traverse I need to implement (the pre-3.11 I can just trverse to the most recent frame object and it'll recursively traverse its f_back, but now I can't rely on this), but also "relate" (finding names of objects), getattr, and dir()... I'll keep working on it. |
Just to fic compilation. It doesn't really work yet. For #41
|
With Python 3.11 becoming more commonly used (especially because of performance improvements), we are looking forward to starting to support it as well. Not wanting to push this issue or change in any way, is there anything to report on the progress of this issue? (please don't feel pushed, I'm just trying to get information). ../Frenck |
|
My apologies it's been on my backburner for way too long. I always had other things to work on that I find more interesting :/ I'll get to it soon. |
Just to fix compilation. It doesn't really work yet. For #41
|
Note to self: Whatever I do here will be broken again by python/cpython@1e197e6 in 3.12. I need to read that commit in more detail some time. |
No relate yet. Dir and getattr lacks support for localsplus. Honestly I'm ot sure how to even do relate for localsplus. It needs both the frame name and the local variable name, which isn't something path supports. Though, we can add it. But then, ByVia classifier would be much less useful if a bunch of local variables share the same name. (Is it possible for, maybe, a pycapsule to represent a _PyInterpreterFrame?) For #41
|
d13f5cc commit above mentions an issue with localsplus. I think I need to explain this one in plain English so I can maybe ask for opinions. As mentioned before python/cpython#88756 in 3.11 changed how frames work. Previously we had (simplified) pystate.h frameobject.h typedef struct _ts {
[...]
PyFrameObject *frame;
[...]
CFrame *cframe;
[...]
} PyThreadState;
typedef struct _frame {
PyObject_VAR_HEAD
struct _frame *f_back; /* previous frame, or NULL */
PyCodeObject *f_code; /* code segment */
PyObject *f_builtins; /* builtin symbol table (PyDictObject) */
PyObject *f_globals; /* global symbol table (PyDictObject) */
PyObject *f_locals; /* local symbol table (any mapping) */
PyObject **f_valuestack; /* points after the last local */
PyObject *f_trace; /* Trace function */
[...]
PyObject *f_localsplus[1]; /* locals+stack, dynamically sized */
} PyFrameObject;The traversal of frames is easy. Since every Python frame is guaranteed a Python-visible "frame object", to find everything each frame uses we can just traverse into these objects. All we have to do is provide accesssors from the "RootState" to every frame in existence. In Python 3.11 this changed. Now we have pystate.h pycore_frame.h: typedef struct _ts {
[...]
CFrame *cframe;
[...]
} PyThreadState;
typedef struct _PyCFrame {
[...]
struct _PyInterpreterFrame *current_frame;
[...]
} _PyCFrame;
typedef struct _PyInterpreterFrame {
/* "Specials" section */
PyFunctionObject *f_func; /* Strong reference */
PyObject *f_globals; /* Borrowed reference */
PyObject *f_builtins; /* Borrowed reference */
PyObject *f_locals; /* Strong reference, may be NULL */
PyCodeObject *f_code; /* Strong reference */
PyFrameObject *frame_obj; /* Strong reference, may be NULL */
/* Linkage section */
struct _PyInterpreterFrame *previous;
[...]
/* Locals and stack */
PyObject *localsplus[1];
} _PyInterpreterFrame;
typedef struct _frame {
PyObject_HEAD
PyFrameObject *f_back; /* previous frame, or NULL */
struct _PyInterpreterFrame *f_frame; /* points to the frame data */
PyObject *f_trace; /* Trace function */
[...]
PyObject *_f_frame_data[1];
} PyFrameObject;Most of the interesting values are in Why is this an issue? Guppy has a feature that it can figure out how to reach an object for memory analysis purposes. Internally this is called "relate", which I think means figuring out the relationship in the path of the traversal. In the README we have >>> from guppy import hpy; h=hpy()
>>> h.heap().byid[0].sp
0: h.Root.i0_modules['os'].__dict__
>>> h.heap()
[...]
>>> _.byvia
Partition of a set of 37968 objects. Total size = 4480344 bytes.
Index Count % Size % Cumulative % Referred Via:
0 1144 3 456096 10 456096 10 '.__dict__'
1 2331 6 412466 9 868562 19 '.__code__'
2 2668 7 254698 6 1123260 25 '.co_code'
3 2088 5 168888 4 1292148 29 '.co_names'
4 637 2 159671 4 1451819 32 '.__doc__', '[0]'
5 514 1 157669 4 1609488 36 "['__doc__']"
6 1856 5 130485 3 1739973 39 '.__qualname__'
7 1571 4 111336 2 1851309 41 '.co_consts'
8 2408 6 111178 2 1962487 44 '.co_lnotab'
9 1467 4 108416 2 2070903 46 '.co_varnames'
<7211 more rows. Type e.g. '_.more' to view.>Let's say you have a lot of large strings; the type classifier will tell you they are strings, but it'll take you a while to find the name of these strings (i.e. the attribute / variable name that refers to these strings). Byvia makes this easy: >>> h.heap()
Partition of a set of 38027 objects. Total size = 4484757 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 10956 29 974101 22 974101 22 str
1 7747 20 537504 12 1511605 34 tuple
2 576 2 481208 11 1992813 44 type
3 2669 7 472264 11 2465077 55 types.CodeType
4 5240 14 377012 8 2842089 63 bytes
5 2448 6 332928 7 3175017 71 function
6 576 2 284384 6 3459401 77 dict of type
7 99 0 183432 4 3642833 81 dict of module
8 300 1 123912 3 3766745 84 dict (no owner)
9 1138 3 81936 2 3848681 86 types.WrapperDescriptorType
<117 more rows. Type e.g. '_.more' to view.>
>>> _[0]
Partition of a set of 10956 objects. Total size = 974101 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 10956 100 974101 100 974101 100 str
>>> _.byvia
Partition of a set of 10956 objects. Total size = 974101 bytes.
Index Count % Size % Cumulative % Referred Via:
0 637 6 159671 16 159671 16 '.__doc__', '[0]'
1 502 5 156901 16 316572 32 "['__doc__']"
2 1856 17 130485 13 447057 46 '.__qualname__'
3 699 6 43171 4 490228 50 'list(_)[0]'
4 509 5 34407 4 524635 54 '[1]'
5 422 4 30315 3 554950 57 '[2]'
6 276 3 18602 2 573552 59 '[3]'
7 248 2 17091 2 590643 61 '[4]'
8 227 2 13788 1 604431 62 '.__name__', '.co_name'
9 186 2 11737 1 616168 63 '[0]'
<2178 more rows. Type e.g. '_.more' to view.>(In this example it's a lot of docstrings) The problem comes when the concern is local variables. The relation originally would show 0 637 6 159671 16 159671 16 ".locals['foo']"Now let's look at 3.11. Guppy's "RootState" maintains a wonderful property that the traversable attributes are, not only just relate-able, but also available by getattr and even dir(). This is Python 3.9: >>> import guppy.heapy.heapyc
>>> import pprint
>>> guppy.heapy.heapyc.RootState is h.Root
True
>>> pprint.pprint(dir(h.Root))
['i0_after_forkers_child',
'i0_after_forkers_parent',
'i0_audit_hooks',
'i0_before_forkers',
'i0_builtins',
'i0_builtins_copy',
'i0_codec_error_registry',
'i0_codec_search_cache',
'i0_codec_search_path',
'i0_dict',
'i0_import_func',
'i0_importlib',
'i0_modules',
'i0_modules_by_index',
'i0_pyexitmodule',
'i0_sysdict',
'i0_t140325531354944_async_exc',
'i0_t140325531354944_async_gen_finalizer',
'i0_t140325531354944_async_gen_firstiter',
'i0_t140325531354944_c_profileobj',
'i0_t140325531354944_c_traceobj',
'i0_t140325531354944_context',
'i0_t140325531354944_curexc_traceback',
'i0_t140325531354944_curexc_type',
'i0_t140325531354944_curexc_value',
'i0_t140325531354944_dict',
'i0_t140325531354944_exc_traceback',
'i0_t140325531354944_exc_type',
'i0_t140325531354944_exc_value',
'i0_t140325531354944_f0']
>>> h.Root.i0_t140325531354944_f0
<frame at 0x7fa013d30680, file '<stdin>', line 1, code <module>>I want to have attribute access for all these values in >>> import guppy.heapy.heapyc
>>> import pprint
>>> pprint.pprint(guppy.heapy.heapyc.RootState)
RootState
>>> pprint.pprint(dir(guppy.heapy.heapyc.RootState))
['i0_after_forkers_child',
'i0_after_forkers_parent',
'i0_audit_hooks',
'i0_before_forkers',
'i0_builtins',
'i0_builtins_copy',
'i0_codec_error_registry',
'i0_codec_search_cache',
'i0_codec_search_path',
'i0_dict',
'i0_import_func',
'i0_importlib',
'i0_modules',
'i0_modules_by_index',
'i0_sysdict',
'i0_t140336295487296_async_exc',
'i0_t140336295487296_async_gen_finalizer',
'i0_t140336295487296_async_gen_firstiter',
'i0_t140336295487296_c_profileobj',
'i0_t140336295487296_c_traceobj',
'i0_t140336295487296_context',
'i0_t140336295487296_curexc_traceback',
'i0_t140336295487296_curexc_type',
'i0_t140336295487296_curexc_value',
'i0_t140336295487296_dict',
'i0_t140336295487296_exc_traceback',
'i0_t140336295487296_exc_type',
'i0_t140336295487296_exc_value',
'i0_t140336295487296_f0_f_builtins',
'i0_t140336295487296_f0_f_code',
'i0_t140336295487296_f0_f_func',
'i0_t140336295487296_f0_f_globals',
'i0_t140336295487296_f0_f_locals',
'i0_t140336295487296_f0_frame_obj']The change is that the attributes of A bit of background for those who are unfamiliar with Python's bytecode architecture: Python is a stack-based architecture, and very similar to Java bytecode in some ways. Instead of having some sort of registers, temporaries are stored on a stack. This is a disassembly of >>> dis.dis(lambda a: a+a)
# push(argument[0])
1 0 LOAD_FAST 0 (a)
# push(argument[0])
2 LOAD_FAST 0 (a)
# push(add(pop(), pop()))
4 BINARY_ADD
# return pop()
6 RETURN_VALUEThe first few elements of the One might say... okay, one can just >>> p = h.heap().byid[0]
>>> p.sp
0: h.Root.i0_modules['os'].__dict__
>>> h.Root.i0_modules['os'].__dict__ is p.theone
TrueNow, for a frame object, Actually accessing static PyObject *
frame_getlocals(PyFrameObject *f, void *closure)
{
if (PyFrame_FastToLocalsWithError(f) < 0)
return NULL;
PyObject *locals = f->f_frame->f_locals;
Py_INCREF(locals);
return locals;
}
static PyGetSetDef frame_getsetlist[] = {
[...]
{"f_locals", (getter)frame_getlocals, NULL, NULL},
[...]
};I'm not sure I want getattr for RootState to invoke a lazy loader, when, say, accessing Another thing comes with the relate. If One might say, we could maybe create multiple components in the path somehow. The thing is, in the path, not only the "path" itself is recorded, but every object in between are, too: >>> p.sp
0: h.Root.i0_modules['os'].__dict__
>>> _[0]
"%s.i0_modules['os'].__dict__"
>>> [type(t) for t in _.path]
[<class 'guppy.heapy.UniSet.IdentitySetSingleton'>, <class 'guppy.heapy.Path.Based_R_ATTRIBUTE'>, <class 'guppy.heapy.UniSet.IdentitySetSingleton'>, <class 'guppy.heapy.Path.Based_R_INDEXVAL'>, <class 'guppy.heapy.UniSet.IdentitySetSingleton'>, <class 'guppy.heapy.Path.Based_R_ATTRIBUTE'>, <class 'guppy.heapy.UniSet.IdentitySetSingleton'>]
>>> p.sp[0].path[0].theone
RootState
>>> p.sp[0].path[2].theone is h.Root.i0_modules
True
>>> p.sp[0].path[4].theone
<module 'os' from '/usr/lib/python3.9/os.py'>
>>> p.sp[0].path[6].theone is p.theone
TrueUnless I want to invoke the lazy loader for every frame when guppy generates the profile, I'd need to make an IdentitySet, but for structs that are not PyObject-s. Some sort of virtual IdentitySet. Guppy doesn't support this, and I have no idea what it would take to make it happen. Could try but it's potentially very complicated. Or alternatively, we invoke all the lazy loader for the frames when profiling. Honestly, probably not too bad of an idea considering it's an interactive profiler that's already very expensive (traverses the whole reference graph, and generates a lot of objects internally). What we would lose on is the ability to figure out which frames objects has been lazy loaded and which have not, prior to running the profiler. At least this isn't a feature regression, since for Python < 3.11 they are always loaded. (Note to self: Or we drop some features... Thoughts? CC @svenil too |
guppy3 is not python3.11 compat zhuyifei1999/guppy3#41 This service will return if and when guppy3 becomes python3.11 compat
* Remove profiler.memory service guppy3 is not python3.11 compat zhuyifei1999/guppy3#41 This service will return if and when guppy3 becomes python3.11 compat * squash * temp remove * temp dump tests * temp dump tests * drop a few more to get a run * drop a few more to get a run * Account for changed python3.11 enum.IntFlag behavior in zha There may be additional changes needed, but I could only see what needed to be updated based on the tests * merge * restore * restore * legacy value * tweak a bit for the python 3.11 timings * block cchardet * conditional * adjust est * test * not yet * tweak * give a little leeway for timing * Fix otbr tests * Increase database test timeout It looks like we need a little more time to run with the addiitonal tests in #87019 * Increase database test timeout It looks like we need a little more time to run with the addiitonal tests in #87019 * Fix aprs tests with python 3.11 * merge fix * hints * Update homeassistant/package_constraints.txt * Update script/gen_requirements_all.py * Constrain uamqp for Python 3.10 only * Bump vulcan-api to 2.3.0 see kapi2289/vulcan-api#126 see #88038 see home-assistant/docker#260 * add ban * Bump python-matter-server to 2.1.1 * revert * Update tests/asyncio_legacy.py --------- Co-authored-by: Erik <erik@montnemery.com> Co-authored-by: Franck Nijhof <git@frenck.dev> Co-authored-by: Marcel van der Veldt <m.vanderveldt@outlook.com>
* Remove profiler.memory service guppy3 is not python3.11 compat zhuyifei1999/guppy3#41 This service will return if and when guppy3 becomes python3.11 compat * squash * temp remove * temp dump tests * temp dump tests * drop a few more to get a run * drop a few more to get a run * Account for changed python3.11 enum.IntFlag behavior in zha There may be additional changes needed, but I could only see what needed to be updated based on the tests * merge * restore * restore * legacy value * tweak a bit for the python 3.11 timings * block cchardet * conditional * adjust est * test * not yet * tweak * give a little leeway for timing * Fix otbr tests * Increase database test timeout It looks like we need a little more time to run with the addiitonal tests in home-assistant#87019 * Increase database test timeout It looks like we need a little more time to run with the addiitonal tests in home-assistant#87019 * Fix aprs tests with python 3.11 * merge fix * hints * Update homeassistant/package_constraints.txt * Update script/gen_requirements_all.py * Constrain uamqp for Python 3.10 only * Bump vulcan-api to 2.3.0 see kapi2289/vulcan-api#126 see home-assistant#88038 see home-assistant/docker#260 * add ban * Bump python-matter-server to 2.1.1 * revert * Update tests/asyncio_legacy.py --------- Co-authored-by: Erik <erik@montnemery.com> Co-authored-by: Franck Nijhof <git@frenck.dev> Co-authored-by: Marcel van der Veldt <m.vanderveldt@outlook.com>
|
With |
|
I was waiting for a response to #41 (comment) then I forgot (too many things I'm working on)... oops I guess with a lack of opinions I'll just go with the simplest solution:
Will do. |
And only use the public APIs with PyFrameObjects. The trace won't be as accurate anymore (because the profile will create all these frame objects rather than just observing them) but at least there won't be a regression (Py < 3.11 always creates these objects). For #41
Python built-in only traverses when FRAME_OWNED_BY_FRAME_OBJECT. :( For #41
Python added _PyType_PreHeaderSize for the preheader and it's no longer just sizeof(PyGC_Head) See python/cpython#101430 For #41
The paths are sorted in an unexpected way. Test both cases (though only one case happens per Python version), and document what is going on behind the test case. In Python 3.11 both the object and the list have the same size. The object grew by 8 bytes compared to Python 3.10. For #41
Some referenced attributes are no longer slotted. I'm not sure how to get a fallback to default relate for slotted attributes so I'm just going to make relate do the entire thing. For #41
Traverse from the top (newest) of stack to oldest, as a linked reference, instead of linking every frame against rootstate. This behavior is expected by the tests. Also fix f_back processing because Py 3.11 only tracks f_back when FRAME_OWNED_BY_FRAME_OBJECT, i.e. the frame finished executing. We don't always have this luxary. For #41
There seems to be no way to distinguish managed dict references vs slots references. I'm not sure how to better do this but I'll just materialize all the dicts so the reference chain can be consistent. For #41
Refpat seems also sorted by size. Though I'm not sure which comes first when both entries are at the same size. For #41
Just to fix compilation. It doesn't really work yet. For #41
No relate yet. Dir and getattr lacks support for localsplus. Honestly I'm ot sure how to even do relate for localsplus. It needs both the frame name and the local variable name, which isn't something path supports. Though, we can add it. But then, ByVia classifier would be much less useful if a bunch of local variables share the same name. (Is it possible for, maybe, a pycapsule to represent a _PyInterpreterFrame?) For #41
And only use the public APIs with PyFrameObjects. The trace won't be as accurate anymore (because the profile will create all these frame objects rather than just observing them) but at least there won't be a regression (Py < 3.11 always creates these objects). For #41
Python built-in only traverses when FRAME_OWNED_BY_FRAME_OBJECT. :( For #41
Python added _PyType_PreHeaderSize for the preheader and it's no longer just sizeof(PyGC_Head) See python/cpython#101430 For #41
The paths are sorted in an unexpected way. Test both cases (though only one case happens per Python version), and document what is going on behind the test case. In Python 3.11 both the object and the list have the same size. The object grew by 8 bytes compared to Python 3.10. For #41
Some referenced attributes are no longer slotted. I'm not sure how to get a fallback to default relate for slotted attributes so I'm just going to make relate do the entire thing. For #41
Traverse from the top (newest) of stack to oldest, as a linked reference, instead of linking every frame against rootstate. This behavior is expected by the tests. Also fix f_back processing because Py 3.11 only tracks f_back when FRAME_OWNED_BY_FRAME_OBJECT, i.e. the frame finished executing. We don't always have this luxary. For #41
There seems to be no way to distinguish managed dict references vs slots references. I'm not sure how to better do this but I'll just materialize all the dicts so the reference chain can be consistent. For #41
Refpat seems also sorted by size. Though I'm not sure which comes first when both entries are at the same size. For #41
|
Gonna leave this open till I get the release done later |
|
v3.1.3 released with support for Python 3.11 Edit: With all the pre-built wheels too ;) https://pypi.org/project/guppy3/3.1.3/#files |
Cannot get this package to build on Python 3.11:
Upstream: home-assistant/core#74650
Build output:
Ref: cython/cython#4461
The text was updated successfully, but these errors were encountered: