diff --git a/Doc/c-api/apiabiversion.rst b/Doc/c-api/apiabiversion.rst index f6c8284daeacb0..96050f59bd5250 100644 --- a/Doc/c-api/apiabiversion.rst +++ b/Doc/c-api/apiabiversion.rst @@ -6,9 +6,13 @@ API and ABI Versioning *********************** + +Build-time version constants +---------------------------- + CPython exposes its version number in the following macros. -Note that these correspond to the version code is **built** with, -not necessarily the version used at **run time**. +Note that these correspond to the version code is **built** with. +See :c:var:`Py_Version` for the version used at **run time**. See :ref:`stable` for a discussion of API and ABI stability across versions. @@ -37,37 +41,83 @@ See :ref:`stable` for a discussion of API and ABI stability across versions. .. c:macro:: PY_VERSION_HEX The Python version number encoded in a single integer. + See :c:func:`Py_PACK_FULL_VERSION` for the encoding details. - The underlying version information can be found by treating it as a 32 bit - number in the following manner: - - +-------+-------------------------+-------------------------+--------------------------+ - | Bytes | Bits (big endian order) | Meaning | Value for ``3.4.1a2`` | - +=======+=========================+=========================+==========================+ - | 1 | 1-8 | ``PY_MAJOR_VERSION`` | ``0x03`` | - +-------+-------------------------+-------------------------+--------------------------+ - | 2 | 9-16 | ``PY_MINOR_VERSION`` | ``0x04`` | - +-------+-------------------------+-------------------------+--------------------------+ - | 3 | 17-24 | ``PY_MICRO_VERSION`` | ``0x01`` | - +-------+-------------------------+-------------------------+--------------------------+ - | 4 | 25-28 | ``PY_RELEASE_LEVEL`` | ``0xA`` | - + +-------------------------+-------------------------+--------------------------+ - | | 29-32 | ``PY_RELEASE_SERIAL`` | ``0x2`` | - +-------+-------------------------+-------------------------+--------------------------+ + Use this for numeric comparisons, for example, + ``#if PY_VERSION_HEX >= ...``. - Thus ``3.4.1a2`` is hexversion ``0x030401a2`` and ``3.10.0`` is - hexversion ``0x030a00f0``. - Use this for numeric comparisons, e.g. ``#if PY_VERSION_HEX >= ...``. - - This version is also available via the symbol :c:var:`Py_Version`. +Run-time version +---------------- .. c:var:: const unsigned long Py_Version - The Python runtime version number encoded in a single constant integer, with - the same format as the :c:macro:`PY_VERSION_HEX` macro. + The Python runtime version number encoded in a single constant integer. + See :c:func:`Py_PACK_FULL_VERSION` for the encoding details. This contains the Python version used at run time. + Use this for numeric comparisons, for example, ``if (Py_Version >= ...)``. + .. versionadded:: 3.11 -All the given macros are defined in :source:`Include/patchlevel.h`. + +Bit-packing macros +------------------ + +.. c:function:: uint32_t Py_PACK_FULL_VERSION(int major, int minor, int micro, int release_level, int release_serial) + + Return the given version, encoded as a single 32-bit integer with + the following structure: + + +------------------+-------+----------------+-----------+--------------------------+ + | | No. | | | Example values | + | | of | | +-------------+------------+ + | Argument | bits | Bit mask | Bit shift | ``3.4.1a2`` | ``3.10.0`` | + +==================+=======+================+===========+=============+============+ + | *major* | 8 | ``0xFF000000`` | 24 | ``0x03`` | ``0x03`` | + +------------------+-------+----------------+-----------+-------------+------------+ + | *minor* | 8 | ``0x00FF0000`` | 16 | ``0x04`` | ``0x0A`` | + +------------------+-------+----------------+-----------+-------------+------------+ + | *micro* | 8 | ``0x0000FF00`` | 8 | ``0x01`` | ``0x00`` | + +------------------+-------+----------------+-----------+-------------+------------+ + | *release_level* | 4 | ``0x000000F0`` | 4 | ``0xA`` | ``0xF`` | + +------------------+-------+----------------+-----------+-------------+------------+ + | *release_serial* | 4 | ``0x0000000F`` | 0 | ``0x2`` | ``0x0`` | + +------------------+-------+----------------+-----------+-------------+------------+ + + For example: + + +-------------+------------------------------------+-----------------+ + | Version | ``Py_PACK_FULL_VERSION`` arguments | Encoded version | + +=============+====================================+=================+ + | ``3.4.1a2`` | ``(3, 4, 1, 0xA, 2)`` | ``0x030401a2`` | + +-------------+------------------------------------+-----------------+ + | ``3.10.0`` | ``(3, 10, 0, 0xF, 0)`` | ``0x030a00f0`` | + +-------------+------------------------------------+-----------------+ + + Out-of range bits in the arguments are ignored. + That is, the macro can be defined as: + + .. code-block:: c + + #ifndef Py_PACK_FULL_VERSION + #define Py_PACK_FULL_VERSION(X, Y, Z, LEVEL, SERIAL) ( \ + (((X) & 0xff) << 24) | \ + (((Y) & 0xff) << 16) | \ + (((Z) & 0xff) << 8) | \ + (((LEVEL) & 0xf) << 4) | \ + (((SERIAL) & 0xf) << 0)) + #endif + + ``Py_PACK_FULL_VERSION`` is primarily a macro, intended for use in + ``#if`` directives, but it is also available as an exported function. + + .. versionadded:: 3.14 + +.. c:function:: uint32_t Py_PACK_VERSION(int major, int minor) + + Equivalent to ``Py_PACK_FULL_VERSION(major, minor, 0, 0, 0)``. + The result does not correspond to any Python release, but is useful + in numeric comparisons. + + .. versionadded:: 3.14 diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index 3201bdc82691f4..209056ef2f8bce 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -229,12 +229,24 @@ There are three ways strings and buffers can be converted to C: Numbers ------- +These formats allow representing Python numbers or single characters as C numbers. +Formats that require :class:`int`, :class:`float` or :class:`complex` can +also use the corresponding special methods :meth:`~object.__index__`, +:meth:`~object.__float__` or :meth:`~object.__complex__` to convert +the Python object to the required type. + +For signed integer formats, :exc:`OverflowError` is raised if the value +is out of range for the C type. +For unsigned integer formats, no range checking is done --- the +most significant bits are silently truncated when the receiving field is too +small to receive the value. + ``b`` (:class:`int`) [unsigned char] - Convert a nonnegative Python integer to an unsigned tiny int, stored in a C + Convert a nonnegative Python integer to an unsigned tiny integer, stored in a C :c:expr:`unsigned char`. ``B`` (:class:`int`) [unsigned char] - Convert a Python integer to a tiny int without overflow checking, stored in a C + Convert a Python integer to a tiny integer without overflow checking, stored in a C :c:expr:`unsigned char`. ``h`` (:class:`int`) [short int] @@ -307,7 +319,7 @@ Other objects .. _o_ampersand: -``O&`` (object) [*converter*, *anything*] +``O&`` (object) [*converter*, *address*] Convert a Python object to a C variable through a *converter* function. This takes two arguments: the first is a function, the second is the address of a C variable (of arbitrary type), converted to :c:expr:`void *`. The *converter* @@ -321,14 +333,20 @@ Other objects the conversion has failed. When the conversion fails, the *converter* function should raise an exception and leave the content of *address* unmodified. - If the *converter* returns ``Py_CLEANUP_SUPPORTED``, it may get called a + .. c:macro:: Py_CLEANUP_SUPPORTED + :no-typesetting: + + If the *converter* returns :c:macro:`!Py_CLEANUP_SUPPORTED`, it may get called a second time if the argument parsing eventually fails, giving the converter a chance to release any memory that it had already allocated. In this second call, the *object* parameter will be ``NULL``; *address* will have the same value as in the original call. + Examples of converters: :c:func:`PyUnicode_FSConverter` and + :c:func:`PyUnicode_FSDecoder`. + .. versionchanged:: 3.1 - ``Py_CLEANUP_SUPPORTED`` was added. + :c:macro:`!Py_CLEANUP_SUPPORTED` was added. ``p`` (:class:`bool`) [int] Tests the value passed in for truth (a boolean **p**\ redicate) and converts @@ -344,12 +362,6 @@ Other objects in *items*. The C arguments must correspond to the individual format units in *items*. Format units for sequences may be nested. -It is possible to pass "long" integers (integers whose value exceeds the -platform's :c:macro:`LONG_MAX`) however no proper range checking is done --- the -most significant bits are silently truncated when the receiving field is too -small to receive the value (actually, the semantics are inherited from downcasts -in C --- your mileage may vary). - A few other characters have a meaning in a format string. These may not occur inside nested parentheses. They are: diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index dd63dd013e32dc..97996a6f69dd22 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1492,6 +1492,17 @@ All of the following functions must be called after :c:func:`Py_Initialize`. .. versionadded:: 3.8 + +.. c:function:: PyObject* PyUnstable_InterpreterState_GetMainModule(PyInterpreterState *interp) + + Return a :term:`strong reference` to the ``__main__`` `module object `_ + for the given interpreter. + + The caller must hold the GIL. + + .. versionadded:: 3.13 + + .. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag) Type of a frame evaluation function. diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index c688afdca8231d..7a7d39aea20baf 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -216,6 +216,38 @@ Operating System Utilities The function now uses the UTF-8 encoding on Windows if :c:member:`PyPreConfig.legacy_windows_fs_encoding` is zero. +.. c:function:: FILE* Py_fopen(PyObject *path, const char *mode) + + Similar to :c:func:`!fopen`, but *path* is a Python object and + an exception is set on error. + + *path* must be a :class:`str` object, a :class:`bytes` object, + or a :term:`path-like object`. + + On success, return the new file pointer. + On error, set an exception and return ``NULL``. + + The file must be closed by :c:func:`Py_fclose` rather than calling directly + :c:func:`!fclose`. + + The file descriptor is created non-inheritable (:pep:`446`). + + The caller must hold the GIL. + + .. versionadded:: next + + +.. c:function:: int Py_fclose(FILE *file) + + Close a file that was opened by :c:func:`Py_fopen`. + + On success, return ``0``. + On error, return ``EOF`` and ``errno`` is set to indicate the error. + In either case, any further access (including another call to + :c:func:`Py_fclose`) to the stream results in undefined behavior. + + .. versionadded:: next + .. _systemfunctions: diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index dcbc8804cd6b89..f19b86a8dbfb66 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -786,16 +786,25 @@ Functions encoding to and decoding from the :term:`filesystem encoding and error handler` (:pep:`383` and :pep:`529`). To encode file names to :class:`bytes` during argument parsing, the ``"O&"`` -converter should be used, passing :c:func:`PyUnicode_FSConverter` as the +converter should be used, passing :c:func:`!PyUnicode_FSConverter` as the conversion function: .. c:function:: int PyUnicode_FSConverter(PyObject* obj, void* result) - ParseTuple converter: encode :class:`str` objects -- obtained directly or + :ref:`PyArg_Parse\* converter `: encode :class:`str` objects -- obtained directly or through the :class:`os.PathLike` interface -- to :class:`bytes` using :c:func:`PyUnicode_EncodeFSDefault`; :class:`bytes` objects are output as-is. - *result* must be a :c:expr:`PyBytesObject*` which must be released when it is - no longer used. + *result* must be an address of a C variable of type :c:expr:`PyObject*` + (or :c:expr:`PyBytesObject*`). + On success, set the variable to a new :term:`strong reference` to + a :ref:`bytes object ` which must be released + when it is no longer used and return a non-zero value + (:c:macro:`Py_CLEANUP_SUPPORTED`). + Embedded null bytes are not allowed in the result. + On failure, return ``0`` with an exception set. + + If *obj* is ``NULL``, the function releases a strong reference + stored in the variable referred by *result* and returns ``1``. .. versionadded:: 3.1 @@ -803,16 +812,26 @@ conversion function: Accepts a :term:`path-like object`. To decode file names to :class:`str` during argument parsing, the ``"O&"`` -converter should be used, passing :c:func:`PyUnicode_FSDecoder` as the +converter should be used, passing :c:func:`!PyUnicode_FSDecoder` as the conversion function: .. c:function:: int PyUnicode_FSDecoder(PyObject* obj, void* result) - ParseTuple converter: decode :class:`bytes` objects -- obtained either + :ref:`PyArg_Parse\* converter `: decode :class:`bytes` objects -- obtained either directly or indirectly through the :class:`os.PathLike` interface -- to :class:`str` using :c:func:`PyUnicode_DecodeFSDefaultAndSize`; :class:`str` - objects are output as-is. *result* must be a :c:expr:`PyUnicodeObject*` which - must be released when it is no longer used. + objects are output as-is. + *result* must be an address of a C variable of type :c:expr:`PyObject*` + (or :c:expr:`PyUnicodeObject*`). + On success, set the variable to a new :term:`strong reference` to + a :ref:`Unicode object ` which must be released + when it is no longer used and return a non-zero value + (:c:macro:`Py_CLEANUP_SUPPORTED`). + Embedded null characters are not allowed in the result. + On failure, return ``0`` with an exception set. + + If *obj* is ``NULL``, release the strong reference + to the object referred to by *result* and return ``1``. .. versionadded:: 3.2 diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat index 6f9d27297e8f65..c15f82603aa944 100644 --- a/Doc/data/stable_abi.dat +++ b/Doc/data/stable_abi.dat @@ -883,6 +883,8 @@ func,Py_Main,3.2,, func,Py_MakePendingCalls,3.2,, func,Py_NewInterpreter,3.2,, func,Py_NewRef,3.10,, +func,Py_PACK_FULL_VERSION,3.14,, +func,Py_PACK_VERSION,3.14,, func,Py_REFCNT,3.14,, func,Py_ReprEnter,3.2,, func,Py_ReprLeave,3.2,, diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index ccb362d8c31ddf..072ab206f25e4f 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -246,6 +246,9 @@ Scheduling callbacks another thread, this function *must* be used, since :meth:`call_soon` is not thread-safe. + This function is safe to be called from a reentrant context or signal handler, + however, it is not safe or fruitful to use the returned handle in such contexts. + Raises :exc:`RuntimeError` if called on a loop that's been closed. This can happen on a secondary thread when the main application is shutting down. @@ -967,6 +970,9 @@ Watching file descriptors invoke *callback* with the specified arguments once *fd* is available for reading. + Any preexisting callback registered for *fd* is cancelled and replaced by + *callback*. + .. method:: loop.remove_reader(fd) Stop monitoring the *fd* file descriptor for read availability. Returns @@ -978,6 +984,9 @@ Watching file descriptors invoke *callback* with the specified arguments once *fd* is available for writing. + Any preexisting callback registered for *fd* is cancelled and replaced by + *callback*. + Use :func:`functools.partial` :ref:`to pass keyword arguments ` to *callback*. diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index 61991bf2f4ed1d..066edd424d150e 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -115,11 +115,11 @@ Queue .. method:: task_done() - Indicate that a formerly enqueued task is complete. + Indicate that a formerly enqueued work item is complete. Used by queue consumers. For each :meth:`~Queue.get` used to - fetch a task, a subsequent call to :meth:`task_done` tells the - queue that the processing on the task is complete. + fetch a work item, a subsequent call to :meth:`task_done` tells the + queue that the processing on the work item is complete. If a :meth:`join` is currently blocking, it will resume when all items have been processed (meaning that a :meth:`task_done` diff --git a/Doc/library/calendar.rst b/Doc/library/calendar.rst index ace8529d6e7e0c..f25930eb0d873e 100644 --- a/Doc/library/calendar.rst +++ b/Doc/library/calendar.rst @@ -38,13 +38,33 @@ interpreted as prescribed by the ISO 8601 standard. Year 0 is 1 BC, year -1 is itself. This is the job of subclasses. - :class:`Calendar` instances have the following methods: + :class:`Calendar` instances have the following methods and attributes: + + .. attribute:: firstweekday + + The first weekday as an integer (0--6). + + This property can also be set and read using + :meth:`~Calendar.setfirstweekday` and + :meth:`~Calendar.getfirstweekday` respectively. + + .. method:: getfirstweekday() + + Return an :class:`int` for the current first weekday (0--6). + + Identical to reading the :attr:`~Calendar.firstweekday` property. + + .. method:: setfirstweekday(firstweekday) + + Set the first weekday to *firstweekday*, passed as an :class:`int` (0--6) + + Identical to setting the :attr:`~Calendar.firstweekday` property. .. method:: iterweekdays() Return an iterator for the week day numbers that will be used for one week. The first value from the iterator will be the same as the value of - the :attr:`firstweekday` property. + the :attr:`~Calendar.firstweekday` property. .. method:: itermonthdates(year, month) diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index 398cb92bac809a..f25bf94417c198 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -1812,6 +1812,8 @@ different ways, depending on the type and number of the parameters in the call: the COM interface as first argument, in addition to those parameters that are specified in the :attr:`!argtypes` tuple. + .. availability:: Windows + The optional *paramflags* parameter creates foreign function wrappers with much more functionality than the features described above. diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst index 314767d0802a08..6b997ee784f6e4 100644 --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -267,7 +267,7 @@ added matters. To illustrate:: Handle a *defect* found on *obj*. When the email package calls this method, *defect* will always be a subclass of - :class:`~email.errors.Defect`. + :class:`~email.errors.MessageDefect`. The default implementation checks the :attr:`raise_on_defect` flag. If it is ``True``, *defect* is raised as an exception. If it is ``False`` @@ -277,7 +277,7 @@ added matters. To illustrate:: .. method:: register_defect(obj, defect) Register a *defect* on *obj*. In the email package, *defect* will always - be a subclass of :class:`~email.errors.Defect`. + be a subclass of :class:`~email.errors.MessageDefect`. The default implementation calls the ``append`` method of the ``defects`` attribute of *obj*. When the email package calls :attr:`handle_defect`, diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst index 69d9d81c848124..8ad5f48c9e5286 100644 --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -368,8 +368,8 @@ The :mod:`functools` module defines the following functions: If :data:`Placeholder` sentinels are present in *args*, they will be filled first when :func:`!partial` is called. This makes it possible to pre-fill any positional - argument with a call to :func:`!partial`; without :data:`!Placeholder`, only the - first positional argument can be pre-filled. + argument with a call to :func:`!partial`; without :data:`!Placeholder`, + only the chosen number of leading positional arguments can be pre-filled. If any :data:`!Placeholder` sentinels are present, all must be filled at call time: diff --git a/Doc/library/importlib.resources.abc.rst b/Doc/library/importlib.resources.abc.rst index 54995ddbfbca12..4085bdf6598d98 100644 --- a/Doc/library/importlib.resources.abc.rst +++ b/Doc/library/importlib.resources.abc.rst @@ -43,7 +43,7 @@ :const:`None`. An object compatible with this ABC should only be returned when the specified module is a package. - .. deprecated-removed:: 3.12 3.14 + .. deprecated:: 3.12 Use :class:`importlib.resources.abc.TraversableResources` instead. .. abstractmethod:: open_resource(resource) diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 169291f74f44a5..4e7046d6d8f6ac 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -258,36 +258,86 @@ Basic Usage the original one. That is, ``loads(dumps(x)) != x`` if x has non-string keys. -.. function:: load(fp, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) +.. function:: load(fp, *, cls=None, object_hook=None, parse_float=None, \ + parse_int=None, parse_constant=None, \ + object_pairs_hook=None, **kw) - Deserialize *fp* (a ``.read()``-supporting :term:`text file` or - :term:`binary file` containing a JSON document) to a Python object using - this :ref:`conversion table `. + Deserialize *fp* to a Python object + using the :ref:`JSON-to-Python conversion table `. - *object_hook* is an optional function that will be called with the result of - any object literal decoded (a :class:`dict`). The return value of - *object_hook* will be used instead of the :class:`dict`. This feature can - be used to implement custom decoders (e.g. `JSON-RPC - `_ class hinting). + :param fp: + A ``.read()``-supporting :term:`text file` or :term:`binary file` + containing the JSON document to be deserialized. + :type fp: :term:`file-like object` - *object_pairs_hook* is an optional function that will be called with the - result of any object literal decoded with an ordered list of pairs. The - return value of *object_pairs_hook* will be used instead of the - :class:`dict`. This feature can be used to implement custom decoders. If - *object_hook* is also defined, the *object_pairs_hook* takes priority. + :param cls: + If set, a custom JSON decoder. + Additional keyword arguments to :func:`!load` + will be passed to the constructor of *cls*. + If ``None`` (the default), :class:`!JSONDecoder` is used. + :type cls: a :class:`JSONDecoder` subclass + + :param object_hook: + If set, a function that is called with the result of + any object literal decoded (a :class:`dict`). + The return value of this function will be used + instead of the :class:`dict`. + This feature can be used to implement custom decoders, + for example `JSON-RPC `_ class hinting. + Default ``None``. + :type object_hook: :term:`callable` | None + + :param object_pairs_hook: + If set, a function that is called with the result of + any object literal decoded with an ordered list of pairs. + The return value of this function will be used + instead of the :class:`dict`. + This feature can be used to implement custom decoders. + If *object_hook* is also set, *object_pairs_hook* takes priority. + Default ``None``. + :type object_pairs_hook: :term:`callable` | None + + :param parse_float: + If set, a function that is called with + the string of every JSON float to be decoded. + If ``None`` (the default), it is equivalent to ``float(num_str)``. + This can be used to parse JSON floats into custom datatypes, + for example :class:`decimal.Decimal`. + :type parse_float: :term:`callable` | None + + :param parse_int: + If set, a function that is called with + the string of every JSON int to be decoded. + If ``None`` (the default), it is equivalent to ``int(num_str)``. + This can be used to parse JSON integers into custom datatypes, + for example :class:`float`. + :type parse_int: :term:`callable` | None + + :param parse_constant: + If set, a function that is called with one of the following strings: + ``'-Infinity'``, ``'Infinity'``, or ``'NaN'``. + This can be used to raise an exception + if invalid JSON numbers are encountered. + Default ``None``. + :type parse_constant: :term:`callable` | None + + :raises JSONDecodeError: + When the data being deserialized is not a valid JSON document. + + :raises UnicodeDecodeError: + When the data being deserialized does not contain + UTF-8, UTF-16 or UTF-32 encoded data. .. versionchanged:: 3.1 - Added support for *object_pairs_hook*. - *parse_float* is an optional function that will be called with the string of - every JSON float to be decoded. By default, this is equivalent to - ``float(num_str)``. This can be used to use another datatype or parser for - JSON floats (e.g. :class:`decimal.Decimal`). + * Added the optional *object_pairs_hook* parameter. + * *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. - *parse_int* is an optional function that will be called with the string of - every JSON int to be decoded. By default, this is equivalent to - ``int(num_str)``. This can be used to use another datatype or parser for - JSON integers (e.g. :class:`float`). + .. versionchanged:: 3.6 + + * All optional parameters are now :ref:`keyword-only `. + * *fp* can now be a :term:`binary file`. + The input encoding should be UTF-8, UTF-16 or UTF-32. .. versionchanged:: 3.11 The default *parse_int* of :func:`int` now limits the maximum length of @@ -295,38 +345,13 @@ Basic Usage conversion length limitation ` to help avoid denial of service attacks. - *parse_constant* is an optional function that will be called with one of the - following strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``. This can be - used to raise an exception if invalid JSON numbers are encountered. - - .. versionchanged:: 3.1 - *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. - - To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` - kwarg; otherwise :class:`JSONDecoder` is used. Additional keyword arguments - will be passed to the constructor of the class. - - If the data being deserialized is not a valid JSON document, a - :exc:`JSONDecodeError` will be raised. - - .. versionchanged:: 3.6 - All optional parameters are now :ref:`keyword-only `. - - .. versionchanged:: 3.6 - *fp* can now be a :term:`binary file`. The input encoding should be - UTF-8, UTF-16 or UTF-32. - .. function:: loads(s, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) - Deserialize *s* (a :class:`str`, :class:`bytes` or :class:`bytearray` + Identical to :func:`load`, but instead of a file-like object, + deserialize *s* (a :class:`str`, :class:`bytes` or :class:`bytearray` instance containing a JSON document) to a Python object using this :ref:`conversion table `. - The other arguments have the same meaning as in :func:`load`. - - If the data being deserialized is not a valid JSON document, a - :exc:`JSONDecodeError` will be raised. - .. versionchanged:: 3.6 *s* can now be of type :class:`bytes` or :class:`bytearray`. The input encoding should be UTF-8, UTF-16 or UTF-32. diff --git a/Doc/tools/.nitignore b/Doc/tools/.nitignore index 7d50aec56a9bf7..6940c95ab2c9a1 100644 --- a/Doc/tools/.nitignore +++ b/Doc/tools/.nitignore @@ -23,7 +23,6 @@ Doc/library/email.charset.rst Doc/library/email.compat32-message.rst Doc/library/email.errors.rst Doc/library/email.parser.rst -Doc/library/email.policy.rst Doc/library/exceptions.rst Doc/library/functools.rst Doc/library/http.cookiejar.rst diff --git a/Doc/whatsnew/3.14.rst b/Doc/whatsnew/3.14.rst index f365db37217e95..72abfebd46f2b9 100644 --- a/Doc/whatsnew/3.14.rst +++ b/Doc/whatsnew/3.14.rst @@ -1237,6 +1237,16 @@ New features :monitoring-event:`BRANCH_LEFT` and :monitoring-event:`BRANCH_RIGHT` events, respectively. +* Add :c:func:`Py_fopen` function to open a file. Similar to the + :c:func:`!fopen` function, but the *path* parameter is a Python object and an + exception is set on error. Add also :c:func:`Py_fclose` function to close a + file. + (Contributed by Victor Stinner in :gh:`127350`.) + +* Add macros :c:func:`Py_PACK_VERSION` and :c:func:`Py_PACK_FULL_VERSION` for + bit-packing Python version numbers. + (Contributed by Petr Viktorin in :gh:`128629`.) + Porting to Python 3.14 ---------------------- diff --git a/Include/cpython/fileutils.h b/Include/cpython/fileutils.h index b386ad107bde1f..702f89aca324c5 100644 --- a/Include/cpython/fileutils.h +++ b/Include/cpython/fileutils.h @@ -2,7 +2,13 @@ # error "this header file must not be included directly" #endif -// Used by _testcapi which must not use the internal C API -PyAPI_FUNC(FILE*) _Py_fopen_obj( +PyAPI_FUNC(FILE*) Py_fopen( PyObject *path, const char *mode); + +// Deprecated alias to Py_fopen() kept for backward compatibility +Py_DEPRECATED(3.14) PyAPI_FUNC(FILE*) _Py_fopen_obj( + PyObject *path, + const char *mode); + +PyAPI_FUNC(int) Py_fclose(FILE *file); diff --git a/Include/internal/pycore_instruments.h b/Include/internal/pycore_instruments.h index 4e5b374968ea98..92d8f056f402fc 100644 --- a/Include/internal/pycore_instruments.h +++ b/Include/internal/pycore_instruments.h @@ -48,8 +48,8 @@ _Py_call_instrumentation_instruction( _Py_CODEUNIT * _Py_call_instrumentation_jump( - PyThreadState *tstate, int event, - _PyInterpreterFrame *frame, _Py_CODEUNIT *instr, _Py_CODEUNIT *target); + _Py_CODEUNIT *instr, PyThreadState *tstate, int event, + _PyInterpreterFrame *frame, _Py_CODEUNIT *src, _Py_CODEUNIT *dest); extern int _Py_call_instrumentation_arg(PyThreadState *tstate, int event, diff --git a/Include/internal/pycore_magic_number.h b/Include/internal/pycore_magic_number.h index 4c3b9c4c71da1b..f9f71d7453331e 100644 --- a/Include/internal/pycore_magic_number.h +++ b/Include/internal/pycore_magic_number.h @@ -262,8 +262,9 @@ Known values: Python 3.14a1 3607 (Add pseudo instructions JUMP_IF_TRUE/FALSE) Python 3.14a1 3608 (Add support for slices) Python 3.14a2 3609 (Add LOAD_SMALL_INT and LOAD_CONST_IMMORTAL instructions, remove RETURN_CONST) - (3610 accidentally omitted) + Python 3.14a4 3610 (Add VALUE_WITH_FAKE_GLOBALS format to annotationlib) Python 3.14a4 3611 (Add NOT_TAKEN instruction) + Python 3.14a4 3612 (Add POP_ITER and INSTRUMENTED_POP_ITER) Python 3.15 will start with 3650 @@ -276,7 +277,7 @@ PC/launcher.c must also be updated. */ -#define PYC_MAGIC_NUMBER 3611 +#define PYC_MAGIC_NUMBER 3612 /* This is equivalent to converting PYC_MAGIC_NUMBER to 2 bytes (little-endian) and then appending b'\r\n'. */ #define PYC_MAGIC_NUMBER_TOKEN \ diff --git a/Include/internal/pycore_object.h b/Include/internal/pycore_object.h index d7d68f938a9f0a..e26cb7673f939c 100644 --- a/Include/internal/pycore_object.h +++ b/Include/internal/pycore_object.h @@ -120,7 +120,7 @@ PyAPI_FUNC(void) _Py_NO_RETURN _Py_FatalRefcountErrorFunc( PyAPI_DATA(Py_ssize_t) _Py_RefTotal; extern void _Py_AddRefTotal(PyThreadState *, Py_ssize_t); -extern void _Py_IncRefTotal(PyThreadState *); +extern PyAPI_FUNC(void) _Py_IncRefTotal(PyThreadState *); extern void _Py_DecRefTotal(PyThreadState *); # define _Py_DEC_REFTOTAL(interp) \ diff --git a/Include/internal/pycore_opcode_metadata.h b/Include/internal/pycore_opcode_metadata.h index 5fb236836dccd9..90d5e277d8d6ce 100644 --- a/Include/internal/pycore_opcode_metadata.h +++ b/Include/internal/pycore_opcode_metadata.h @@ -245,6 +245,8 @@ int _PyOpcode_num_popped(int opcode, int oparg) { return 0; case INSTRUMENTED_NOT_TAKEN: return 0; + case INSTRUMENTED_POP_ITER: + return 1; case INSTRUMENTED_POP_JUMP_IF_FALSE: return 0; case INSTRUMENTED_POP_JUMP_IF_NONE: @@ -375,6 +377,8 @@ int _PyOpcode_num_popped(int opcode, int oparg) { return 0; case POP_EXCEPT: return 1; + case POP_ITER: + return 1; case POP_JUMP_IF_FALSE: return 1; case POP_JUMP_IF_NONE: @@ -708,6 +712,8 @@ int _PyOpcode_num_pushed(int opcode, int oparg) { return 0; case INSTRUMENTED_NOT_TAKEN: return 0; + case INSTRUMENTED_POP_ITER: + return 0; case INSTRUMENTED_POP_JUMP_IF_FALSE: return 0; case INSTRUMENTED_POP_JUMP_IF_NONE: @@ -838,6 +844,8 @@ int _PyOpcode_num_pushed(int opcode, int oparg) { return 0; case POP_EXCEPT: return 0; + case POP_ITER: + return 0; case POP_JUMP_IF_FALSE: return 0; case POP_JUMP_IF_NONE: @@ -1399,6 +1407,10 @@ int _PyOpcode_max_stack_effect(int opcode, int oparg, int *effect) { *effect = 0; return 0; } + case INSTRUMENTED_POP_ITER: { + *effect = -1; + return 0; + } case INSTRUMENTED_POP_JUMP_IF_FALSE: { *effect = 0; return 0; @@ -1659,6 +1671,10 @@ int _PyOpcode_max_stack_effect(int opcode, int oparg, int *effect) { *effect = -1; return 0; } + case POP_ITER: { + *effect = -1; + return 0; + } case POP_JUMP_IF_FALSE: { *effect = -1; return 0; @@ -1921,6 +1937,7 @@ enum InstructionFormat { #define HAS_PASSTHROUGH_FLAG (4096) #define HAS_OPARG_AND_1_FLAG (8192) #define HAS_ERROR_NO_POP_FLAG (16384) +#define HAS_NO_SAVE_IP_FLAG (32768) #define OPCODE_HAS_ARG(OP) (_PyOpcode_opcode_metadata[OP].flags & (HAS_ARG_FLAG)) #define OPCODE_HAS_CONST(OP) (_PyOpcode_opcode_metadata[OP].flags & (HAS_CONST_FLAG)) #define OPCODE_HAS_NAME(OP) (_PyOpcode_opcode_metadata[OP].flags & (HAS_NAME_FLAG)) @@ -1936,6 +1953,7 @@ enum InstructionFormat { #define OPCODE_HAS_PASSTHROUGH(OP) (_PyOpcode_opcode_metadata[OP].flags & (HAS_PASSTHROUGH_FLAG)) #define OPCODE_HAS_OPARG_AND_1(OP) (_PyOpcode_opcode_metadata[OP].flags & (HAS_OPARG_AND_1_FLAG)) #define OPCODE_HAS_ERROR_NO_POP(OP) (_PyOpcode_opcode_metadata[OP].flags & (HAS_ERROR_NO_POP_FLAG)) +#define OPCODE_HAS_NO_SAVE_IP(OP) (_PyOpcode_opcode_metadata[OP].flags & (HAS_NO_SAVE_IP_FLAG)) #define OPARG_FULL 0 #define OPARG_CACHE_1 1 @@ -1948,8 +1966,8 @@ enum InstructionFormat { struct opcode_metadata { uint8_t valid_entry; - int8_t instr_format; - int16_t flags; + uint8_t instr_format; + uint16_t flags; }; extern const struct opcode_metadata _PyOpcode_opcode_metadata[266]; @@ -2028,7 +2046,7 @@ const struct opcode_metadata _PyOpcode_opcode_metadata[266] = { [DICT_MERGE] = { true, INSTR_FMT_IB, HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ESCAPES_FLAG }, [DICT_UPDATE] = { true, INSTR_FMT_IB, HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ESCAPES_FLAG }, [END_ASYNC_FOR] = { true, INSTR_FMT_IX, HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG | HAS_ESCAPES_FLAG }, - [END_FOR] = { true, INSTR_FMT_IX, HAS_PURE_FLAG }, + [END_FOR] = { true, INSTR_FMT_IX, HAS_NO_SAVE_IP_FLAG }, [END_SEND] = { true, INSTR_FMT_IX, HAS_PURE_FLAG }, [ENTER_EXECUTOR] = { true, INSTR_FMT_IB, HAS_ARG_FLAG }, [EXIT_INIT_CHECK] = { true, INSTR_FMT_IX, HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG | HAS_ESCAPES_FLAG }, @@ -2051,15 +2069,16 @@ const struct opcode_metadata _PyOpcode_opcode_metadata[266] = { [INSTRUMENTED_CALL] = { true, INSTR_FMT_IBC00, HAS_ARG_FLAG | HAS_EVAL_BREAK_FLAG | HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG | HAS_ESCAPES_FLAG }, [INSTRUMENTED_CALL_FUNCTION_EX] = { true, INSTR_FMT_IX, 0 }, [INSTRUMENTED_CALL_KW] = { true, INSTR_FMT_IBC00, HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ESCAPES_FLAG }, - [INSTRUMENTED_END_FOR] = { true, INSTR_FMT_IX, HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG | HAS_ESCAPES_FLAG }, + [INSTRUMENTED_END_FOR] = { true, INSTR_FMT_IX, HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG | HAS_ESCAPES_FLAG | HAS_NO_SAVE_IP_FLAG }, [INSTRUMENTED_END_SEND] = { true, INSTR_FMT_IX, HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG | HAS_ESCAPES_FLAG }, - [INSTRUMENTED_FOR_ITER] = { true, INSTR_FMT_IBC, HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG | HAS_ESCAPES_FLAG }, + [INSTRUMENTED_FOR_ITER] = { true, INSTR_FMT_IBC, HAS_ARG_FLAG | HAS_JUMP_FLAG | HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG | HAS_ESCAPES_FLAG }, [INSTRUMENTED_INSTRUCTION] = { true, INSTR_FMT_IX, HAS_ERROR_FLAG | HAS_ESCAPES_FLAG }, [INSTRUMENTED_JUMP_BACKWARD] = { true, INSTR_FMT_IBC, HAS_ARG_FLAG | HAS_EVAL_BREAK_FLAG | HAS_ERROR_FLAG | HAS_ESCAPES_FLAG }, [INSTRUMENTED_JUMP_FORWARD] = { true, INSTR_FMT_IB, HAS_ARG_FLAG }, [INSTRUMENTED_LINE] = { true, INSTR_FMT_IX, HAS_ESCAPES_FLAG }, [INSTRUMENTED_LOAD_SUPER_ATTR] = { true, INSTR_FMT_IXC, 0 }, [INSTRUMENTED_NOT_TAKEN] = { true, INSTR_FMT_IX, 0 }, + [INSTRUMENTED_POP_ITER] = { true, INSTR_FMT_IX, 0 }, [INSTRUMENTED_POP_JUMP_IF_FALSE] = { true, INSTR_FMT_IBC, HAS_ARG_FLAG }, [INSTRUMENTED_POP_JUMP_IF_NONE] = { true, INSTR_FMT_IBC, HAS_ARG_FLAG }, [INSTRUMENTED_POP_JUMP_IF_NOT_NONE] = { true, INSTR_FMT_IBC, HAS_ARG_FLAG }, @@ -2119,6 +2138,7 @@ const struct opcode_metadata _PyOpcode_opcode_metadata[266] = { [NOP] = { true, INSTR_FMT_IX, HAS_PURE_FLAG }, [NOT_TAKEN] = { true, INSTR_FMT_IX, HAS_PURE_FLAG }, [POP_EXCEPT] = { true, INSTR_FMT_IX, HAS_ESCAPES_FLAG }, + [POP_ITER] = { true, INSTR_FMT_IX, HAS_PURE_FLAG }, [POP_JUMP_IF_FALSE] = { true, INSTR_FMT_IBC, HAS_ARG_FLAG | HAS_JUMP_FLAG }, [POP_JUMP_IF_NONE] = { true, INSTR_FMT_IBC, HAS_ARG_FLAG | HAS_JUMP_FLAG }, [POP_JUMP_IF_NOT_NONE] = { true, INSTR_FMT_IBC, HAS_ARG_FLAG | HAS_JUMP_FLAG }, @@ -2261,7 +2281,7 @@ _PyOpcode_macro_expansion[256] = { [DELETE_SUBSCR] = { .nuops = 1, .uops = { { _DELETE_SUBSCR, 0, 0 } } }, [DICT_MERGE] = { .nuops = 1, .uops = { { _DICT_MERGE, 0, 0 } } }, [DICT_UPDATE] = { .nuops = 1, .uops = { { _DICT_UPDATE, 0, 0 } } }, - [END_FOR] = { .nuops = 1, .uops = { { _POP_TOP, 0, 0 } } }, + [END_FOR] = { .nuops = 1, .uops = { { _END_FOR, 0, 0 } } }, [END_SEND] = { .nuops = 1, .uops = { { _END_SEND, 0, 0 } } }, [EXIT_INIT_CHECK] = { .nuops = 1, .uops = { { _EXIT_INIT_CHECK, 0, 0 } } }, [FORMAT_SIMPLE] = { .nuops = 1, .uops = { { _FORMAT_SIMPLE, 0, 0 } } }, @@ -2324,6 +2344,7 @@ _PyOpcode_macro_expansion[256] = { [NOP] = { .nuops = 1, .uops = { { _NOP, 0, 0 } } }, [NOT_TAKEN] = { .nuops = 1, .uops = { { _NOP, 0, 0 } } }, [POP_EXCEPT] = { .nuops = 1, .uops = { { _POP_EXCEPT, 0, 0 } } }, + [POP_ITER] = { .nuops = 1, .uops = { { _POP_TOP, 0, 0 } } }, [POP_JUMP_IF_FALSE] = { .nuops = 1, .uops = { { _POP_JUMP_IF_FALSE, 9, 1 } } }, [POP_JUMP_IF_NONE] = { .nuops = 2, .uops = { { _IS_NONE, 0, 0 }, { _POP_JUMP_IF_TRUE, 9, 1 } } }, [POP_JUMP_IF_NOT_NONE] = { .nuops = 2, .uops = { { _IS_NONE, 0, 0 }, { _POP_JUMP_IF_FALSE, 9, 1 } } }, @@ -2482,6 +2503,7 @@ const char *_PyOpcode_OpName[266] = { [INSTRUMENTED_LINE] = "INSTRUMENTED_LINE", [INSTRUMENTED_LOAD_SUPER_ATTR] = "INSTRUMENTED_LOAD_SUPER_ATTR", [INSTRUMENTED_NOT_TAKEN] = "INSTRUMENTED_NOT_TAKEN", + [INSTRUMENTED_POP_ITER] = "INSTRUMENTED_POP_ITER", [INSTRUMENTED_POP_JUMP_IF_FALSE] = "INSTRUMENTED_POP_JUMP_IF_FALSE", [INSTRUMENTED_POP_JUMP_IF_NONE] = "INSTRUMENTED_POP_JUMP_IF_NONE", [INSTRUMENTED_POP_JUMP_IF_NOT_NONE] = "INSTRUMENTED_POP_JUMP_IF_NOT_NONE", @@ -2547,6 +2569,7 @@ const char *_PyOpcode_OpName[266] = { [NOT_TAKEN] = "NOT_TAKEN", [POP_BLOCK] = "POP_BLOCK", [POP_EXCEPT] = "POP_EXCEPT", + [POP_ITER] = "POP_ITER", [POP_JUMP_IF_FALSE] = "POP_JUMP_IF_FALSE", [POP_JUMP_IF_NONE] = "POP_JUMP_IF_NONE", [POP_JUMP_IF_NOT_NONE] = "POP_JUMP_IF_NOT_NONE", @@ -2740,6 +2763,7 @@ const uint8_t _PyOpcode_Deopt[256] = { [INSTRUMENTED_LINE] = INSTRUMENTED_LINE, [INSTRUMENTED_LOAD_SUPER_ATTR] = INSTRUMENTED_LOAD_SUPER_ATTR, [INSTRUMENTED_NOT_TAKEN] = INSTRUMENTED_NOT_TAKEN, + [INSTRUMENTED_POP_ITER] = INSTRUMENTED_POP_ITER, [INSTRUMENTED_POP_JUMP_IF_FALSE] = INSTRUMENTED_POP_JUMP_IF_FALSE, [INSTRUMENTED_POP_JUMP_IF_NONE] = INSTRUMENTED_POP_JUMP_IF_NONE, [INSTRUMENTED_POP_JUMP_IF_NOT_NONE] = INSTRUMENTED_POP_JUMP_IF_NOT_NONE, @@ -2799,6 +2823,7 @@ const uint8_t _PyOpcode_Deopt[256] = { [NOP] = NOP, [NOT_TAKEN] = NOT_TAKEN, [POP_EXCEPT] = POP_EXCEPT, + [POP_ITER] = POP_ITER, [POP_JUMP_IF_FALSE] = POP_JUMP_IF_FALSE, [POP_JUMP_IF_NONE] = POP_JUMP_IF_NONE, [POP_JUMP_IF_NOT_NONE] = POP_JUMP_IF_NOT_NONE, @@ -2856,7 +2881,6 @@ const uint8_t _PyOpcode_Deopt[256] = { #endif // NEED_OPCODE_METADATA #define EXTRA_CASES \ - case 117: \ case 118: \ case 119: \ case 120: \ @@ -2895,7 +2919,6 @@ const uint8_t _PyOpcode_Deopt[256] = { case 232: \ case 233: \ case 234: \ - case 235: \ ; struct pseudo_targets { uint8_t as_sequence; diff --git a/Include/internal/pycore_uop_ids.h b/Include/internal/pycore_uop_ids.h index 92515b4230ccb4..21690a28839565 100644 --- a/Include/internal/pycore_uop_ids.h +++ b/Include/internal/pycore_uop_ids.h @@ -101,6 +101,7 @@ extern "C" { #define _DO_CALL_FUNCTION_EX 358 #define _DO_CALL_KW 359 #define _DYNAMIC_EXIT 360 +#define _END_FOR END_FOR #define _END_SEND END_SEND #define _ERROR_POP_N 361 #define _EXIT_INIT_CHECK EXIT_INIT_CHECK diff --git a/Include/internal/pycore_uop_metadata.h b/Include/internal/pycore_uop_metadata.h index 73fc29eb78a7a4..83e578cdd76fbd 100644 --- a/Include/internal/pycore_uop_metadata.h +++ b/Include/internal/pycore_uop_metadata.h @@ -55,6 +55,7 @@ const uint16_t _PyUop_Flags[MAX_UOP_ID+1] = { [_STORE_FAST_STORE_FAST] = HAS_ARG_FLAG | HAS_LOCAL_FLAG, [_POP_TOP] = HAS_PURE_FLAG, [_PUSH_NULL] = HAS_PURE_FLAG, + [_END_FOR] = HAS_NO_SAVE_IP_FLAG, [_END_SEND] = HAS_PURE_FLAG, [_UNARY_NEGATIVE] = HAS_ERROR_FLAG | HAS_ESCAPES_FLAG, [_UNARY_NOT] = HAS_PURE_FLAG, @@ -391,6 +392,7 @@ const char *const _PyOpcode_uop_name[MAX_UOP_ID+1] = { [_DICT_MERGE] = "_DICT_MERGE", [_DICT_UPDATE] = "_DICT_UPDATE", [_DYNAMIC_EXIT] = "_DYNAMIC_EXIT", + [_END_FOR] = "_END_FOR", [_END_SEND] = "_END_SEND", [_ERROR_POP_N] = "_ERROR_POP_N", [_EXIT_INIT_CHECK] = "_EXIT_INIT_CHECK", @@ -655,6 +657,8 @@ int _PyUop_num_popped(int opcode, int oparg) return 1; case _PUSH_NULL: return 0; + case _END_FOR: + return 1; case _END_SEND: return 2; case _UNARY_NEGATIVE: diff --git a/Include/opcode_ids.h b/Include/opcode_ids.h index 3cd189b93dd9d6..09e261fadd5544 100644 --- a/Include/opcode_ids.h +++ b/Include/opcode_ids.h @@ -40,93 +40,94 @@ extern "C" { #define NOP 27 #define NOT_TAKEN 28 #define POP_EXCEPT 29 -#define POP_TOP 30 -#define PUSH_EXC_INFO 31 -#define PUSH_NULL 32 -#define RETURN_GENERATOR 33 -#define RETURN_VALUE 34 -#define SETUP_ANNOTATIONS 35 -#define STORE_SLICE 36 -#define STORE_SUBSCR 37 -#define TO_BOOL 38 -#define UNARY_INVERT 39 -#define UNARY_NEGATIVE 40 -#define UNARY_NOT 41 -#define WITH_EXCEPT_START 42 -#define BINARY_OP 43 -#define BUILD_LIST 44 -#define BUILD_MAP 45 -#define BUILD_SET 46 -#define BUILD_SLICE 47 -#define BUILD_STRING 48 -#define BUILD_TUPLE 49 -#define CALL 50 -#define CALL_FUNCTION_EX 51 -#define CALL_INTRINSIC_1 52 -#define CALL_INTRINSIC_2 53 -#define CALL_KW 54 -#define COMPARE_OP 55 -#define CONTAINS_OP 56 -#define CONVERT_VALUE 57 -#define COPY 58 -#define COPY_FREE_VARS 59 -#define DELETE_ATTR 60 -#define DELETE_DEREF 61 -#define DELETE_FAST 62 -#define DELETE_GLOBAL 63 -#define DELETE_NAME 64 -#define DICT_MERGE 65 -#define DICT_UPDATE 66 -#define EXTENDED_ARG 67 -#define FOR_ITER 68 -#define GET_AWAITABLE 69 -#define IMPORT_FROM 70 -#define IMPORT_NAME 71 -#define IS_OP 72 -#define JUMP_BACKWARD 73 -#define JUMP_BACKWARD_NO_INTERRUPT 74 -#define JUMP_FORWARD 75 -#define LIST_APPEND 76 -#define LIST_EXTEND 77 -#define LOAD_ATTR 78 -#define LOAD_COMMON_CONSTANT 79 -#define LOAD_CONST 80 -#define LOAD_DEREF 81 -#define LOAD_FAST 82 -#define LOAD_FAST_AND_CLEAR 83 -#define LOAD_FAST_CHECK 84 -#define LOAD_FAST_LOAD_FAST 85 -#define LOAD_FROM_DICT_OR_DEREF 86 -#define LOAD_FROM_DICT_OR_GLOBALS 87 -#define LOAD_GLOBAL 88 -#define LOAD_NAME 89 -#define LOAD_SMALL_INT 90 -#define LOAD_SPECIAL 91 -#define LOAD_SUPER_ATTR 92 -#define MAKE_CELL 93 -#define MAP_ADD 94 -#define MATCH_CLASS 95 -#define POP_JUMP_IF_FALSE 96 -#define POP_JUMP_IF_NONE 97 -#define POP_JUMP_IF_NOT_NONE 98 -#define POP_JUMP_IF_TRUE 99 -#define RAISE_VARARGS 100 -#define RERAISE 101 -#define SEND 102 -#define SET_ADD 103 -#define SET_FUNCTION_ATTRIBUTE 104 -#define SET_UPDATE 105 -#define STORE_ATTR 106 -#define STORE_DEREF 107 -#define STORE_FAST 108 -#define STORE_FAST_LOAD_FAST 109 -#define STORE_FAST_STORE_FAST 110 -#define STORE_GLOBAL 111 -#define STORE_NAME 112 -#define SWAP 113 -#define UNPACK_EX 114 -#define UNPACK_SEQUENCE 115 -#define YIELD_VALUE 116 +#define POP_ITER 30 +#define POP_TOP 31 +#define PUSH_EXC_INFO 32 +#define PUSH_NULL 33 +#define RETURN_GENERATOR 34 +#define RETURN_VALUE 35 +#define SETUP_ANNOTATIONS 36 +#define STORE_SLICE 37 +#define STORE_SUBSCR 38 +#define TO_BOOL 39 +#define UNARY_INVERT 40 +#define UNARY_NEGATIVE 41 +#define UNARY_NOT 42 +#define WITH_EXCEPT_START 43 +#define BINARY_OP 44 +#define BUILD_LIST 45 +#define BUILD_MAP 46 +#define BUILD_SET 47 +#define BUILD_SLICE 48 +#define BUILD_STRING 49 +#define BUILD_TUPLE 50 +#define CALL 51 +#define CALL_FUNCTION_EX 52 +#define CALL_INTRINSIC_1 53 +#define CALL_INTRINSIC_2 54 +#define CALL_KW 55 +#define COMPARE_OP 56 +#define CONTAINS_OP 57 +#define CONVERT_VALUE 58 +#define COPY 59 +#define COPY_FREE_VARS 60 +#define DELETE_ATTR 61 +#define DELETE_DEREF 62 +#define DELETE_FAST 63 +#define DELETE_GLOBAL 64 +#define DELETE_NAME 65 +#define DICT_MERGE 66 +#define DICT_UPDATE 67 +#define EXTENDED_ARG 68 +#define FOR_ITER 69 +#define GET_AWAITABLE 70 +#define IMPORT_FROM 71 +#define IMPORT_NAME 72 +#define IS_OP 73 +#define JUMP_BACKWARD 74 +#define JUMP_BACKWARD_NO_INTERRUPT 75 +#define JUMP_FORWARD 76 +#define LIST_APPEND 77 +#define LIST_EXTEND 78 +#define LOAD_ATTR 79 +#define LOAD_COMMON_CONSTANT 80 +#define LOAD_CONST 81 +#define LOAD_DEREF 82 +#define LOAD_FAST 83 +#define LOAD_FAST_AND_CLEAR 84 +#define LOAD_FAST_CHECK 85 +#define LOAD_FAST_LOAD_FAST 86 +#define LOAD_FROM_DICT_OR_DEREF 87 +#define LOAD_FROM_DICT_OR_GLOBALS 88 +#define LOAD_GLOBAL 89 +#define LOAD_NAME 90 +#define LOAD_SMALL_INT 91 +#define LOAD_SPECIAL 92 +#define LOAD_SUPER_ATTR 93 +#define MAKE_CELL 94 +#define MAP_ADD 95 +#define MATCH_CLASS 96 +#define POP_JUMP_IF_FALSE 97 +#define POP_JUMP_IF_NONE 98 +#define POP_JUMP_IF_NOT_NONE 99 +#define POP_JUMP_IF_TRUE 100 +#define RAISE_VARARGS 101 +#define RERAISE 102 +#define SEND 103 +#define SET_ADD 104 +#define SET_FUNCTION_ATTRIBUTE 105 +#define SET_UPDATE 106 +#define STORE_ATTR 107 +#define STORE_DEREF 108 +#define STORE_FAST 109 +#define STORE_FAST_LOAD_FAST 110 +#define STORE_FAST_STORE_FAST 111 +#define STORE_GLOBAL 112 +#define STORE_NAME 113 +#define SWAP 114 +#define UNPACK_EX 115 +#define UNPACK_SEQUENCE 116 +#define YIELD_VALUE 117 #define RESUME 149 #define BINARY_OP_ADD_FLOAT 150 #define BINARY_OP_ADD_INT 151 @@ -206,7 +207,8 @@ extern "C" { #define UNPACK_SEQUENCE_LIST 225 #define UNPACK_SEQUENCE_TUPLE 226 #define UNPACK_SEQUENCE_TWO_TUPLE 227 -#define INSTRUMENTED_END_FOR 236 +#define INSTRUMENTED_END_FOR 235 +#define INSTRUMENTED_POP_ITER 236 #define INSTRUMENTED_END_SEND 237 #define INSTRUMENTED_LOAD_SUPER_ATTR 238 #define INSTRUMENTED_FOR_ITER 239 @@ -237,9 +239,9 @@ extern "C" { #define SETUP_WITH 264 #define STORE_FAST_MAYBE_NULL 265 -#define HAVE_ARGUMENT 42 +#define HAVE_ARGUMENT 43 #define MIN_SPECIALIZED_OPCODE 150 -#define MIN_INSTRUMENTED_OPCODE 236 +#define MIN_INSTRUMENTED_OPCODE 235 #ifdef __cplusplus } diff --git a/Include/patchlevel.h b/Include/patchlevel.h index 6d4f719fcde5a8..eca2ca08a4337c 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -1,4 +1,5 @@ - +#ifndef _Py_PATCHLEVEL_H +#define _Py_PATCHLEVEL_H /* Python version identification scheme. When the major or minor version changes, the VERSION variable in @@ -26,10 +27,23 @@ #define PY_VERSION "3.14.0a3+" /*--end constants--*/ + +#define _Py_PACK_FULL_VERSION(X, Y, Z, LEVEL, SERIAL) ( \ + (((X) & 0xff) << 24) | \ + (((Y) & 0xff) << 16) | \ + (((Z) & 0xff) << 8) | \ + (((LEVEL) & 0xf) << 4) | \ + (((SERIAL) & 0xf) << 0)) + /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. Use this for numeric comparisons, e.g. #if PY_VERSION_HEX >= ... */ -#define PY_VERSION_HEX ((PY_MAJOR_VERSION << 24) | \ - (PY_MINOR_VERSION << 16) | \ - (PY_MICRO_VERSION << 8) | \ - (PY_RELEASE_LEVEL << 4) | \ - (PY_RELEASE_SERIAL << 0)) +#define PY_VERSION_HEX _Py_PACK_FULL_VERSION( \ + PY_MAJOR_VERSION, \ + PY_MINOR_VERSION, \ + PY_MICRO_VERSION, \ + PY_RELEASE_LEVEL, \ + PY_RELEASE_SERIAL) + +// Public Py_PACK_VERSION is declared in pymacro.h; it needs . + +#endif //_Py_PATCHLEVEL_H diff --git a/Include/pymacro.h b/Include/pymacro.h index e0378f9d27a048..a82f347866e8d0 100644 --- a/Include/pymacro.h +++ b/Include/pymacro.h @@ -190,4 +190,13 @@ // "comparison of unsigned expression in '< 0' is always false". #define _Py_IS_TYPE_SIGNED(type) ((type)(-1) <= 0) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030E0000 // 3.14 +// Version helpers. These are primarily macros, but have exported equivalents. +PyAPI_FUNC(uint32_t) Py_PACK_FULL_VERSION(int x, int y, int z, int level, int serial); +PyAPI_FUNC(uint32_t) Py_PACK_VERSION(int x, int y); +#define Py_PACK_FULL_VERSION _Py_PACK_FULL_VERSION +#define Py_PACK_VERSION(X, Y) Py_PACK_FULL_VERSION(X, Y, 0, 0, 0) +#endif // Py_LIMITED_API < 3.14 + + #endif /* Py_PYMACRO_H */ diff --git a/Lib/_opcode_metadata.py b/Lib/_opcode_metadata.py index dada2cb5fa033f..64ee56fd10556f 100644 --- a/Lib/_opcode_metadata.py +++ b/Lib/_opcode_metadata.py @@ -233,94 +233,96 @@ 'NOP': 27, 'NOT_TAKEN': 28, 'POP_EXCEPT': 29, - 'POP_TOP': 30, - 'PUSH_EXC_INFO': 31, - 'PUSH_NULL': 32, - 'RETURN_GENERATOR': 33, - 'RETURN_VALUE': 34, - 'SETUP_ANNOTATIONS': 35, - 'STORE_SLICE': 36, - 'STORE_SUBSCR': 37, - 'TO_BOOL': 38, - 'UNARY_INVERT': 39, - 'UNARY_NEGATIVE': 40, - 'UNARY_NOT': 41, - 'WITH_EXCEPT_START': 42, - 'BINARY_OP': 43, - 'BUILD_LIST': 44, - 'BUILD_MAP': 45, - 'BUILD_SET': 46, - 'BUILD_SLICE': 47, - 'BUILD_STRING': 48, - 'BUILD_TUPLE': 49, - 'CALL': 50, - 'CALL_FUNCTION_EX': 51, - 'CALL_INTRINSIC_1': 52, - 'CALL_INTRINSIC_2': 53, - 'CALL_KW': 54, - 'COMPARE_OP': 55, - 'CONTAINS_OP': 56, - 'CONVERT_VALUE': 57, - 'COPY': 58, - 'COPY_FREE_VARS': 59, - 'DELETE_ATTR': 60, - 'DELETE_DEREF': 61, - 'DELETE_FAST': 62, - 'DELETE_GLOBAL': 63, - 'DELETE_NAME': 64, - 'DICT_MERGE': 65, - 'DICT_UPDATE': 66, - 'EXTENDED_ARG': 67, - 'FOR_ITER': 68, - 'GET_AWAITABLE': 69, - 'IMPORT_FROM': 70, - 'IMPORT_NAME': 71, - 'IS_OP': 72, - 'JUMP_BACKWARD': 73, - 'JUMP_BACKWARD_NO_INTERRUPT': 74, - 'JUMP_FORWARD': 75, - 'LIST_APPEND': 76, - 'LIST_EXTEND': 77, - 'LOAD_ATTR': 78, - 'LOAD_COMMON_CONSTANT': 79, - 'LOAD_CONST': 80, - 'LOAD_DEREF': 81, - 'LOAD_FAST': 82, - 'LOAD_FAST_AND_CLEAR': 83, - 'LOAD_FAST_CHECK': 84, - 'LOAD_FAST_LOAD_FAST': 85, - 'LOAD_FROM_DICT_OR_DEREF': 86, - 'LOAD_FROM_DICT_OR_GLOBALS': 87, - 'LOAD_GLOBAL': 88, - 'LOAD_NAME': 89, - 'LOAD_SMALL_INT': 90, - 'LOAD_SPECIAL': 91, - 'LOAD_SUPER_ATTR': 92, - 'MAKE_CELL': 93, - 'MAP_ADD': 94, - 'MATCH_CLASS': 95, - 'POP_JUMP_IF_FALSE': 96, - 'POP_JUMP_IF_NONE': 97, - 'POP_JUMP_IF_NOT_NONE': 98, - 'POP_JUMP_IF_TRUE': 99, - 'RAISE_VARARGS': 100, - 'RERAISE': 101, - 'SEND': 102, - 'SET_ADD': 103, - 'SET_FUNCTION_ATTRIBUTE': 104, - 'SET_UPDATE': 105, - 'STORE_ATTR': 106, - 'STORE_DEREF': 107, - 'STORE_FAST': 108, - 'STORE_FAST_LOAD_FAST': 109, - 'STORE_FAST_STORE_FAST': 110, - 'STORE_GLOBAL': 111, - 'STORE_NAME': 112, - 'SWAP': 113, - 'UNPACK_EX': 114, - 'UNPACK_SEQUENCE': 115, - 'YIELD_VALUE': 116, - 'INSTRUMENTED_END_FOR': 236, + 'POP_ITER': 30, + 'POP_TOP': 31, + 'PUSH_EXC_INFO': 32, + 'PUSH_NULL': 33, + 'RETURN_GENERATOR': 34, + 'RETURN_VALUE': 35, + 'SETUP_ANNOTATIONS': 36, + 'STORE_SLICE': 37, + 'STORE_SUBSCR': 38, + 'TO_BOOL': 39, + 'UNARY_INVERT': 40, + 'UNARY_NEGATIVE': 41, + 'UNARY_NOT': 42, + 'WITH_EXCEPT_START': 43, + 'BINARY_OP': 44, + 'BUILD_LIST': 45, + 'BUILD_MAP': 46, + 'BUILD_SET': 47, + 'BUILD_SLICE': 48, + 'BUILD_STRING': 49, + 'BUILD_TUPLE': 50, + 'CALL': 51, + 'CALL_FUNCTION_EX': 52, + 'CALL_INTRINSIC_1': 53, + 'CALL_INTRINSIC_2': 54, + 'CALL_KW': 55, + 'COMPARE_OP': 56, + 'CONTAINS_OP': 57, + 'CONVERT_VALUE': 58, + 'COPY': 59, + 'COPY_FREE_VARS': 60, + 'DELETE_ATTR': 61, + 'DELETE_DEREF': 62, + 'DELETE_FAST': 63, + 'DELETE_GLOBAL': 64, + 'DELETE_NAME': 65, + 'DICT_MERGE': 66, + 'DICT_UPDATE': 67, + 'EXTENDED_ARG': 68, + 'FOR_ITER': 69, + 'GET_AWAITABLE': 70, + 'IMPORT_FROM': 71, + 'IMPORT_NAME': 72, + 'IS_OP': 73, + 'JUMP_BACKWARD': 74, + 'JUMP_BACKWARD_NO_INTERRUPT': 75, + 'JUMP_FORWARD': 76, + 'LIST_APPEND': 77, + 'LIST_EXTEND': 78, + 'LOAD_ATTR': 79, + 'LOAD_COMMON_CONSTANT': 80, + 'LOAD_CONST': 81, + 'LOAD_DEREF': 82, + 'LOAD_FAST': 83, + 'LOAD_FAST_AND_CLEAR': 84, + 'LOAD_FAST_CHECK': 85, + 'LOAD_FAST_LOAD_FAST': 86, + 'LOAD_FROM_DICT_OR_DEREF': 87, + 'LOAD_FROM_DICT_OR_GLOBALS': 88, + 'LOAD_GLOBAL': 89, + 'LOAD_NAME': 90, + 'LOAD_SMALL_INT': 91, + 'LOAD_SPECIAL': 92, + 'LOAD_SUPER_ATTR': 93, + 'MAKE_CELL': 94, + 'MAP_ADD': 95, + 'MATCH_CLASS': 96, + 'POP_JUMP_IF_FALSE': 97, + 'POP_JUMP_IF_NONE': 98, + 'POP_JUMP_IF_NOT_NONE': 99, + 'POP_JUMP_IF_TRUE': 100, + 'RAISE_VARARGS': 101, + 'RERAISE': 102, + 'SEND': 103, + 'SET_ADD': 104, + 'SET_FUNCTION_ATTRIBUTE': 105, + 'SET_UPDATE': 106, + 'STORE_ATTR': 107, + 'STORE_DEREF': 108, + 'STORE_FAST': 109, + 'STORE_FAST_LOAD_FAST': 110, + 'STORE_FAST_STORE_FAST': 111, + 'STORE_GLOBAL': 112, + 'STORE_NAME': 113, + 'SWAP': 114, + 'UNPACK_EX': 115, + 'UNPACK_SEQUENCE': 116, + 'YIELD_VALUE': 117, + 'INSTRUMENTED_END_FOR': 235, + 'INSTRUMENTED_POP_ITER': 236, 'INSTRUMENTED_END_SEND': 237, 'INSTRUMENTED_LOAD_SUPER_ATTR': 238, 'INSTRUMENTED_FOR_ITER': 239, @@ -350,5 +352,5 @@ 'STORE_FAST_MAYBE_NULL': 265, } -HAVE_ARGUMENT = 42 -MIN_INSTRUMENTED_OPCODE = 236 +HAVE_ARGUMENT = 43 +MIN_INSTRUMENTED_OPCODE = 235 diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py index 5dbe4b28d236d3..6e6e5aaac15caf 100644 --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -477,7 +477,12 @@ def create_task(self, coro, *, name=None, context=None): task.set_name(name) - return task + try: + return task + finally: + # gh-128552: prevent a refcycle of + # task.exception().__traceback__->BaseEventLoop.create_task->task + del task def set_task_factory(self, factory): """Set a task factory that will be used by loop.create_task(). @@ -873,7 +878,10 @@ def call_soon_threadsafe(self, callback, *args, context=None): self._check_closed() if self._debug: self._check_callback(callback, 'call_soon_threadsafe') - handle = self._call_soon(callback, args, context) + handle = events._ThreadSafeHandle(callback, args, self, context) + self._ready.append(handle) + if handle._source_traceback: + del handle._source_traceback[-1] if handle._source_traceback: del handle._source_traceback[-1] self._write_to_self() diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py index 6e291d28ec81ae..2ee9870e80f20b 100644 --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -113,6 +113,34 @@ def _run(self): self._loop.call_exception_handler(context) self = None # Needed to break cycles when an exception occurs. +# _ThreadSafeHandle is used for callbacks scheduled with call_soon_threadsafe +# and is thread safe unlike Handle which is not thread safe. +class _ThreadSafeHandle(Handle): + + __slots__ = ('_lock',) + + def __init__(self, callback, args, loop, context=None): + super().__init__(callback, args, loop, context) + self._lock = threading.RLock() + + def cancel(self): + with self._lock: + return super().cancel() + + def cancelled(self): + with self._lock: + return super().cancelled() + + def _run(self): + # The event loop checks for cancellation without holding the lock + # It is possible that the handle is cancelled after the check + # but before the callback is called so check it again after acquiring + # the lock and return without calling the callback if it is cancelled. + with self._lock: + if self._cancelled: + return + return super()._run() + class TimerHandle(Handle): """Object returned by timed callback registration methods.""" diff --git a/Lib/asyncio/taskgroups.py b/Lib/asyncio/taskgroups.py index 9fa772ca9d02cc..8af199d6dcc41a 100644 --- a/Lib/asyncio/taskgroups.py +++ b/Lib/asyncio/taskgroups.py @@ -205,7 +205,12 @@ def create_task(self, coro, *, name=None, context=None): else: self._tasks.add(task) task.add_done_callback(self._on_task_done) - return task + try: + return task + finally: + # gh-128552: prevent a refcycle of + # task.exception().__traceback__->TaskGroup.create_task->task + del task # Since Python 3.8 Tasks propagate all exceptions correctly, # except for KeyboardInterrupt and SystemExit which are diff --git a/Lib/asyncio/timeouts.py b/Lib/asyncio/timeouts.py index e6f5100691d362..09342dc7c1310b 100644 --- a/Lib/asyncio/timeouts.py +++ b/Lib/asyncio/timeouts.py @@ -1,7 +1,6 @@ import enum from types import TracebackType -from typing import final, Optional, Type from . import events from . import exceptions @@ -23,14 +22,13 @@ class _State(enum.Enum): EXITED = "finished" -@final class Timeout: """Asynchronous context manager for cancelling overdue coroutines. Use `timeout()` or `timeout_at()` rather than instantiating this class directly. """ - def __init__(self, when: Optional[float]) -> None: + def __init__(self, when: float | None) -> None: """Schedule a timeout that will trigger at a given loop time. - If `when` is `None`, the timeout will never trigger. @@ -39,15 +37,15 @@ def __init__(self, when: Optional[float]) -> None: """ self._state = _State.CREATED - self._timeout_handler: Optional[events.TimerHandle] = None - self._task: Optional[tasks.Task] = None + self._timeout_handler: events.TimerHandle | None = None + self._task: tasks.Task | None = None self._when = when - def when(self) -> Optional[float]: + def when(self) -> float | None: """Return the current deadline.""" return self._when - def reschedule(self, when: Optional[float]) -> None: + def reschedule(self, when: float | None) -> None: """Reschedule the timeout.""" if self._state is not _State.ENTERED: if self._state is _State.CREATED: @@ -96,10 +94,10 @@ async def __aenter__(self) -> "Timeout": async def __aexit__( self, - exc_type: Optional[Type[BaseException]], - exc_val: Optional[BaseException], - exc_tb: Optional[TracebackType], - ) -> Optional[bool]: + exc_type: type[BaseException] | None, + exc_val: BaseException | None, + exc_tb: TracebackType | None, + ) -> bool | None: assert self._state in (_State.ENTERED, _State.EXPIRING) if self._timeout_handler is not None: @@ -142,7 +140,7 @@ def _insert_timeout_error(exc_val: BaseException) -> None: exc_val = exc_val.__context__ -def timeout(delay: Optional[float]) -> Timeout: +def timeout(delay: float | None) -> Timeout: """Timeout async context manager. Useful in cases when you want to apply timeout logic around block @@ -162,7 +160,7 @@ def timeout(delay: Optional[float]) -> Timeout: return Timeout(loop.time() + delay if delay is not None else None) -def timeout_at(when: Optional[float]) -> Timeout: +def timeout_at(when: float | None) -> Timeout: """Schedule the timeout at absolute time. Like timeout() but argument gives absolute time in the same clock system diff --git a/Lib/email/message.py b/Lib/email/message.py index a58afc5fe5f68e..87fcab68868d46 100644 --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -286,8 +286,12 @@ def get_payload(self, i=None, decode=False): if i is not None and not isinstance(self._payload, list): raise TypeError('Expected list, got %s' % type(self._payload)) payload = self._payload - # cte might be a Header, so for now stringify it. - cte = str(self.get('content-transfer-encoding', '')).lower() + cte = self.get('content-transfer-encoding', '') + if hasattr(cte, 'cte'): + cte = cte.cte + else: + # cte might be a Header, so for now stringify it. + cte = str(cte).strip().lower() # payload may be bytes here. if not decode: if isinstance(payload, str) and utils._has_surrogates(payload): diff --git a/Lib/pydoc.py b/Lib/pydoc.py index c863794ea14ef9..9e84292aaf825f 100644 --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -53,6 +53,7 @@ class or function within a module or module in a package. If the # the current directory is changed with os.chdir(), an incorrect # path will be displayed. +import ast import __future__ import builtins import importlib._bootstrap @@ -384,21 +385,29 @@ def ispackage(path): return False def source_synopsis(file): - line = file.readline() - while line[:1] == '#' or not line.strip(): - line = file.readline() - if not line: break - line = line.strip() - if line[:4] == 'r"""': line = line[1:] - if line[:3] == '"""': - line = line[3:] - if line[-1:] == '\\': line = line[:-1] - while not line.strip(): - line = file.readline() - if not line: break - result = line.split('"""')[0].strip() - else: result = None - return result + """Return the one-line summary of a file object, if present""" + + string = '' + try: + tokens = tokenize.generate_tokens(file.readline) + for tok_type, tok_string, _, _, _ in tokens: + if tok_type == tokenize.STRING: + string += tok_string + elif tok_type == tokenize.NEWLINE: + with warnings.catch_warnings(): + # Ignore the "invalid escape sequence" warning. + warnings.simplefilter("ignore", SyntaxWarning) + docstring = ast.literal_eval(string) + if not isinstance(docstring, str): + return None + return docstring.strip().split('\n')[0].strip() + elif tok_type == tokenize.OP and tok_string in ('(', ')'): + string += tok_string + elif tok_type not in (tokenize.COMMENT, tokenize.NL, tokenize.ENCODING): + return None + except (tokenize.TokenError, UnicodeDecodeError, SyntaxError): + return None + return None def synopsis(filename, cache={}): """Get the one-line summary out of a module file.""" diff --git a/Lib/sysconfig/__init__.py b/Lib/sysconfig/__init__.py index ed7b6a335d01d4..20d506bcd45abc 100644 --- a/Lib/sysconfig/__init__.py +++ b/Lib/sysconfig/__init__.py @@ -485,10 +485,10 @@ def _init_config_vars(): _init_posix(_CONFIG_VARS) # If we are cross-compiling, load the prefixes from the Makefile instead. if '_PYTHON_PROJECT_BASE' in os.environ: - prefix = _CONFIG_VARS['prefix'] - exec_prefix = _CONFIG_VARS['exec_prefix'] - base_prefix = _CONFIG_VARS['prefix'] - base_exec_prefix = _CONFIG_VARS['exec_prefix'] + prefix = _CONFIG_VARS['host_prefix'] + exec_prefix = _CONFIG_VARS['host_exec_prefix'] + base_prefix = _CONFIG_VARS['host_prefix'] + base_exec_prefix = _CONFIG_VARS['host_exec_prefix'] abiflags = _CONFIG_VARS['ABIFLAGS'] # Normalized versions of prefix and exec_prefix are handy to have; diff --git a/Lib/test/test__interpreters.py b/Lib/test/test__interpreters.py index bf3165e2341949..fd444f1f06ce48 100644 --- a/Lib/test/test__interpreters.py +++ b/Lib/test/test__interpreters.py @@ -557,7 +557,7 @@ def setUp(self): self.id = _interpreters.create() def test_signatures(self): - # for method in ['exec', 'run_string', 'run_func']: + # See https://github.com/python/cpython/issues/126654 msg = "expected 'shared' to be a dict" with self.assertRaisesRegex(TypeError, msg): _interpreters.exec(self.id, 'a', 1) @@ -568,6 +568,17 @@ def test_signatures(self): with self.assertRaisesRegex(TypeError, msg): _interpreters.run_func(self.id, lambda: None, shared=1) + def test_invalid_shared_encoding(self): + # See https://github.com/python/cpython/issues/127196 + bad_shared = {"\uD82A": 0} + msg = 'surrogates not allowed' + with self.assertRaisesRegex(UnicodeEncodeError, msg): + _interpreters.exec(self.id, 'a', shared=bad_shared) + with self.assertRaisesRegex(UnicodeEncodeError, msg): + _interpreters.run_string(self.id, 'a', shared=bad_shared) + with self.assertRaisesRegex(UnicodeEncodeError, msg): + _interpreters.run_func(self.id, lambda: None, shared=bad_shared) + class RunStringTests(TestBase): diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py index c8439c9af5e6ba..ed75b909317357 100644 --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -353,6 +353,124 @@ def run_in_thread(): t.join() self.assertEqual(results, ['hello', 'world']) + def test_call_soon_threadsafe_handle_block_check_cancelled(self): + results = [] + + callback_started = threading.Event() + callback_finished = threading.Event() + def callback(arg): + callback_started.set() + results.append(arg) + time.sleep(1) + callback_finished.set() + + def run_in_thread(): + handle = self.loop.call_soon_threadsafe(callback, 'hello') + self.assertIsInstance(handle, events._ThreadSafeHandle) + callback_started.wait() + # callback started so it should block checking for cancellation + # until it finishes + self.assertFalse(handle.cancelled()) + self.assertTrue(callback_finished.is_set()) + self.loop.call_soon_threadsafe(self.loop.stop) + + t = threading.Thread(target=run_in_thread) + t.start() + + self.loop.run_forever() + t.join() + self.assertEqual(results, ['hello']) + + def test_call_soon_threadsafe_handle_block_cancellation(self): + results = [] + + callback_started = threading.Event() + callback_finished = threading.Event() + def callback(arg): + callback_started.set() + results.append(arg) + time.sleep(1) + callback_finished.set() + + def run_in_thread(): + handle = self.loop.call_soon_threadsafe(callback, 'hello') + self.assertIsInstance(handle, events._ThreadSafeHandle) + callback_started.wait() + # callback started so it cannot be cancelled from other thread until + # it finishes + handle.cancel() + self.assertTrue(callback_finished.is_set()) + self.loop.call_soon_threadsafe(self.loop.stop) + + t = threading.Thread(target=run_in_thread) + t.start() + + self.loop.run_forever() + t.join() + self.assertEqual(results, ['hello']) + + def test_call_soon_threadsafe_handle_cancel_same_thread(self): + results = [] + callback_started = threading.Event() + callback_finished = threading.Event() + + fut = concurrent.futures.Future() + def callback(arg): + callback_started.set() + handle = fut.result() + handle.cancel() + results.append(arg) + callback_finished.set() + self.loop.stop() + + def run_in_thread(): + handle = self.loop.call_soon_threadsafe(callback, 'hello') + fut.set_result(handle) + self.assertIsInstance(handle, events._ThreadSafeHandle) + callback_started.wait() + # callback cancels itself from same thread so it has no effect + # it runs to completion + self.assertTrue(handle.cancelled()) + self.assertTrue(callback_finished.is_set()) + self.loop.call_soon_threadsafe(self.loop.stop) + + t = threading.Thread(target=run_in_thread) + t.start() + + self.loop.run_forever() + t.join() + self.assertEqual(results, ['hello']) + + def test_call_soon_threadsafe_handle_cancel_other_thread(self): + results = [] + ev = threading.Event() + + callback_finished = threading.Event() + def callback(arg): + results.append(arg) + callback_finished.set() + self.loop.stop() + + def run_in_thread(): + handle = self.loop.call_soon_threadsafe(callback, 'hello') + # handle can be cancelled from other thread if not started yet + self.assertIsInstance(handle, events._ThreadSafeHandle) + handle.cancel() + self.assertTrue(handle.cancelled()) + self.assertFalse(callback_finished.is_set()) + ev.set() + self.loop.call_soon_threadsafe(self.loop.stop) + + # block the main loop until the callback is added and cancelled in the + # other thread + self.loop.call_soon(ev.wait) + t = threading.Thread(target=run_in_thread) + t.start() + self.loop.run_forever() + t.join() + self.assertEqual(results, []) + self.assertFalse(callback_finished.is_set()) + def test_call_soon_threadsafe_same_thread(self): results = [] diff --git a/Lib/test/test_asyncio/test_taskgroups.py b/Lib/test/test_asyncio/test_taskgroups.py index c47bf4ec9ed64b..870fa8dbbf2714 100644 --- a/Lib/test/test_asyncio/test_taskgroups.py +++ b/Lib/test/test_asyncio/test_taskgroups.py @@ -1,6 +1,7 @@ # Adapted with permission from the EdgeDB project; # license: PSFL. +import weakref import sys import gc import asyncio @@ -38,7 +39,25 @@ def no_other_refs(): return [coro] -class TestTaskGroup(unittest.IsolatedAsyncioTestCase): +def set_gc_state(enabled): + was_enabled = gc.isenabled() + if enabled: + gc.enable() + else: + gc.disable() + return was_enabled + + +@contextlib.contextmanager +def disable_gc(): + was_enabled = set_gc_state(enabled=False) + try: + yield + finally: + set_gc_state(enabled=was_enabled) + + +class BaseTestTaskGroup: async def test_taskgroup_01(self): @@ -832,15 +851,15 @@ async def test_taskgroup_without_parent_task(self): with self.assertRaisesRegex(RuntimeError, "has not been entered"): tg.create_task(coro) - def test_coro_closed_when_tg_closed(self): + async def test_coro_closed_when_tg_closed(self): async def run_coro_after_tg_closes(): async with taskgroups.TaskGroup() as tg: pass coro = asyncio.sleep(0) with self.assertRaisesRegex(RuntimeError, "is finished"): tg.create_task(coro) - loop = asyncio.get_event_loop() - loop.run_until_complete(run_coro_after_tg_closes()) + + await run_coro_after_tg_closes() async def test_cancelling_level_preserved(self): async def raise_after(t, e): @@ -965,6 +984,30 @@ async def coro_fn(): self.assertIsInstance(exc, _Done) self.assertListEqual(gc.get_referrers(exc), no_other_refs()) + + async def test_exception_refcycles_parent_task_wr(self): + """Test that TaskGroup deletes self._parent_task and create_task() deletes task""" + tg = asyncio.TaskGroup() + exc = None + + class _Done(Exception): + pass + + async def coro_fn(): + async with tg: + raise _Done + + with disable_gc(): + try: + async with asyncio.TaskGroup() as tg2: + task_wr = weakref.ref(tg2.create_task(coro_fn())) + except* _Done as excs: + exc = excs.exceptions[0].exceptions[0] + + self.assertIsNone(task_wr()) + self.assertIsInstance(exc, _Done) + self.assertListEqual(gc.get_referrers(exc), no_other_refs()) + async def test_exception_refcycles_propagate_cancellation_error(self): """Test that TaskGroup deletes propagate_cancellation_error""" tg = asyncio.TaskGroup() @@ -998,5 +1041,16 @@ class MyKeyboardInterrupt(KeyboardInterrupt): self.assertListEqual(gc.get_referrers(exc), no_other_refs()) +class TestTaskGroup(BaseTestTaskGroup, unittest.IsolatedAsyncioTestCase): + loop_factory = asyncio.EventLoop + +class TestEagerTaskTaskGroup(BaseTestTaskGroup, unittest.IsolatedAsyncioTestCase): + @staticmethod + def loop_factory(): + loop = asyncio.EventLoop() + loop.set_task_factory(asyncio.eager_task_factory) + return loop + + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_capi/test_file.py b/Lib/test/test_capi/test_file.py new file mode 100644 index 00000000000000..a67a5121c4588b --- /dev/null +++ b/Lib/test/test_capi/test_file.py @@ -0,0 +1,84 @@ +import os +import unittest +from test import support +from test.support import import_helper, os_helper + +_testcapi = import_helper.import_module('_testcapi') + +NULL = None + + +class CAPIFileTest(unittest.TestCase): + def test_py_fopen(self): + # Test Py_fopen() and Py_fclose() + + with open(__file__, "rb") as fp: + source = fp.read() + + for filename in (__file__, os.fsencode(__file__)): + with self.subTest(filename=filename): + data = _testcapi.py_fopen(filename, "rb") + self.assertEqual(data, source[:256]) + + data = _testcapi.py_fopen(os_helper.FakePath(filename), "rb") + self.assertEqual(data, source[:256]) + + filenames = [ + os_helper.TESTFN, + os.fsencode(os_helper.TESTFN), + ] + if os_helper.TESTFN_UNDECODABLE is not None: + filenames.append(os_helper.TESTFN_UNDECODABLE) + filenames.append(os.fsdecode(os_helper.TESTFN_UNDECODABLE)) + if os_helper.TESTFN_UNENCODABLE is not None: + filenames.append(os_helper.TESTFN_UNENCODABLE) + for filename in filenames: + with self.subTest(filename=filename): + try: + with open(filename, "wb") as fp: + fp.write(source) + except OSError: + # TESTFN_UNDECODABLE cannot be used to create a file + # on macOS/WASI. + filename = None + continue + try: + data = _testcapi.py_fopen(filename, "rb") + self.assertEqual(data, source[:256]) + finally: + os_helper.unlink(filename) + + # embedded null character/byte in the filename + with self.assertRaises(ValueError): + _testcapi.py_fopen("a\x00b", "rb") + with self.assertRaises(ValueError): + _testcapi.py_fopen(b"a\x00b", "rb") + + # non-ASCII mode failing with "Invalid argument" + with self.assertRaises(OSError): + _testcapi.py_fopen(__file__, b"\xc2\x80") + with self.assertRaises(OSError): + # \x98 is invalid in cp1250, cp1251, cp1257 + # \x9d is invalid in cp1252-cp1255, cp1258 + _testcapi.py_fopen(__file__, b"\xc2\x98\xc2\x9d") + # UnicodeDecodeError can come from the audit hook code + with self.assertRaises((UnicodeDecodeError, OSError)): + _testcapi.py_fopen(__file__, b"\x98\x9d") + + # invalid filename type + for invalid_type in (123, object()): + with self.subTest(filename=invalid_type): + with self.assertRaises(TypeError): + _testcapi.py_fopen(invalid_type, "rb") + + if support.MS_WINDOWS: + with self.assertRaises(OSError): + # On Windows, the file mode is limited to 10 characters + _testcapi.py_fopen(__file__, "rt+, ccs=UTF-8") + + # CRASHES _testcapi.py_fopen(NULL, 'rb') + # CRASHES _testcapi.py_fopen(__file__, NULL) + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_capi/test_misc.py b/Lib/test/test_capi/test_misc.py index ada30181aeeca9..b62bc4c2ecd980 100644 --- a/Lib/test/test_capi/test_misc.py +++ b/Lib/test/test_capi/test_misc.py @@ -3335,6 +3335,49 @@ def run(self): self.assertEqual(len(set(py_thread_ids)), len(py_thread_ids), py_thread_ids) +class TestVersions(unittest.TestCase): + full_cases = ( + (3, 4, 1, 0xA, 2, 0x030401a2), + (3, 10, 0, 0xF, 0, 0x030a00f0), + (0x103, 0x10B, 0xFF00, -1, 0xF0, 0x030b00f0), # test masking + ) + xy_cases = ( + (3, 4, 0x03040000), + (3, 10, 0x030a0000), + (0x103, 0x10B, 0x030b0000), # test masking + ) + + def test_pack_full_version(self): + for *args, expected in self.full_cases: + with self.subTest(hexversion=hex(expected)): + result = _testlimitedcapi.pack_full_version(*args) + self.assertEqual(result, expected) + + def test_pack_version(self): + for *args, expected in self.xy_cases: + with self.subTest(hexversion=hex(expected)): + result = _testlimitedcapi.pack_version(*args) + self.assertEqual(result, expected) + + def test_pack_full_version_ctypes(self): + ctypes = import_helper.import_module('ctypes') + ctypes_func = ctypes.pythonapi.Py_PACK_FULL_VERSION + ctypes_func.restype = ctypes.c_uint32 + ctypes_func.argtypes = [ctypes.c_int] * 5 + for *args, expected in self.full_cases: + with self.subTest(hexversion=hex(expected)): + result = ctypes_func(*args) + self.assertEqual(result, expected) + + def test_pack_version_ctypes(self): + ctypes = import_helper.import_module('ctypes') + ctypes_func = ctypes.pythonapi.Py_PACK_VERSION + ctypes_func.restype = ctypes.c_uint32 + ctypes_func.argtypes = [ctypes.c_int] * 2 + for *args, expected in self.xy_cases: + with self.subTest(hexversion=hex(expected)): + result = ctypes_func(*args) + self.assertEqual(result, expected) if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_capi/test_opt.py b/Lib/test/test_capi/test_opt.py index 4cf9b66170c055..d84702411afe41 100644 --- a/Lib/test/test_capi/test_opt.py +++ b/Lib/test/test_capi/test_opt.py @@ -1,4 +1,5 @@ import contextlib +import itertools import sys import textwrap import unittest @@ -1511,6 +1512,49 @@ def test_jit_error_pops(self): with self.assertRaises(TypeError): {item for item in items} + def test_power_type_depends_on_input_values(self): + template = textwrap.dedent(""" + import _testinternalcapi + + L, R, X, Y = {l}, {r}, {x}, {y} + + def check(actual: complex, expected: complex) -> None: + assert actual == expected, (actual, expected) + assert type(actual) is type(expected), (actual, expected) + + def f(l: complex, r: complex) -> None: + expected_local_local = pow(l, r) + pow(l, r) + expected_const_local = pow(L, r) + pow(L, r) + expected_local_const = pow(l, R) + pow(l, R) + expected_const_const = pow(L, R) + pow(L, R) + for _ in range(_testinternalcapi.TIER2_THRESHOLD): + # Narrow types: + l + l, r + r + # The powers produce results, and the addition is unguarded: + check(l ** r + l ** r, expected_local_local) + check(L ** r + L ** r, expected_const_local) + check(l ** R + l ** R, expected_local_const) + check(L ** R + L ** R, expected_const_const) + + # JIT for one pair of values... + f(L, R) + # ...then run with another: + f(X, Y) + """) + interesting = [ + (1, 1), # int ** int -> int + (1, -1), # int ** int -> float + (1.0, 1), # float ** int -> float + (1, 1.0), # int ** float -> float + (-1, 0.5), # int ** float -> complex + (1.0, 1.0), # float ** float -> float + (-1.0, 0.5), # float ** float -> complex + ] + for (l, r), (x, y) in itertools.product(interesting, repeat=2): + s = template.format(l=l, r=r, x=x, y=y) + with self.subTest(l=l, r=r, x=x, y=y): + script_helper.assert_python_ok("-c", s) + def global_identity(x): return x diff --git a/Lib/test/test_code.py b/Lib/test/test_code.py index 2a1b26e8a1ffd1..7ffa4eb8639add 100644 --- a/Lib/test/test_code.py +++ b/Lib/test/test_code.py @@ -215,6 +215,8 @@ from test.support import threading_helper, import_helper from test.support.bytecode_helper import instructions_with_positions from opcode import opmap, opname +from _testcapi import code_offset_to_line + COPY_FREE_VARS = opmap['COPY_FREE_VARS'] @@ -896,6 +898,44 @@ async def async_func(): rc, out, err = assert_python_ok('-OO', '-c', code) + def test_co_branches(self): + + def get_line_branches(func): + code = func.__code__ + base = code.co_firstlineno + return [ + ( + code_offset_to_line(code, src) - base, + code_offset_to_line(code, left) - base, + code_offset_to_line(code, right) - base + ) for (src, left, right) in + code.co_branches() + ] + + def simple(x): + if x: + A + else: + B + + self.assertEqual( + get_line_branches(simple), + [(1,2,4)]) + + def with_extended_args(x): + if x: + A.x; A.x; A.x; A.x; A.x; A.x; + A.x; A.x; A.x; A.x; A.x; A.x; + A.x; A.x; A.x; A.x; A.x; A.x; + A.x; A.x; A.x; A.x; A.x; A.x; + A.x; A.x; A.x; A.x; A.x; A.x; + else: + B + + self.assertEqual( + get_line_branches(with_extended_args), + [(1,2,8)]) + if check_impl_detail(cpython=True) and ctypes is not None: py = ctypes.pythonapi freefunc = ctypes.CFUNCTYPE(None,ctypes.c_voidp) diff --git a/Lib/test/test_compiler_codegen.py b/Lib/test/test_compiler_codegen.py index f8c4fc14c91ebe..cf5e2d901db4de 100644 --- a/Lib/test/test_compiler_codegen.py +++ b/Lib/test/test_compiler_codegen.py @@ -29,7 +29,6 @@ def test_if_expression(self): ('LOAD_CONST', 0, 1), ('TO_BOOL', 0, 1), ('POP_JUMP_IF_FALSE', false_lbl := self.Label(), 1), - ('NOT_TAKEN', None, 1), ('LOAD_SMALL_INT', 42, 1), ('JUMP_NO_INTERRUPT', exit_lbl := self.Label()), false_lbl, @@ -50,7 +49,6 @@ def test_for_loop(self): ('GET_ITER', None, 1), loop_lbl := self.Label(), ('FOR_ITER', exit_lbl := self.Label(), 1), - ('NOT_TAKEN', None, 1), ('NOP', None, 1, 1), ('STORE_NAME', 1, 1), ('LOAD_NAME', 2, 2), @@ -61,7 +59,7 @@ def test_for_loop(self): ('JUMP', loop_lbl), exit_lbl, ('END_FOR', None), - ('POP_TOP', None), + ('POP_ITER', None), ('LOAD_CONST', 0), ('RETURN_VALUE', None), ] diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py index 955a3e4cb9e4f7..76d9b5401d7d8e 100644 --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -175,14 +175,13 @@ def bug708901(): %3d CALL 2 GET_ITER - L1: FOR_ITER 4 (to L2) - NOT_TAKEN + L1: FOR_ITER 3 (to L2) STORE_FAST 0 (res) -%3d JUMP_BACKWARD 6 (to L1) +%3d JUMP_BACKWARD 5 (to L1) %3d L2: END_FOR - POP_TOP + POP_ITER LOAD_CONST 0 (None) RETURN_VALUE """ % (bug708901.__code__.co_firstlineno, @@ -201,8 +200,7 @@ def bug1333982(x=[]): dis_bug1333982 = """\ %3d RESUME 0 -%3d NOT_TAKEN - LOAD_COMMON_CONSTANT 0 (AssertionError) +%3d LOAD_COMMON_CONSTANT 0 (AssertionError) LOAD_CONST 0 ( at 0x..., file "%s", line %d>) MAKE_FUNCTION LOAD_FAST 0 (x) @@ -434,7 +432,7 @@ def foo(a: int, b: str) -> str: 1 LOAD_SMALL_INT 0 STORE_NAME 0 (x) - 2 L1: NOT_TAKEN + 2 L1: NOP 3 LOAD_NAME 0 (x) LOAD_SMALL_INT 1 @@ -649,11 +647,11 @@ async def _asyncwith(c): L20: CLEANUP_THROW L21: END_SEND TO_BOOL - POP_JUMP_IF_TRUE 2 (to L22) - NOT_TAKEN - RERAISE 2 - L22: POP_TOP - L23: POP_EXCEPT + POP_JUMP_IF_TRUE 2 (to L24) + L22: NOT_TAKEN + L23: RERAISE 2 + L24: POP_TOP + L25: POP_EXCEPT POP_TOP POP_TOP POP_TOP @@ -663,24 +661,25 @@ async def _asyncwith(c): LOAD_CONST 0 (None) RETURN_VALUE - -- L24: COPY 3 + -- L26: COPY 3 POP_EXCEPT RERAISE 1 - L25: CALL_INTRINSIC_1 3 (INTRINSIC_STOPITERATION_ERROR) + L27: CALL_INTRINSIC_1 3 (INTRINSIC_STOPITERATION_ERROR) RERAISE 1 ExceptionTable: - L1 to L3 -> L25 [0] lasti + L1 to L3 -> L27 [0] lasti L3 to L4 -> L12 [4] - L4 to L6 -> L25 [0] lasti + L4 to L6 -> L27 [0] lasti L6 to L7 -> L16 [2] lasti - L7 to L9 -> L25 [0] lasti + L7 to L9 -> L27 [0] lasti L9 to L10 -> L14 [2] - L10 to L13 -> L25 [0] lasti - L14 to L15 -> L25 [0] lasti - L16 to L18 -> L24 [4] lasti + L10 to L13 -> L27 [0] lasti + L14 to L15 -> L27 [0] lasti + L16 to L18 -> L26 [4] lasti L18 to L19 -> L20 [7] - L19 to L23 -> L24 [4] lasti - L23 to L25 -> L25 [0] lasti + L19 to L22 -> L26 [4] lasti + L23 to L25 -> L26 [4] lasti + L25 to L27 -> L27 [0] lasti """ % (_asyncwith.__code__.co_firstlineno, _asyncwith.__code__.co_firstlineno + 1, _asyncwith.__code__.co_firstlineno + 2, @@ -844,8 +843,7 @@ def foo(x): L1: RESUME 0 LOAD_FAST 0 (.0) GET_ITER - L2: FOR_ITER 11 (to L3) - NOT_TAKEN + L2: FOR_ITER 10 (to L3) STORE_FAST 1 (z) LOAD_DEREF 2 (x) LOAD_FAST 1 (z) @@ -853,9 +851,9 @@ def foo(x): YIELD_VALUE 0 RESUME 5 POP_TOP - JUMP_BACKWARD 13 (to L2) + JUMP_BACKWARD 12 (to L2) L3: END_FOR - POP_TOP + POP_ITER LOAD_CONST 0 (None) RETURN_VALUE @@ -899,18 +897,17 @@ def loop_test(): LOAD_SMALL_INT 3 BINARY_OP 5 (*) GET_ITER - L1: FOR_ITER_LIST 15 (to L2) - NOT_TAKEN + L1: FOR_ITER_LIST 14 (to L2) STORE_FAST 0 (i) %3d LOAD_GLOBAL_MODULE 1 (load_test + NULL) LOAD_FAST 0 (i) CALL_PY_GENERAL 1 POP_TOP - JUMP_BACKWARD 17 (to L1) + JUMP_BACKWARD 16 (to L1) %3d L2: END_FOR - POP_TOP + POP_ITER LOAD_CONST_IMMORTAL 1 (None) RETURN_VALUE """ % (loop_test.__code__.co_firstlineno, @@ -1706,214 +1703,211 @@ def _prepare_test_cases(): Instruction = dis.Instruction expected_opinfo_outer = [ - Instruction(opname='MAKE_CELL', opcode=93, arg=0, argval='a', argrepr='a', offset=0, start_offset=0, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='MAKE_CELL', opcode=93, arg=1, argval='b', argrepr='b', offset=2, start_offset=2, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='MAKE_CELL', opcode=94, arg=0, argval='a', argrepr='a', offset=0, start_offset=0, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='MAKE_CELL', opcode=94, arg=1, argval='b', argrepr='b', offset=2, start_offset=2, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), Instruction(opname='RESUME', opcode=149, arg=0, argval=0, argrepr='', offset=4, start_offset=4, starts_line=True, line_number=1, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=3, argval=(3, 4), argrepr='(3, 4)', offset=6, start_offset=6, starts_line=True, line_number=2, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=0, argval='a', argrepr='a', offset=8, start_offset=8, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=1, argval='b', argrepr='b', offset=10, start_offset=10, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), - Instruction(opname='BUILD_TUPLE', opcode=49, arg=2, argval=2, argrepr='', offset=12, start_offset=12, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=0, argval=code_object_f, argrepr=repr(code_object_f), offset=14, start_offset=14, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=3, argval=(3, 4), argrepr='(3, 4)', offset=6, start_offset=6, starts_line=True, line_number=2, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=0, argval='a', argrepr='a', offset=8, start_offset=8, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=1, argval='b', argrepr='b', offset=10, start_offset=10, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), + Instruction(opname='BUILD_TUPLE', opcode=50, arg=2, argval=2, argrepr='', offset=12, start_offset=12, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=0, argval=code_object_f, argrepr=repr(code_object_f), offset=14, start_offset=14, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), Instruction(opname='MAKE_FUNCTION', opcode=23, arg=None, argval=None, argrepr='', offset=16, start_offset=16, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), - Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=104, arg=8, argval=8, argrepr='closure', offset=18, start_offset=18, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), - Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=104, arg=1, argval=1, argrepr='defaults', offset=20, start_offset=20, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), - Instruction(opname='STORE_FAST', opcode=108, arg=2, argval='f', argrepr='f', offset=22, start_offset=22, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=1, argval='print', argrepr='print + NULL', offset=24, start_offset=24, starts_line=True, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_DEREF', opcode=81, arg=0, argval='a', argrepr='a', offset=34, start_offset=34, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_DEREF', opcode=81, arg=1, argval='b', argrepr='b', offset=36, start_offset=36, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=1, argval='', argrepr="''", offset=38, start_offset=38, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_SMALL_INT', opcode=90, arg=1, argval=1, argrepr='', offset=40, start_offset=40, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), - Instruction(opname='BUILD_LIST', opcode=44, arg=0, argval=0, argrepr='', offset=42, start_offset=42, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), - Instruction(opname='BUILD_MAP', opcode=45, arg=0, argval=0, argrepr='', offset=44, start_offset=44, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=2, argval='Hello world!', argrepr="'Hello world!'", offset=46, start_offset=46, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=7, argval=7, argrepr='', offset=48, start_offset=48, starts_line=False, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=56, start_offset=56, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=2, argval='f', argrepr='f', offset=58, start_offset=58, starts_line=True, line_number=8, label=None, positions=None, cache_info=None), - Instruction(opname='RETURN_VALUE', opcode=34, arg=None, argval=None, argrepr='', offset=60, start_offset=60, starts_line=False, line_number=8, label=None, positions=None, cache_info=None), + Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=105, arg=8, argval=8, argrepr='closure', offset=18, start_offset=18, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), + Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=105, arg=1, argval=1, argrepr='defaults', offset=20, start_offset=20, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), + Instruction(opname='STORE_FAST', opcode=109, arg=2, argval='f', argrepr='f', offset=22, start_offset=22, starts_line=False, line_number=2, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=1, argval='print', argrepr='print + NULL', offset=24, start_offset=24, starts_line=True, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_DEREF', opcode=82, arg=0, argval='a', argrepr='a', offset=34, start_offset=34, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_DEREF', opcode=82, arg=1, argval='b', argrepr='b', offset=36, start_offset=36, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=1, argval='', argrepr="''", offset=38, start_offset=38, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_SMALL_INT', opcode=91, arg=1, argval=1, argrepr='', offset=40, start_offset=40, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), + Instruction(opname='BUILD_LIST', opcode=45, arg=0, argval=0, argrepr='', offset=42, start_offset=42, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), + Instruction(opname='BUILD_MAP', opcode=46, arg=0, argval=0, argrepr='', offset=44, start_offset=44, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=2, argval='Hello world!', argrepr="'Hello world!'", offset=46, start_offset=46, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=7, argval=7, argrepr='', offset=48, start_offset=48, starts_line=False, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=56, start_offset=56, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=2, argval='f', argrepr='f', offset=58, start_offset=58, starts_line=True, line_number=8, label=None, positions=None, cache_info=None), + Instruction(opname='RETURN_VALUE', opcode=35, arg=None, argval=None, argrepr='', offset=60, start_offset=60, starts_line=False, line_number=8, label=None, positions=None, cache_info=None), ] expected_opinfo_f = [ - Instruction(opname='COPY_FREE_VARS', opcode=59, arg=2, argval=2, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='MAKE_CELL', opcode=93, arg=0, argval='c', argrepr='c', offset=2, start_offset=2, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='MAKE_CELL', opcode=93, arg=1, argval='d', argrepr='d', offset=4, start_offset=4, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='COPY_FREE_VARS', opcode=60, arg=2, argval=2, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='MAKE_CELL', opcode=94, arg=0, argval='c', argrepr='c', offset=2, start_offset=2, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='MAKE_CELL', opcode=94, arg=1, argval='d', argrepr='d', offset=4, start_offset=4, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), Instruction(opname='RESUME', opcode=149, arg=0, argval=0, argrepr='', offset=6, start_offset=6, starts_line=True, line_number=2, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=1, argval=(5, 6), argrepr='(5, 6)', offset=8, start_offset=8, starts_line=True, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=3, argval='a', argrepr='a', offset=10, start_offset=10, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=4, argval='b', argrepr='b', offset=12, start_offset=12, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=0, argval='c', argrepr='c', offset=14, start_offset=14, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=1, argval='d', argrepr='d', offset=16, start_offset=16, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='BUILD_TUPLE', opcode=49, arg=4, argval=4, argrepr='', offset=18, start_offset=18, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=0, argval=code_object_inner, argrepr=repr(code_object_inner), offset=20, start_offset=20, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=1, argval=(5, 6), argrepr='(5, 6)', offset=8, start_offset=8, starts_line=True, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=3, argval='a', argrepr='a', offset=10, start_offset=10, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=4, argval='b', argrepr='b', offset=12, start_offset=12, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=0, argval='c', argrepr='c', offset=14, start_offset=14, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=1, argval='d', argrepr='d', offset=16, start_offset=16, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='BUILD_TUPLE', opcode=50, arg=4, argval=4, argrepr='', offset=18, start_offset=18, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=0, argval=code_object_inner, argrepr=repr(code_object_inner), offset=20, start_offset=20, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), Instruction(opname='MAKE_FUNCTION', opcode=23, arg=None, argval=None, argrepr='', offset=22, start_offset=22, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=104, arg=8, argval=8, argrepr='closure', offset=24, start_offset=24, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=104, arg=1, argval=1, argrepr='defaults', offset=26, start_offset=26, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='STORE_FAST', opcode=108, arg=2, argval='inner', argrepr='inner', offset=28, start_offset=28, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=1, argval='print', argrepr='print + NULL', offset=30, start_offset=30, starts_line=True, line_number=5, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_DEREF', opcode=81, arg=3, argval='a', argrepr='a', offset=40, start_offset=40, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_DEREF', opcode=81, arg=4, argval='b', argrepr='b', offset=42, start_offset=42, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_DEREF', opcode=81, arg=0, argval='c', argrepr='c', offset=44, start_offset=44, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_DEREF', opcode=81, arg=1, argval='d', argrepr='d', offset=46, start_offset=46, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=4, argval=4, argrepr='', offset=48, start_offset=48, starts_line=False, line_number=5, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=56, start_offset=56, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=2, argval='inner', argrepr='inner', offset=58, start_offset=58, starts_line=True, line_number=6, label=None, positions=None, cache_info=None), - Instruction(opname='RETURN_VALUE', opcode=34, arg=None, argval=None, argrepr='', offset=60, start_offset=60, starts_line=False, line_number=6, label=None, positions=None, cache_info=None), + Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=105, arg=8, argval=8, argrepr='closure', offset=24, start_offset=24, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='SET_FUNCTION_ATTRIBUTE', opcode=105, arg=1, argval=1, argrepr='defaults', offset=26, start_offset=26, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='STORE_FAST', opcode=109, arg=2, argval='inner', argrepr='inner', offset=28, start_offset=28, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=1, argval='print', argrepr='print + NULL', offset=30, start_offset=30, starts_line=True, line_number=5, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_DEREF', opcode=82, arg=3, argval='a', argrepr='a', offset=40, start_offset=40, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_DEREF', opcode=82, arg=4, argval='b', argrepr='b', offset=42, start_offset=42, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_DEREF', opcode=82, arg=0, argval='c', argrepr='c', offset=44, start_offset=44, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_DEREF', opcode=82, arg=1, argval='d', argrepr='d', offset=46, start_offset=46, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=4, argval=4, argrepr='', offset=48, start_offset=48, starts_line=False, line_number=5, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=56, start_offset=56, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=2, argval='inner', argrepr='inner', offset=58, start_offset=58, starts_line=True, line_number=6, label=None, positions=None, cache_info=None), + Instruction(opname='RETURN_VALUE', opcode=35, arg=None, argval=None, argrepr='', offset=60, start_offset=60, starts_line=False, line_number=6, label=None, positions=None, cache_info=None), ] expected_opinfo_inner = [ - Instruction(opname='COPY_FREE_VARS', opcode=59, arg=4, argval=4, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='COPY_FREE_VARS', opcode=60, arg=4, argval=4, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), Instruction(opname='RESUME', opcode=149, arg=0, argval=0, argrepr='', offset=2, start_offset=2, starts_line=True, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=1, argval='print', argrepr='print + NULL', offset=4, start_offset=4, starts_line=True, line_number=4, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_DEREF', opcode=81, arg=2, argval='a', argrepr='a', offset=14, start_offset=14, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_DEREF', opcode=81, arg=3, argval='b', argrepr='b', offset=16, start_offset=16, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_DEREF', opcode=81, arg=4, argval='c', argrepr='c', offset=18, start_offset=18, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_DEREF', opcode=81, arg=5, argval='d', argrepr='d', offset=20, start_offset=20, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST_LOAD_FAST', opcode=85, arg=1, argval=('e', 'f'), argrepr='e, f', offset=22, start_offset=22, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=6, argval=6, argrepr='', offset=24, start_offset=24, starts_line=False, line_number=4, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=32, start_offset=32, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=0, argval=None, argrepr='None', offset=34, start_offset=34, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), - Instruction(opname='RETURN_VALUE', opcode=34, arg=None, argval=None, argrepr='', offset=36, start_offset=36, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=1, argval='print', argrepr='print + NULL', offset=4, start_offset=4, starts_line=True, line_number=4, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_DEREF', opcode=82, arg=2, argval='a', argrepr='a', offset=14, start_offset=14, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_DEREF', opcode=82, arg=3, argval='b', argrepr='b', offset=16, start_offset=16, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_DEREF', opcode=82, arg=4, argval='c', argrepr='c', offset=18, start_offset=18, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_DEREF', opcode=82, arg=5, argval='d', argrepr='d', offset=20, start_offset=20, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST_LOAD_FAST', opcode=86, arg=1, argval=('e', 'f'), argrepr='e, f', offset=22, start_offset=22, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=6, argval=6, argrepr='', offset=24, start_offset=24, starts_line=False, line_number=4, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=32, start_offset=32, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=0, argval=None, argrepr='None', offset=34, start_offset=34, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), + Instruction(opname='RETURN_VALUE', opcode=35, arg=None, argval=None, argrepr='', offset=36, start_offset=36, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), ] expected_opinfo_jumpy = [ Instruction(opname='RESUME', opcode=149, arg=0, argval=0, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=1, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=1, argval='range', argrepr='range + NULL', offset=2, start_offset=2, starts_line=True, line_number=3, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_SMALL_INT', opcode=90, arg=10, argval=10, argrepr='', offset=12, start_offset=12, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=1, argval=1, argrepr='', offset=14, start_offset=14, starts_line=False, line_number=3, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=1, argval='range', argrepr='range + NULL', offset=2, start_offset=2, starts_line=True, line_number=3, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_SMALL_INT', opcode=91, arg=10, argval=10, argrepr='', offset=12, start_offset=12, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=1, argval=1, argrepr='', offset=14, start_offset=14, starts_line=False, line_number=3, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), Instruction(opname='GET_ITER', opcode=16, arg=None, argval=None, argrepr='', offset=22, start_offset=22, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='FOR_ITER', opcode=68, arg=34, argval=96, argrepr='to L4', offset=24, start_offset=24, starts_line=False, line_number=3, label=1, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=28, start_offset=28, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='STORE_FAST', opcode=108, arg=0, argval='i', argrepr='i', offset=30, start_offset=30, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=3, argval='print', argrepr='print + NULL', offset=32, start_offset=32, starts_line=True, line_number=4, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_FAST', opcode=82, arg=0, argval='i', argrepr='i', offset=42, start_offset=42, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=1, argval=1, argrepr='', offset=44, start_offset=44, starts_line=False, line_number=4, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=52, start_offset=52, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=0, argval='i', argrepr='i', offset=54, start_offset=54, starts_line=True, line_number=5, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_SMALL_INT', opcode=90, arg=4, argval=4, argrepr='', offset=56, start_offset=56, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), - Instruction(opname='COMPARE_OP', opcode=55, arg=18, argval='<', argrepr='bool(<)', offset=58, start_offset=58, starts_line=False, line_number=5, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_JUMP_IF_FALSE', opcode=96, arg=3, argval=72, argrepr='to L2', offset=62, start_offset=62, starts_line=False, line_number=5, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=66, start_offset=66, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), - Instruction(opname='JUMP_BACKWARD', opcode=73, arg=24, argval=24, argrepr='to L1', offset=68, start_offset=68, starts_line=True, line_number=6, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='LOAD_FAST', opcode=82, arg=0, argval='i', argrepr='i', offset=72, start_offset=72, starts_line=True, line_number=7, label=2, positions=None, cache_info=None), - Instruction(opname='LOAD_SMALL_INT', opcode=90, arg=6, argval=6, argrepr='', offset=74, start_offset=74, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), - Instruction(opname='COMPARE_OP', opcode=55, arg=148, argval='>', argrepr='bool(>)', offset=76, start_offset=76, starts_line=False, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_JUMP_IF_TRUE', opcode=99, arg=3, argval=90, argrepr='to L3', offset=80, start_offset=80, starts_line=False, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=84, start_offset=84, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), - Instruction(opname='JUMP_BACKWARD', opcode=73, arg=33, argval=24, argrepr='to L1', offset=86, start_offset=86, starts_line=False, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='NOP', opcode=27, arg=None, argval=None, argrepr='', offset=90, start_offset=90, starts_line=True, line_number=None, label=3, positions=None, cache_info=None), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=92, start_offset=92, starts_line=True, line_number=8, label=None, positions=None, cache_info=None), - Instruction(opname='JUMP_FORWARD', opcode=75, arg=13, argval=122, argrepr='to L5', offset=94, start_offset=94, starts_line=False, line_number=8, label=None, positions=None, cache_info=None), - Instruction(opname='END_FOR', opcode=9, arg=None, argval=None, argrepr='', offset=96, start_offset=96, starts_line=True, line_number=3, label=4, positions=None, cache_info=None), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=98, start_offset=98, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=3, argval='print', argrepr='print + NULL', offset=100, start_offset=100, starts_line=True, line_number=10, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=80, arg=0, argval='I can haz else clause?', argrepr="'I can haz else clause?'", offset=110, start_offset=110, starts_line=False, line_number=10, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=1, argval=1, argrepr='', offset=112, start_offset=112, starts_line=False, line_number=10, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=120, start_offset=120, starts_line=False, line_number=10, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST_CHECK', opcode=84, arg=0, argval='i', argrepr='i', offset=122, start_offset=122, starts_line=True, line_number=11, label=5, positions=None, cache_info=None), - Instruction(opname='TO_BOOL', opcode=38, arg=None, argval=None, argrepr='', offset=124, start_offset=124, starts_line=False, line_number=11, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_JUMP_IF_FALSE', opcode=96, arg=37, argval=210, argrepr='to L8', offset=132, start_offset=132, starts_line=False, line_number=11, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=136, start_offset=136, starts_line=False, line_number=11, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=3, argval='print', argrepr='print + NULL', offset=138, start_offset=138, starts_line=True, line_number=12, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_FAST', opcode=82, arg=0, argval='i', argrepr='i', offset=148, start_offset=148, starts_line=False, line_number=12, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=1, argval=1, argrepr='', offset=150, start_offset=150, starts_line=False, line_number=12, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=158, start_offset=158, starts_line=False, line_number=12, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=0, argval='i', argrepr='i', offset=160, start_offset=160, starts_line=True, line_number=13, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_SMALL_INT', opcode=90, arg=1, argval=1, argrepr='', offset=162, start_offset=162, starts_line=False, line_number=13, label=None, positions=None, cache_info=None), - Instruction(opname='BINARY_OP', opcode=43, arg=23, argval=23, argrepr='-=', offset=164, start_offset=164, starts_line=False, line_number=13, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='STORE_FAST', opcode=108, arg=0, argval='i', argrepr='i', offset=168, start_offset=168, starts_line=False, line_number=13, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=0, argval='i', argrepr='i', offset=170, start_offset=170, starts_line=True, line_number=14, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_SMALL_INT', opcode=90, arg=6, argval=6, argrepr='', offset=172, start_offset=172, starts_line=False, line_number=14, label=None, positions=None, cache_info=None), - Instruction(opname='COMPARE_OP', opcode=55, arg=148, argval='>', argrepr='bool(>)', offset=174, start_offset=174, starts_line=False, line_number=14, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_JUMP_IF_FALSE', opcode=96, arg=3, argval=188, argrepr='to L6', offset=178, start_offset=178, starts_line=False, line_number=14, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=182, start_offset=182, starts_line=False, line_number=14, label=None, positions=None, cache_info=None), - Instruction(opname='JUMP_BACKWARD', opcode=73, arg=33, argval=122, argrepr='to L5', offset=184, start_offset=184, starts_line=True, line_number=15, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='LOAD_FAST', opcode=82, arg=0, argval='i', argrepr='i', offset=188, start_offset=188, starts_line=True, line_number=16, label=6, positions=None, cache_info=None), - Instruction(opname='LOAD_SMALL_INT', opcode=90, arg=4, argval=4, argrepr='', offset=190, start_offset=190, starts_line=False, line_number=16, label=None, positions=None, cache_info=None), - Instruction(opname='COMPARE_OP', opcode=55, arg=18, argval='<', argrepr='bool(<)', offset=192, start_offset=192, starts_line=False, line_number=16, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_JUMP_IF_TRUE', opcode=99, arg=3, argval=206, argrepr='to L7', offset=196, start_offset=196, starts_line=False, line_number=16, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=200, start_offset=200, starts_line=False, line_number=16, label=None, positions=None, cache_info=None), - Instruction(opname='JUMP_BACKWARD', opcode=73, arg=42, argval=122, argrepr='to L5', offset=202, start_offset=202, starts_line=False, line_number=16, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='NOP', opcode=27, arg=None, argval=None, argrepr='', offset=206, start_offset=206, starts_line=True, line_number=None, label=7, positions=None, cache_info=None), - Instruction(opname='JUMP_FORWARD', opcode=75, arg=11, argval=232, argrepr='to L9', offset=208, start_offset=208, starts_line=True, line_number=17, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=3, argval='print', argrepr='print + NULL', offset=210, start_offset=210, starts_line=True, line_number=19, label=8, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=80, arg=1, argval='Who let lolcatz into this test suite?', argrepr="'Who let lolcatz into this test suite?'", offset=220, start_offset=220, starts_line=False, line_number=19, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=1, argval=1, argrepr='', offset=222, start_offset=222, starts_line=False, line_number=19, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=230, start_offset=230, starts_line=False, line_number=19, label=None, positions=None, cache_info=None), - Instruction(opname='NOP', opcode=27, arg=None, argval=None, argrepr='', offset=232, start_offset=232, starts_line=True, line_number=20, label=9, positions=None, cache_info=None), - Instruction(opname='LOAD_SMALL_INT', opcode=90, arg=1, argval=1, argrepr='', offset=234, start_offset=234, starts_line=True, line_number=21, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_SMALL_INT', opcode=90, arg=0, argval=0, argrepr='', offset=236, start_offset=236, starts_line=False, line_number=21, label=None, positions=None, cache_info=None), - Instruction(opname='BINARY_OP', opcode=43, arg=11, argval=11, argrepr='/', offset=238, start_offset=238, starts_line=False, line_number=21, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=242, start_offset=242, starts_line=False, line_number=21, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_FAST', opcode=82, arg=0, argval='i', argrepr='i', offset=244, start_offset=244, starts_line=True, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='COPY', opcode=58, arg=1, argval=1, argrepr='', offset=246, start_offset=246, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_SPECIAL', opcode=91, arg=1, argval=1, argrepr='__exit__', offset=248, start_offset=248, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='SWAP', opcode=113, arg=2, argval=2, argrepr='', offset=250, start_offset=250, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='SWAP', opcode=113, arg=3, argval=3, argrepr='', offset=252, start_offset=252, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_SPECIAL', opcode=91, arg=0, argval=0, argrepr='__enter__', offset=254, start_offset=254, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=0, argval=0, argrepr='', offset=256, start_offset=256, starts_line=False, line_number=25, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='STORE_FAST', opcode=108, arg=1, argval='dodgy', argrepr='dodgy', offset=264, start_offset=264, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=3, argval='print', argrepr='print + NULL', offset=266, start_offset=266, starts_line=True, line_number=26, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=80, arg=2, argval='Never reach this', argrepr="'Never reach this'", offset=276, start_offset=276, starts_line=False, line_number=26, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=1, argval=1, argrepr='', offset=278, start_offset=278, starts_line=False, line_number=26, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=286, start_offset=286, starts_line=False, line_number=26, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=3, argval=None, argrepr='None', offset=288, start_offset=288, starts_line=True, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=3, argval=None, argrepr='None', offset=290, start_offset=290, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=3, argval=None, argrepr='None', offset=292, start_offset=292, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=3, argval=3, argrepr='', offset=294, start_offset=294, starts_line=False, line_number=25, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=302, start_offset=302, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=3, argval='print', argrepr='print + NULL', offset=304, start_offset=304, starts_line=True, line_number=28, label=10, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=80, arg=5, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=314, start_offset=314, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=1, argval=1, argrepr='', offset=316, start_offset=316, starts_line=False, line_number=28, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=324, start_offset=324, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=3, argval=None, argrepr='None', offset=326, start_offset=326, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='RETURN_VALUE', opcode=34, arg=None, argval=None, argrepr='', offset=328, start_offset=328, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='PUSH_EXC_INFO', opcode=31, arg=None, argval=None, argrepr='', offset=330, start_offset=330, starts_line=True, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='WITH_EXCEPT_START', opcode=42, arg=None, argval=None, argrepr='', offset=332, start_offset=332, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='TO_BOOL', opcode=38, arg=None, argval=None, argrepr='', offset=334, start_offset=334, starts_line=False, line_number=25, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_JUMP_IF_TRUE', opcode=99, arg=2, argval=350, argrepr='to L11', offset=342, start_offset=342, starts_line=False, line_number=25, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=346, start_offset=346, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='RERAISE', opcode=101, arg=2, argval=2, argrepr='', offset=348, start_offset=348, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=350, start_offset=350, starts_line=False, line_number=25, label=11, positions=None, cache_info=None), - Instruction(opname='POP_EXCEPT', opcode=29, arg=None, argval=None, argrepr='', offset=352, start_offset=352, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=354, start_offset=354, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=356, start_offset=356, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=358, start_offset=358, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='JUMP_BACKWARD_NO_INTERRUPT', opcode=74, arg=29, argval=304, argrepr='to L10', offset=360, start_offset=360, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), - Instruction(opname='COPY', opcode=58, arg=3, argval=3, argrepr='', offset=362, start_offset=362, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='POP_EXCEPT', opcode=29, arg=None, argval=None, argrepr='', offset=364, start_offset=364, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='RERAISE', opcode=101, arg=1, argval=1, argrepr='', offset=366, start_offset=366, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='PUSH_EXC_INFO', opcode=31, arg=None, argval=None, argrepr='', offset=368, start_offset=368, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=4, argval='ZeroDivisionError', argrepr='ZeroDivisionError', offset=370, start_offset=370, starts_line=True, line_number=22, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='CHECK_EXC_MATCH', opcode=5, arg=None, argval=None, argrepr='', offset=380, start_offset=380, starts_line=False, line_number=22, label=None, positions=None, cache_info=None), - Instruction(opname='POP_JUMP_IF_FALSE', opcode=96, arg=15, argval=416, argrepr='to L12', offset=382, start_offset=382, starts_line=False, line_number=22, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), - Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=386, start_offset=386, starts_line=False, line_number=22, label=None, positions=None, cache_info=None), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=388, start_offset=388, starts_line=False, line_number=22, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=3, argval='print', argrepr='print + NULL', offset=390, start_offset=390, starts_line=True, line_number=23, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=80, arg=4, argval='Here we go, here we go, here we go...', argrepr="'Here we go, here we go, here we go...'", offset=400, start_offset=400, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=1, argval=1, argrepr='', offset=402, start_offset=402, starts_line=False, line_number=23, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=410, start_offset=410, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), - Instruction(opname='POP_EXCEPT', opcode=29, arg=None, argval=None, argrepr='', offset=412, start_offset=412, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), - Instruction(opname='JUMP_BACKWARD_NO_INTERRUPT', opcode=74, arg=56, argval=304, argrepr='to L10', offset=414, start_offset=414, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), - Instruction(opname='RERAISE', opcode=101, arg=0, argval=0, argrepr='', offset=416, start_offset=416, starts_line=True, line_number=22, label=12, positions=None, cache_info=None), - Instruction(opname='COPY', opcode=58, arg=3, argval=3, argrepr='', offset=418, start_offset=418, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='POP_EXCEPT', opcode=29, arg=None, argval=None, argrepr='', offset=420, start_offset=420, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='RERAISE', opcode=101, arg=1, argval=1, argrepr='', offset=422, start_offset=422, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='PUSH_EXC_INFO', opcode=31, arg=None, argval=None, argrepr='', offset=424, start_offset=424, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='LOAD_GLOBAL', opcode=88, arg=3, argval='print', argrepr='print + NULL', offset=426, start_offset=426, starts_line=True, line_number=28, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), - Instruction(opname='LOAD_CONST', opcode=80, arg=5, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=436, start_offset=436, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='CALL', opcode=50, arg=1, argval=1, argrepr='', offset=438, start_offset=438, starts_line=False, line_number=28, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), - Instruction(opname='POP_TOP', opcode=30, arg=None, argval=None, argrepr='', offset=446, start_offset=446, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='RERAISE', opcode=101, arg=0, argval=0, argrepr='', offset=448, start_offset=448, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), - Instruction(opname='COPY', opcode=58, arg=3, argval=3, argrepr='', offset=450, start_offset=450, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='POP_EXCEPT', opcode=29, arg=None, argval=None, argrepr='', offset=452, start_offset=452, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), - Instruction(opname='RERAISE', opcode=101, arg=1, argval=1, argrepr='', offset=454, start_offset=454, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='FOR_ITER', opcode=69, arg=32, argval=92, argrepr='to L4', offset=24, start_offset=24, starts_line=False, line_number=3, label=1, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='STORE_FAST', opcode=109, arg=0, argval='i', argrepr='i', offset=28, start_offset=28, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=3, argval='print', argrepr='print + NULL', offset=30, start_offset=30, starts_line=True, line_number=4, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_FAST', opcode=83, arg=0, argval='i', argrepr='i', offset=40, start_offset=40, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=1, argval=1, argrepr='', offset=42, start_offset=42, starts_line=False, line_number=4, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=50, start_offset=50, starts_line=False, line_number=4, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=0, argval='i', argrepr='i', offset=52, start_offset=52, starts_line=True, line_number=5, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_SMALL_INT', opcode=91, arg=4, argval=4, argrepr='', offset=54, start_offset=54, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), + Instruction(opname='COMPARE_OP', opcode=56, arg=18, argval='<', argrepr='bool(<)', offset=56, start_offset=56, starts_line=False, line_number=5, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='POP_JUMP_IF_FALSE', opcode=97, arg=3, argval=70, argrepr='to L2', offset=60, start_offset=60, starts_line=False, line_number=5, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=64, start_offset=64, starts_line=False, line_number=5, label=None, positions=None, cache_info=None), + Instruction(opname='JUMP_BACKWARD', opcode=74, arg=23, argval=24, argrepr='to L1', offset=66, start_offset=66, starts_line=True, line_number=6, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='LOAD_FAST', opcode=83, arg=0, argval='i', argrepr='i', offset=70, start_offset=70, starts_line=True, line_number=7, label=2, positions=None, cache_info=None), + Instruction(opname='LOAD_SMALL_INT', opcode=91, arg=6, argval=6, argrepr='', offset=72, start_offset=72, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), + Instruction(opname='COMPARE_OP', opcode=56, arg=148, argval='>', argrepr='bool(>)', offset=74, start_offset=74, starts_line=False, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='POP_JUMP_IF_TRUE', opcode=100, arg=3, argval=88, argrepr='to L3', offset=78, start_offset=78, starts_line=False, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=82, start_offset=82, starts_line=False, line_number=7, label=None, positions=None, cache_info=None), + Instruction(opname='JUMP_BACKWARD', opcode=74, arg=32, argval=24, argrepr='to L1', offset=84, start_offset=84, starts_line=False, line_number=7, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=88, start_offset=88, starts_line=True, line_number=8, label=3, positions=None, cache_info=None), + Instruction(opname='JUMP_FORWARD', opcode=76, arg=13, argval=118, argrepr='to L5', offset=90, start_offset=90, starts_line=False, line_number=8, label=None, positions=None, cache_info=None), + Instruction(opname='END_FOR', opcode=9, arg=None, argval=None, argrepr='', offset=92, start_offset=92, starts_line=True, line_number=3, label=4, positions=None, cache_info=None), + Instruction(opname='POP_ITER', opcode=30, arg=None, argval=None, argrepr='', offset=94, start_offset=94, starts_line=False, line_number=3, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=3, argval='print', argrepr='print + NULL', offset=96, start_offset=96, starts_line=True, line_number=10, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_CONST', opcode=81, arg=0, argval='I can haz else clause?', argrepr="'I can haz else clause?'", offset=106, start_offset=106, starts_line=False, line_number=10, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=1, argval=1, argrepr='', offset=108, start_offset=108, starts_line=False, line_number=10, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=116, start_offset=116, starts_line=False, line_number=10, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST_CHECK', opcode=85, arg=0, argval='i', argrepr='i', offset=118, start_offset=118, starts_line=True, line_number=11, label=5, positions=None, cache_info=None), + Instruction(opname='TO_BOOL', opcode=39, arg=None, argval=None, argrepr='', offset=120, start_offset=120, starts_line=False, line_number=11, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_JUMP_IF_FALSE', opcode=97, arg=36, argval=204, argrepr='to L8', offset=128, start_offset=128, starts_line=False, line_number=11, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=132, start_offset=132, starts_line=False, line_number=11, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=3, argval='print', argrepr='print + NULL', offset=134, start_offset=134, starts_line=True, line_number=12, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_FAST', opcode=83, arg=0, argval='i', argrepr='i', offset=144, start_offset=144, starts_line=False, line_number=12, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=1, argval=1, argrepr='', offset=146, start_offset=146, starts_line=False, line_number=12, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=154, start_offset=154, starts_line=False, line_number=12, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=0, argval='i', argrepr='i', offset=156, start_offset=156, starts_line=True, line_number=13, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_SMALL_INT', opcode=91, arg=1, argval=1, argrepr='', offset=158, start_offset=158, starts_line=False, line_number=13, label=None, positions=None, cache_info=None), + Instruction(opname='BINARY_OP', opcode=44, arg=23, argval=23, argrepr='-=', offset=160, start_offset=160, starts_line=False, line_number=13, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='STORE_FAST', opcode=109, arg=0, argval='i', argrepr='i', offset=164, start_offset=164, starts_line=False, line_number=13, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=0, argval='i', argrepr='i', offset=166, start_offset=166, starts_line=True, line_number=14, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_SMALL_INT', opcode=91, arg=6, argval=6, argrepr='', offset=168, start_offset=168, starts_line=False, line_number=14, label=None, positions=None, cache_info=None), + Instruction(opname='COMPARE_OP', opcode=56, arg=148, argval='>', argrepr='bool(>)', offset=170, start_offset=170, starts_line=False, line_number=14, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='POP_JUMP_IF_FALSE', opcode=97, arg=3, argval=184, argrepr='to L6', offset=174, start_offset=174, starts_line=False, line_number=14, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=178, start_offset=178, starts_line=False, line_number=14, label=None, positions=None, cache_info=None), + Instruction(opname='JUMP_BACKWARD', opcode=74, arg=33, argval=118, argrepr='to L5', offset=180, start_offset=180, starts_line=True, line_number=15, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='LOAD_FAST', opcode=83, arg=0, argval='i', argrepr='i', offset=184, start_offset=184, starts_line=True, line_number=16, label=6, positions=None, cache_info=None), + Instruction(opname='LOAD_SMALL_INT', opcode=91, arg=4, argval=4, argrepr='', offset=186, start_offset=186, starts_line=False, line_number=16, label=None, positions=None, cache_info=None), + Instruction(opname='COMPARE_OP', opcode=56, arg=18, argval='<', argrepr='bool(<)', offset=188, start_offset=188, starts_line=False, line_number=16, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='POP_JUMP_IF_TRUE', opcode=100, arg=3, argval=202, argrepr='to L7', offset=192, start_offset=192, starts_line=False, line_number=16, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=196, start_offset=196, starts_line=False, line_number=16, label=None, positions=None, cache_info=None), + Instruction(opname='JUMP_BACKWARD', opcode=74, arg=42, argval=118, argrepr='to L5', offset=198, start_offset=198, starts_line=False, line_number=16, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='JUMP_FORWARD', opcode=76, arg=11, argval=226, argrepr='to L9', offset=202, start_offset=202, starts_line=True, line_number=17, label=7, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=3, argval='print', argrepr='print + NULL', offset=204, start_offset=204, starts_line=True, line_number=19, label=8, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_CONST', opcode=81, arg=1, argval='Who let lolcatz into this test suite?', argrepr="'Who let lolcatz into this test suite?'", offset=214, start_offset=214, starts_line=False, line_number=19, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=1, argval=1, argrepr='', offset=216, start_offset=216, starts_line=False, line_number=19, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=224, start_offset=224, starts_line=False, line_number=19, label=None, positions=None, cache_info=None), + Instruction(opname='NOP', opcode=27, arg=None, argval=None, argrepr='', offset=226, start_offset=226, starts_line=True, line_number=20, label=9, positions=None, cache_info=None), + Instruction(opname='LOAD_SMALL_INT', opcode=91, arg=1, argval=1, argrepr='', offset=228, start_offset=228, starts_line=True, line_number=21, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_SMALL_INT', opcode=91, arg=0, argval=0, argrepr='', offset=230, start_offset=230, starts_line=False, line_number=21, label=None, positions=None, cache_info=None), + Instruction(opname='BINARY_OP', opcode=44, arg=11, argval=11, argrepr='/', offset=232, start_offset=232, starts_line=False, line_number=21, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=236, start_offset=236, starts_line=False, line_number=21, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_FAST', opcode=83, arg=0, argval='i', argrepr='i', offset=238, start_offset=238, starts_line=True, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='COPY', opcode=59, arg=1, argval=1, argrepr='', offset=240, start_offset=240, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_SPECIAL', opcode=92, arg=1, argval=1, argrepr='__exit__', offset=242, start_offset=242, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='SWAP', opcode=114, arg=2, argval=2, argrepr='', offset=244, start_offset=244, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='SWAP', opcode=114, arg=3, argval=3, argrepr='', offset=246, start_offset=246, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_SPECIAL', opcode=92, arg=0, argval=0, argrepr='__enter__', offset=248, start_offset=248, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=0, argval=0, argrepr='', offset=250, start_offset=250, starts_line=False, line_number=25, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='STORE_FAST', opcode=109, arg=1, argval='dodgy', argrepr='dodgy', offset=258, start_offset=258, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=3, argval='print', argrepr='print + NULL', offset=260, start_offset=260, starts_line=True, line_number=26, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_CONST', opcode=81, arg=2, argval='Never reach this', argrepr="'Never reach this'", offset=270, start_offset=270, starts_line=False, line_number=26, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=1, argval=1, argrepr='', offset=272, start_offset=272, starts_line=False, line_number=26, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=280, start_offset=280, starts_line=False, line_number=26, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=3, argval=None, argrepr='None', offset=282, start_offset=282, starts_line=True, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=3, argval=None, argrepr='None', offset=284, start_offset=284, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=3, argval=None, argrepr='None', offset=286, start_offset=286, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=3, argval=3, argrepr='', offset=288, start_offset=288, starts_line=False, line_number=25, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=296, start_offset=296, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=3, argval='print', argrepr='print + NULL', offset=298, start_offset=298, starts_line=True, line_number=28, label=10, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_CONST', opcode=81, arg=5, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=308, start_offset=308, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=1, argval=1, argrepr='', offset=310, start_offset=310, starts_line=False, line_number=28, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=318, start_offset=318, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=3, argval=None, argrepr='None', offset=320, start_offset=320, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), + Instruction(opname='RETURN_VALUE', opcode=35, arg=None, argval=None, argrepr='', offset=322, start_offset=322, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), + Instruction(opname='PUSH_EXC_INFO', opcode=32, arg=None, argval=None, argrepr='', offset=324, start_offset=324, starts_line=True, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='WITH_EXCEPT_START', opcode=43, arg=None, argval=None, argrepr='', offset=326, start_offset=326, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='TO_BOOL', opcode=39, arg=None, argval=None, argrepr='', offset=328, start_offset=328, starts_line=False, line_number=25, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_JUMP_IF_TRUE', opcode=100, arg=2, argval=344, argrepr='to L11', offset=336, start_offset=336, starts_line=False, line_number=25, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=340, start_offset=340, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='RERAISE', opcode=102, arg=2, argval=2, argrepr='', offset=342, start_offset=342, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=344, start_offset=344, starts_line=False, line_number=25, label=11, positions=None, cache_info=None), + Instruction(opname='POP_EXCEPT', opcode=29, arg=None, argval=None, argrepr='', offset=346, start_offset=346, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=348, start_offset=348, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=350, start_offset=350, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=352, start_offset=352, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='JUMP_BACKWARD_NO_INTERRUPT', opcode=75, arg=29, argval=298, argrepr='to L10', offset=354, start_offset=354, starts_line=False, line_number=25, label=None, positions=None, cache_info=None), + Instruction(opname='COPY', opcode=59, arg=3, argval=3, argrepr='', offset=356, start_offset=356, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='POP_EXCEPT', opcode=29, arg=None, argval=None, argrepr='', offset=358, start_offset=358, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='RERAISE', opcode=102, arg=1, argval=1, argrepr='', offset=360, start_offset=360, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='PUSH_EXC_INFO', opcode=32, arg=None, argval=None, argrepr='', offset=362, start_offset=362, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=4, argval='ZeroDivisionError', argrepr='ZeroDivisionError', offset=364, start_offset=364, starts_line=True, line_number=22, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='CHECK_EXC_MATCH', opcode=5, arg=None, argval=None, argrepr='', offset=374, start_offset=374, starts_line=False, line_number=22, label=None, positions=None, cache_info=None), + Instruction(opname='POP_JUMP_IF_FALSE', opcode=97, arg=15, argval=410, argrepr='to L12', offset=376, start_offset=376, starts_line=False, line_number=22, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00')]), + Instruction(opname='NOT_TAKEN', opcode=28, arg=None, argval=None, argrepr='', offset=380, start_offset=380, starts_line=False, line_number=22, label=None, positions=None, cache_info=None), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=382, start_offset=382, starts_line=False, line_number=22, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=3, argval='print', argrepr='print + NULL', offset=384, start_offset=384, starts_line=True, line_number=23, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_CONST', opcode=81, arg=4, argval='Here we go, here we go, here we go...', argrepr="'Here we go, here we go, here we go...'", offset=394, start_offset=394, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=1, argval=1, argrepr='', offset=396, start_offset=396, starts_line=False, line_number=23, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=404, start_offset=404, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), + Instruction(opname='POP_EXCEPT', opcode=29, arg=None, argval=None, argrepr='', offset=406, start_offset=406, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), + Instruction(opname='JUMP_BACKWARD_NO_INTERRUPT', opcode=75, arg=56, argval=298, argrepr='to L10', offset=408, start_offset=408, starts_line=False, line_number=23, label=None, positions=None, cache_info=None), + Instruction(opname='RERAISE', opcode=102, arg=0, argval=0, argrepr='', offset=410, start_offset=410, starts_line=True, line_number=22, label=12, positions=None, cache_info=None), + Instruction(opname='COPY', opcode=59, arg=3, argval=3, argrepr='', offset=412, start_offset=412, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='POP_EXCEPT', opcode=29, arg=None, argval=None, argrepr='', offset=414, start_offset=414, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='RERAISE', opcode=102, arg=1, argval=1, argrepr='', offset=416, start_offset=416, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='PUSH_EXC_INFO', opcode=32, arg=None, argval=None, argrepr='', offset=418, start_offset=418, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='LOAD_GLOBAL', opcode=89, arg=3, argval='print', argrepr='print + NULL', offset=420, start_offset=420, starts_line=True, line_number=28, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('index', 1, b'\x00\x00'), ('module_keys_version', 1, b'\x00\x00'), ('builtin_keys_version', 1, b'\x00\x00')]), + Instruction(opname='LOAD_CONST', opcode=81, arg=5, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=430, start_offset=430, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), + Instruction(opname='CALL', opcode=51, arg=1, argval=1, argrepr='', offset=432, start_offset=432, starts_line=False, line_number=28, label=None, positions=None, cache_info=[('counter', 1, b'\x00\x00'), ('func_version', 2, b'\x00\x00\x00\x00')]), + Instruction(opname='POP_TOP', opcode=31, arg=None, argval=None, argrepr='', offset=440, start_offset=440, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), + Instruction(opname='RERAISE', opcode=102, arg=0, argval=0, argrepr='', offset=442, start_offset=442, starts_line=False, line_number=28, label=None, positions=None, cache_info=None), + Instruction(opname='COPY', opcode=59, arg=3, argval=3, argrepr='', offset=444, start_offset=444, starts_line=True, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='POP_EXCEPT', opcode=29, arg=None, argval=None, argrepr='', offset=446, start_offset=446, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), + Instruction(opname='RERAISE', opcode=102, arg=1, argval=1, argrepr='', offset=448, start_offset=448, starts_line=False, line_number=None, label=None, positions=None, cache_info=None), ] # One last piece of inspect fodder to check the default line number handling def simple(): pass expected_opinfo_simple = [ Instruction(opname='RESUME', opcode=149, arg=0, argval=0, argrepr='', offset=0, start_offset=0, starts_line=True, line_number=simple.__code__.co_firstlineno, label=None, positions=None), - Instruction(opname='LOAD_CONST', opcode=80, arg=0, argval=None, argrepr='None', offset=2, start_offset=2, starts_line=False, line_number=simple.__code__.co_firstlineno, label=None), - Instruction(opname='RETURN_VALUE', opcode=34, arg=None, argval=None, argrepr='', offset=4, start_offset=4, starts_line=False, line_number=simple.__code__.co_firstlineno, label=None), + Instruction(opname='LOAD_CONST', opcode=81, arg=0, argval=None, argrepr='None', offset=2, start_offset=2, starts_line=False, line_number=simple.__code__.co_firstlineno, label=None), + Instruction(opname='RETURN_VALUE', opcode=35, arg=None, argval=None, argrepr='', offset=4, start_offset=4, starts_line=False, line_number=simple.__code__.co_firstlineno, label=None), ] diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index abe9ef2e94409f..2deb35721576b8 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -810,6 +810,16 @@ def test_unicode_body_defaults_to_utf8_encoding(self): w4kgdGVzdGFiYwo= """)) + def test_string_payload_with_base64_cte(self): + msg = email.message_from_string(textwrap.dedent("""\ + Content-Transfer-Encoding: base64 + + SGVsbG8uIFRlc3Rpbmc= + """), policy=email.policy.default) + self.assertEqual(msg.get_payload(decode=True), b"Hello. Testing") + self.assertDefectsEqual(msg['content-transfer-encoding'].defects, []) + + # Test the email.encoders module class TestEncoders(unittest.TestCase): @@ -2352,6 +2362,40 @@ def test_missing_header_body_separator(self): self.assertDefectsEqual(msg.defects, [errors.MissingHeaderBodySeparatorDefect]) + def test_string_payload_with_extra_space_after_cte(self): + # https://github.com/python/cpython/issues/98188 + cte = "base64 " + msg = email.message_from_string(textwrap.dedent(f"""\ + Content-Transfer-Encoding: {cte} + + SGVsbG8uIFRlc3Rpbmc= + """), policy=email.policy.default) + self.assertEqual(msg.get_payload(decode=True), b"Hello. Testing") + self.assertDefectsEqual(msg['content-transfer-encoding'].defects, []) + + def test_string_payload_with_extra_text_after_cte(self): + msg = email.message_from_string(textwrap.dedent("""\ + Content-Transfer-Encoding: base64 some text + + SGVsbG8uIFRlc3Rpbmc= + """), policy=email.policy.default) + self.assertEqual(msg.get_payload(decode=True), b"Hello. Testing") + cte = msg['content-transfer-encoding'] + self.assertDefectsEqual(cte.defects, [email.errors.InvalidHeaderDefect]) + + def test_string_payload_with_extra_space_after_cte_compat32(self): + cte = "base64 " + msg = email.message_from_string(textwrap.dedent(f"""\ + Content-Transfer-Encoding: {cte} + + SGVsbG8uIFRlc3Rpbmc= + """), policy=email.policy.compat32) + pasted_cte = msg['content-transfer-encoding'] + self.assertEqual(pasted_cte, cte) + self.assertEqual(msg.get_payload(decode=True), b"Hello. Testing") + self.assertDefectsEqual(msg.defects, []) + + # Test RFC 2047 header encoding and decoding class TestRFC2047(TestEmailBase): diff --git a/Lib/test/test_email/test_headerregistry.py b/Lib/test/test_email/test_headerregistry.py index 4c0523f410332f..ff7a6da644d572 100644 --- a/Lib/test/test_email/test_headerregistry.py +++ b/Lib/test/test_email/test_headerregistry.py @@ -837,6 +837,11 @@ def cte_as_value(self, '7bit', [errors.InvalidHeaderDefect]), + 'extra_space_after_cte': ( + 'base64 ', + 'base64', + []), + } diff --git a/Lib/test/test_generated_cases.py b/Lib/test/test_generated_cases.py index 9c65e81dfe4be1..75cbd8dd94e9cb 100644 --- a/Lib/test/test_generated_cases.py +++ b/Lib/test/test_generated_cases.py @@ -1713,6 +1713,31 @@ def test_pop_dead_inputs_with_output(self): """ self.run_cases_test(input, output) + def test_no_escaping_calls_in_branching_macros(self): + + input = """ + inst(OP, ( -- )) { + DEOPT_IF(escaping_call()); + } + """ + with self.assertRaises(SyntaxError): + self.run_cases_test(input, "") + + input = """ + inst(OP, ( -- )) { + EXIT_IF(escaping_call()); + } + """ + with self.assertRaises(SyntaxError): + self.run_cases_test(input, "") + + input = """ + inst(OP, ( -- )) { + ERROR_IF(escaping_call(), error); + } + """ + with self.assertRaises(SyntaxError): + self.run_cases_test(input, "") class TestGeneratedAbstractCases(unittest.TestCase): def setUp(self) -> None: diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index 81c17b2731cc58..33e0161241e87e 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -3933,6 +3933,7 @@ def test_issue35928(self): self.assertEqual(res + f.readline(), 'foo\nbar\n') @unittest.skipUnless(hasattr(os, "pipe"), "requires os.pipe()") + @unittest.skipIf(support.is_emscripten, "Would be fixed by emscripten-core/emscripten#23306") def test_read_non_blocking(self): import os r, w = os.pipe() diff --git a/Lib/test/test_monitoring.py b/Lib/test/test_monitoring.py index 32b3a6ac049e28..43e3e56639db62 100644 --- a/Lib/test/test_monitoring.py +++ b/Lib/test/test_monitoring.py @@ -1589,11 +1589,11 @@ def whilefunc(n=0): ('branch right', 'whilefunc', 1, 3)]) self.check_events(func, recorders = BRANCH_OFFSET_RECORDERS, expected = [ - ('branch left', 'func', 28, 34), - ('branch right', 'func', 46, 60), - ('branch left', 'func', 28, 34), - ('branch left', 'func', 46, 52), - ('branch right', 'func', 28, 72)]) + ('branch left', 'func', 28, 32), + ('branch right', 'func', 44, 58), + ('branch left', 'func', 28, 32), + ('branch left', 'func', 44, 50), + ('branch right', 'func', 28, 70)]) def test_except_star(self): @@ -1658,6 +1658,88 @@ def foo(n=0): exit_loop]) +class TestBranchConsistency(MonitoringTestBase, unittest.TestCase): + + def check_branches(self, func, tool=TEST_TOOL, recorders=BRANCH_OFFSET_RECORDERS): + try: + self.assertEqual(sys.monitoring._all_events(), {}) + event_list = [] + all_events = 0 + for recorder in recorders: + ev = recorder.event_type + sys.monitoring.register_callback(tool, ev, recorder(event_list)) + all_events |= ev + sys.monitoring.set_local_events(tool, func.__code__, all_events) + func() + sys.monitoring.set_local_events(tool, func.__code__, 0) + for recorder in recorders: + sys.monitoring.register_callback(tool, recorder.event_type, None) + lefts = set() + rights = set() + for (src, left, right) in func.__code__.co_branches(): + lefts.add((src, left)) + rights.add((src, right)) + for event in event_list: + way, _, src, dest = event + if "left" in way: + self.assertIn((src, dest), lefts) + else: + self.assertIn("right", way) + self.assertIn((src, dest), rights) + finally: + sys.monitoring.set_local_events(tool, func.__code__, 0) + for recorder in recorders: + sys.monitoring.register_callback(tool, recorder.event_type, None) + + def test_simple(self): + + def func(): + x = 1 + for a in range(2): + if a: + x = 4 + else: + x = 6 + 7 + + self.check_branches(func) + + def whilefunc(n=0): + while n < 3: + n += 1 # line 2 + 3 + + self.check_branches(whilefunc) + + def test_except_star(self): + + class Foo: + def meth(self): + pass + + def func(): + try: + try: + raise KeyError + except* Exception as e: + f = Foo(); f.meth() + except KeyError: + pass + + + self.check_branches(func) + + def test4(self): + + def foo(n=0): + while n<4: + pass + n += 1 + return None + + self.check_branches(foo) + + class TestLoadSuperAttr(CheckEvents): RECORDERS = CallRecorder, LineRecorder, CRaiseRecorder, CReturnRecorder diff --git a/Lib/test/test_opcache.py b/Lib/test/test_opcache.py index 79f452f8068c7f..c7cd4c2e8a3146 100644 --- a/Lib/test/test_opcache.py +++ b/Lib/test/test_opcache.py @@ -1598,6 +1598,39 @@ def __getitem__(self, item): self.assert_specialized(binary_subscr_getitems, "BINARY_SUBSCR_GETITEM") self.assert_no_opcode(binary_subscr_getitems, "BINARY_SUBSCR") + @cpython_only + @requires_specialization_ft + def test_compare_op(self): + def compare_op_int(): + for _ in range(100): + a, b = 1, 2 + c = a == b + self.assertFalse(c) + + compare_op_int() + self.assert_specialized(compare_op_int, "COMPARE_OP_INT") + self.assert_no_opcode(compare_op_int, "COMPARE_OP") + + def compare_op_float(): + for _ in range(100): + a, b = 1.0, 2.0 + c = a == b + self.assertFalse(c) + + compare_op_float() + self.assert_specialized(compare_op_float, "COMPARE_OP_FLOAT") + self.assert_no_opcode(compare_op_float, "COMPARE_OP") + + def compare_op_str(): + for _ in range(100): + a, b = "spam", "ham" + c = a == b + self.assertFalse(c) + + compare_op_str() + self.assert_specialized(compare_op_str, "COMPARE_OP_STR") + self.assert_no_opcode(compare_op_str, "COMPARE_OP") + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py index d688a225538c11..d2c4dff3c9a0e5 100644 --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -4979,6 +4979,7 @@ def test_unpickable(self): self.assertRaises(TypeError, pickle.dumps, scandir_iter, filename) scandir_iter.close() + @unittest.skipIf(support.is_emscripten, "Fixed by emscripten-core/emscripten#23139, remove when next Emscripten release comes out") def check_entry(self, entry, name, is_dir, is_file, is_symlink): self.assertIsInstance(entry, os.DirEntry) self.assertEqual(entry.name, name) diff --git a/Lib/test/test_peepholer.py b/Lib/test/test_peepholer.py index c7da151dce3b37..b5b2b350e77a3b 100644 --- a/Lib/test/test_peepholer.py +++ b/Lib/test/test_peepholer.py @@ -1193,5 +1193,56 @@ def get_insts(lno1, lno2, op1, op2): ] self.cfg_optimization_test(insts, expected_insts, consts=list(range(5))) + def test_list_to_tuple_get_iter(self): + # for _ in (*foo, *bar) -> for _ in [*foo, *bar] + INTRINSIC_LIST_TO_TUPLE = 6 + insts = [ + ("BUILD_LIST", 0, 1), + ("LOAD_FAST", 0, 2), + ("LIST_EXTEND", 1, 3), + ("LOAD_FAST", 1, 4), + ("LIST_EXTEND", 1, 5), + ("CALL_INTRINSIC_1", INTRINSIC_LIST_TO_TUPLE, 6), + ("GET_ITER", None, 7), + top := self.Label(), + ("FOR_ITER", end := self.Label(), 8), + ("STORE_FAST", 2, 9), + ("JUMP", top, 10), + end, + ("END_FOR", None, 11), + ("POP_TOP", None, 12), + ("LOAD_CONST", 0, 13), + ("RETURN_VALUE", None, 14), + ] + expected_insts = [ + ("BUILD_LIST", 0, 1), + ("LOAD_FAST", 0, 2), + ("LIST_EXTEND", 1, 3), + ("LOAD_FAST", 1, 4), + ("LIST_EXTEND", 1, 5), + ("NOP", None, 6), # ("CALL_INTRINSIC_1", INTRINSIC_LIST_TO_TUPLE, 6), + ("GET_ITER", None, 7), + top := self.Label(), + ("FOR_ITER", end := self.Label(), 8), + ("STORE_FAST", 2, 9), + ("JUMP", top, 10), + end, + ("END_FOR", None, 11), + ("POP_TOP", None, 12), + ("LOAD_CONST", 0, 13), + ("RETURN_VALUE", None, 14), + ] + self.cfg_optimization_test(insts, expected_insts, consts=[None]) + + def test_list_to_tuple_get_iter_is_safe(self): + a, b = [], [] + for item in (*(items := [0, 1, 2, 3]),): + a.append(item) + b.append(items.pop()) + self.assertEqual(a, [0, 1, 2, 3]) + self.assertEqual(b, [3, 2, 1, 0]) + self.assertEqual(items, []) + + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_pydoc/test_pydoc.py b/Lib/test/test_pydoc/test_pydoc.py index c798b11f5aa56e..cec18aa9440c9e 100644 --- a/Lib/test/test_pydoc/test_pydoc.py +++ b/Lib/test/test_pydoc/test_pydoc.py @@ -4,6 +4,7 @@ import contextlib import importlib.util import inspect +import io import pydoc import py_compile import keyword @@ -899,6 +900,82 @@ def test_synopsis(self): synopsis = pydoc.synopsis(TESTFN, {}) self.assertEqual(synopsis, 'line 1: h\xe9') + def test_source_synopsis(self): + def check(source, expected, encoding=None): + if isinstance(source, str): + source_file = StringIO(source) + else: + source_file = io.TextIOWrapper(io.BytesIO(source), encoding=encoding) + with source_file: + result = pydoc.source_synopsis(source_file) + self.assertEqual(result, expected) + + check('"""Single line docstring."""', + 'Single line docstring.') + check('"""First line of docstring.\nSecond line.\nThird line."""', + 'First line of docstring.') + check('"""First line of docstring.\\nSecond line.\\nThird line."""', + 'First line of docstring.') + check('""" Whitespace around docstring. """', + 'Whitespace around docstring.') + check('import sys\n"""No docstring"""', + None) + check(' \n"""Docstring after empty line."""', + 'Docstring after empty line.') + check('# Comment\n"""Docstring after comment."""', + 'Docstring after comment.') + check(' # Indented comment\n"""Docstring after comment."""', + 'Docstring after comment.') + check('""""""', # Empty docstring + '') + check('', # Empty file + None) + check('"""Embedded\0null byte"""', + None) + check('"""Embedded null byte"""\0', + None) + check('"""Café and résumé."""', + 'Café and résumé.') + check("'''Triple single quotes'''", + 'Triple single quotes') + check('"Single double quotes"', + 'Single double quotes') + check("'Single single quotes'", + 'Single single quotes') + check('"""split\\\nline"""', + 'splitline') + check('"""Unrecognized escape \\sequence"""', + 'Unrecognized escape \\sequence') + check('"""Invalid escape seq\\uence"""', + None) + check('r"""Raw \\stri\\ng"""', + 'Raw \\stri\\ng') + check('b"""Bytes literal"""', + None) + check('f"""f-string"""', + None) + check('"""Concatenated""" \\\n"string" \'literals\'', + 'Concatenatedstringliterals') + check('"""String""" + """expression"""', + None) + check('("""In parentheses""")', + 'In parentheses') + check('("""Multiple lines """\n"""in parentheses""")', + 'Multiple lines in parentheses') + check('()', # tuple + None) + check(b'# coding: iso-8859-15\n"""\xa4uro sign"""', + '€uro sign', encoding='iso-8859-15') + check(b'"""\xa4"""', # Decoding error + None, encoding='utf-8') + + with tempfile.NamedTemporaryFile(mode='w+', encoding='utf-8') as temp_file: + temp_file.write('"""Real file test."""\n') + temp_file.flush() + temp_file.seek(0) + result = pydoc.source_synopsis(temp_file) + self.assertEqual(result, "Real file test.") + @requires_docstrings def test_synopsis_sourceless(self): os = import_helper.import_fresh_module('os') diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py index 1f18b1f09b5858..2d79d2ffede461 100644 --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -1587,6 +1587,7 @@ def test_copyfile_same_file(self): # the path as a directory, but on AIX the trailing slash has no effect # and is considered as a file. @unittest.skipIf(AIX, 'Not valid on AIX, see gh-92670') + @unittest.skipIf(support.is_emscripten, 'Fixed by emscripten-core/emscripten#23218, remove when next Emscripten release comes out') def test_copyfile_nonexistent_dir(self): # Issue 43219 src_dir = self.mkdtemp() diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index c16ef3f96f9a21..9863f3ffe97656 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1325,8 +1325,7 @@ def test_load_verify_cadata(self): def test_load_dh_params(self): ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) ctx.load_dh_params(DHFILE) - if os.name != 'nt': - ctx.load_dh_params(BYTES_DHFILE) + ctx.load_dh_params(BYTES_DHFILE) self.assertRaises(TypeError, ctx.load_dh_params) self.assertRaises(TypeError, ctx.load_dh_params, None) with self.assertRaises(FileNotFoundError) as cm: diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py index fa08dc6a25b0ea..f3724ce6d4d15a 100644 --- a/Lib/test/test_stable_abi_ctypes.py +++ b/Lib/test/test_stable_abi_ctypes.py @@ -901,6 +901,8 @@ def test_windows_feature_macros(self): "Py_MakePendingCalls", "Py_NewInterpreter", "Py_NewRef", + "Py_PACK_FULL_VERSION", + "Py_PACK_VERSION", "Py_REFCNT", "Py_ReprEnter", "Py_ReprLeave", diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py index 54d329a15d4d25..2549b6b35adc29 100644 --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -3800,6 +3800,7 @@ def test_absolute_hardlink(self): "'parent' is a link to an absolute path") @symlink_test + @unittest.skipIf(support.is_emscripten, "Fixed by emscripten-core/emscripten#23136, remove when next Emscripten release comes out") def test_sly_relative0(self): # Inspired by 'relative0' in jwilk/traversal-archives with ArchiveMaker() as arc: diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index ef3cfc9517085e..c51ee763890af2 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -45,6 +45,7 @@ import textwrap import typing import weakref +import warnings import types from test.support import captured_stderr, cpython_only, infinite_recursion, requires_docstrings, import_helper, run_code @@ -5182,6 +5183,18 @@ class C(B[int]): x = pickle.loads(z) self.assertEqual(s, x) + # Test ParamSpec args and kwargs + global PP + PP = ParamSpec('PP') + for thing in [PP.args, PP.kwargs]: + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(thing=thing, proto=proto): + self.assertEqual( + pickle.loads(pickle.dumps(thing, proto)), + thing, + ) + del PP + def test_copy_and_deepcopy(self): T = TypeVar('T') class Node(Generic[T]): ... @@ -7140,6 +7153,25 @@ class C: self.assertEqual(get_type_hints(C, format=annotationlib.Format.STRING), {'x': 'undefined'}) + def test_get_type_hints_format_function(self): + def func(x: undefined) -> undefined: ... + + # VALUE + with self.assertRaises(NameError): + get_type_hints(func) + with self.assertRaises(NameError): + get_type_hints(func, format=annotationlib.Format.VALUE) + + # FORWARDREF + self.assertEqual( + get_type_hints(func, format=annotationlib.Format.FORWARDREF), + {'x': ForwardRef('undefined'), 'return': ForwardRef('undefined')}, + ) + + # STRING + self.assertEqual(get_type_hints(func, format=annotationlib.Format.STRING), + {'x': 'undefined', 'return': 'undefined'}) + class GetUtilitiesTestCase(TestCase): def test_get_origin(self): @@ -7242,6 +7274,51 @@ class C(Generic[T]): pass self.assertEqual(get_args(Unpack[tuple[Unpack[Ts]]]), (tuple[Unpack[Ts]],)) +class EvaluateForwardRefTests(BaseTestCase): + def test_evaluate_forward_ref(self): + int_ref = ForwardRef('int') + missing = ForwardRef('missing') + self.assertIs( + typing.evaluate_forward_ref(int_ref, type_params=()), + int, + ) + self.assertIs( + typing.evaluate_forward_ref( + int_ref, type_params=(), format=annotationlib.Format.FORWARDREF, + ), + int, + ) + self.assertIs( + typing.evaluate_forward_ref( + missing, type_params=(), format=annotationlib.Format.FORWARDREF, + ), + missing, + ) + self.assertEqual( + typing.evaluate_forward_ref( + int_ref, type_params=(), format=annotationlib.Format.STRING, + ), + 'int', + ) + + def test_evaluate_forward_ref_no_type_params(self): + ref = ForwardRef('int') + with self.assertWarnsRegex( + DeprecationWarning, + ( + "Failing to pass a value to the 'type_params' parameter " + "of 'typing.evaluate_forward_ref' is deprecated, " + "as it leads to incorrect behaviour" + ), + ): + typing.evaluate_forward_ref(ref) + + # No warnings when `type_params` is passed: + with warnings.catch_warnings(record=True) as w: + typing.evaluate_forward_ref(ref, type_params=()) + self.assertEqual(w, []) + + class CollectionsAbcTests(BaseTestCase): def test_hashable(self): @@ -8912,13 +8989,13 @@ class Child1(Base1): self.assertEqual(Child1.__mutable_keys__, frozenset({'b'})) class Base2(TypedDict): - a: ReadOnly[int] + a: int class Child2(Base2): - b: str + b: ReadOnly[str] - self.assertEqual(Child1.__readonly_keys__, frozenset({'a'})) - self.assertEqual(Child1.__mutable_keys__, frozenset({'b'})) + self.assertEqual(Child2.__readonly_keys__, frozenset({'b'})) + self.assertEqual(Child2.__mutable_keys__, frozenset({'a'})) def test_cannot_make_mutable_key_readonly(self): class Base(TypedDict): @@ -10129,6 +10206,18 @@ def test_valid_uses(self): self.assertEqual(C4.__args__, (Concatenate[int, T, P], T)) self.assertEqual(C4.__parameters__, (T, P)) + def test_invalid_uses(self): + with self.assertRaisesRegex(TypeError, 'Concatenate of no types'): + Concatenate[()] + with self.assertRaisesRegex( + TypeError, + ( + 'The last parameter to Concatenate should be a ' + 'ParamSpec variable or ellipsis' + ), + ): + Concatenate[int] + def test_var_substitution(self): T = TypeVar('T') P = ParamSpec('P') diff --git a/Lib/test/test_xml_dom_xmlbuilder.py b/Lib/test/test_xml_dom_xmlbuilder.py new file mode 100644 index 00000000000000..5f5f2eb328df9f --- /dev/null +++ b/Lib/test/test_xml_dom_xmlbuilder.py @@ -0,0 +1,88 @@ +import io +import unittest +from http import client +from test.test_httplib import FakeSocket +from unittest import mock +from xml.dom import getDOMImplementation, minidom, xmlbuilder + +SMALL_SAMPLE = b""" + + +Introduction to XSL +
+

A. Namespace

+""" + + +class XMLBuilderTest(unittest.TestCase): + def test_entity_resolver(self): + body = ( + b"HTTP/1.1 200 OK\r\nContent-Type: text/xml; charset=utf-8\r\n\r\n" + + SMALL_SAMPLE + ) + + sock = FakeSocket(body) + response = client.HTTPResponse(sock) + response.begin() + attrs = {"open.return_value": response} + opener = mock.Mock(**attrs) + + resolver = xmlbuilder.DOMEntityResolver() + + with mock.patch("urllib.request.build_opener") as mock_build: + mock_build.return_value = opener + source = resolver.resolveEntity(None, "http://example.com/2000/svg") + + self.assertIsInstance(source, xmlbuilder.DOMInputSource) + self.assertIsNone(source.publicId) + self.assertEqual(source.systemId, "http://example.com/2000/svg") + self.assertEqual(source.baseURI, "http://example.com/2000/") + self.assertEqual(source.encoding, "utf-8") + self.assertIs(source.byteStream, response) + + self.assertIsNone(source.characterStream) + self.assertIsNone(source.stringData) + + def test_builder(self): + imp = getDOMImplementation() + self.assertIsInstance(imp, xmlbuilder.DOMImplementationLS) + + builder = imp.createDOMBuilder(imp.MODE_SYNCHRONOUS, None) + self.assertIsInstance(builder, xmlbuilder.DOMBuilder) + + def test_parse_uri(self): + body = ( + b"HTTP/1.1 200 OK\r\nContent-Type: text/xml; charset=utf-8\r\n\r\n" + + SMALL_SAMPLE + ) + + sock = FakeSocket(body) + response = client.HTTPResponse(sock) + response.begin() + attrs = {"open.return_value": response} + opener = mock.Mock(**attrs) + + with mock.patch("urllib.request.build_opener") as mock_build: + mock_build.return_value = opener + + imp = getDOMImplementation() + builder = imp.createDOMBuilder(imp.MODE_SYNCHRONOUS, None) + document = builder.parseURI("http://example.com/2000/svg") + + self.assertIsInstance(document, minidom.Document) + self.assertEqual(len(document.childNodes), 1) + + def test_parse_with_systemId(self): + response = io.BytesIO(SMALL_SAMPLE) + + with mock.patch("urllib.request.urlopen") as mock_open: + mock_open.return_value = response + + imp = getDOMImplementation() + source = imp.createDOMInputSource() + builder = imp.createDOMBuilder(imp.MODE_SYNCHRONOUS, None) + source.systemId = "http://example.com/2000/svg" + document = builder.parse(source) + + self.assertIsInstance(document, minidom.Document) + self.assertEqual(len(document.childNodes), 1) diff --git a/Lib/test/test_zipfile/test_core.py b/Lib/test/test_zipfile/test_core.py index 49f39b9337df85..79e7337606b4bc 100644 --- a/Lib/test/test_zipfile/test_core.py +++ b/Lib/test/test_zipfile/test_core.py @@ -22,7 +22,8 @@ from test.support import script_helper from test.support import ( findfile, requires_zlib, requires_bz2, requires_lzma, - captured_stdout, captured_stderr, requires_subprocess + captured_stdout, captured_stderr, requires_subprocess, + is_emscripten ) from test.support.os_helper import ( TESTFN, unlink, rmtree, temp_dir, temp_cwd, fd_count, FakePath @@ -622,6 +623,7 @@ def test_write_to_readonly(self): with self.assertRaises(ValueError): zipfp.open(TESTFN, mode='w') + @unittest.skipIf(is_emscripten, "Fixed by emscripten-core/emscripten#23310") def test_add_file_before_1980(self): # Set atime and mtime to 1970-01-01 os.utime(TESTFN, (0, 0)) diff --git a/Lib/test/test_zipimport.py b/Lib/test/test_zipimport.py index 1f288c8b45d589..65f8b17f2f88c0 100644 --- a/Lib/test/test_zipimport.py +++ b/Lib/test/test_zipimport.py @@ -1038,6 +1038,7 @@ def testEmptyFile(self): self.assertZipFailure(TESTMOD) @unittest.skipIf(support.is_wasi, "mode 000 not supported.") + @unittest.skipIf(support.is_emscripten, "Fixed by emscripten-core/emscripten#23137, remove when next Emscripten release comes out") def testFileUnreadable(self): os_helper.unlink(TESTMOD) fd = os.open(TESTMOD, os.O_CREAT, 000) diff --git a/Lib/tokenize.py b/Lib/tokenize.py index 7ece4e9b70d31b..1a60fd32a77ea4 100644 --- a/Lib/tokenize.py +++ b/Lib/tokenize.py @@ -318,16 +318,10 @@ def untokenize(iterable): with at least two elements, a token number and token value. If only two tokens are passed, the resulting output is poor. - Round-trip invariant for full input: - Untokenized source will match input source exactly - - Round-trip invariant for limited input: - # Output bytes will tokenize back to the input - t1 = [tok[:2] for tok in tokenize(f.readline)] - newcode = untokenize(t1) - readline = BytesIO(newcode).readline - t2 = [tok[:2] for tok in tokenize(readline)] - assert t1 == t2 + The result is guaranteed to tokenize back to match the input so + that the conversion is lossless and round-trips are assured. + The guarantee applies only to the token type and token string as + the spacing between tokens (column positions) may change. """ ut = Untokenizer() out = ut.untokenize(iterable) diff --git a/Lib/typing.py b/Lib/typing.py index e69b485422cbd2..66570db7a5bd74 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -1024,7 +1024,7 @@ def evaluate_forward_ref( owner=None, globals=None, locals=None, - type_params=None, + type_params=_sentinel, format=annotationlib.Format.VALUE, _recursive_guard=frozenset(), ): diff --git a/Lib/xml/dom/xmlbuilder.py b/Lib/xml/dom/xmlbuilder.py index 8a200263497b89..a8852625a2f9a2 100644 --- a/Lib/xml/dom/xmlbuilder.py +++ b/Lib/xml/dom/xmlbuilder.py @@ -189,7 +189,7 @@ def parse(self, input): options.filter = self.filter options.errorHandler = self.errorHandler fp = input.byteStream - if fp is None and options.systemId: + if fp is None and input.systemId: import urllib.request fp = urllib.request.urlopen(input.systemId) return self._parse_bytestream(fp, options) @@ -247,10 +247,12 @@ def _create_opener(self): def _guess_media_encoding(self, source): info = source.byteStream.info() - if "Content-Type" in info: - for param in info.getplist(): - if param.startswith("charset="): - return param.split("=", 1)[1].lower() + # import email.message + # assert isinstance(info, email.message.Message) + charset = info.get_param('charset') + if charset is not None: + return charset.lower() + return None class DOMInputSource(object): diff --git a/Misc/ACKS b/Misc/ACKS index c6e53317b37d78..deda334bee7417 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1038,6 +1038,7 @@ Erno Kuusela Kabir Kwatra Ross Lagerwall Cameron Laird +Filipe Laíns Loïc Lajeanne Alexander Lakeev David Lam @@ -1129,6 +1130,7 @@ Gregor Lingl Everett Lipman Mirko Liss Alexander Liu +Hui Liu Yuan Liu Nick Lockwood Stephanie Lockwood diff --git a/Misc/NEWS.d/next/Build/2025-01-04-22-39-10.gh-issue-128472.Wt5E6M.rst b/Misc/NEWS.d/next/Build/2025-01-04-22-39-10.gh-issue-128472.Wt5E6M.rst new file mode 100644 index 00000000000000..c6233e1f2d8693 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2025-01-04-22-39-10.gh-issue-128472.Wt5E6M.rst @@ -0,0 +1,2 @@ +Skip BOLT optimization of functions using computed gotos, fixing errors on +build with LLVM 19. diff --git a/Misc/NEWS.d/next/C_API/2024-12-11-13-01-26.gh-issue-127350.uEBZZ4.rst b/Misc/NEWS.d/next/C_API/2024-12-11-13-01-26.gh-issue-127350.uEBZZ4.rst new file mode 100644 index 00000000000000..d1b528c673442f --- /dev/null +++ b/Misc/NEWS.d/next/C_API/2024-12-11-13-01-26.gh-issue-127350.uEBZZ4.rst @@ -0,0 +1,5 @@ +Add :c:func:`Py_fopen` function to open a file. Similar to the :c:func:`!fopen` +function, but the *path* parameter is a Python object and an exception is set +on error. Add also :c:func:`Py_fclose` function to close a file, function +needed for Windows support. +Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C_API/2025-01-08-13-13-18.gh-issue-128629.gSmzyl.rst b/Misc/NEWS.d/next/C_API/2025-01-08-13-13-18.gh-issue-128629.gSmzyl.rst new file mode 100644 index 00000000000000..cde5bf38f754b6 --- /dev/null +++ b/Misc/NEWS.d/next/C_API/2025-01-08-13-13-18.gh-issue-128629.gSmzyl.rst @@ -0,0 +1,2 @@ +Add macros :c:func:`Py_PACK_VERSION` and :c:func:`Py_PACK_FULL_VERSION` for +bit-packing Python version numbers. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2024-12-02-18-15-37.gh-issue-126862.fdIK7T.rst b/Misc/NEWS.d/next/Core_and_Builtins/2024-12-02-18-15-37.gh-issue-126862.fdIK7T.rst new file mode 100644 index 00000000000000..d930c2963e3632 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2024-12-02-18-15-37.gh-issue-126862.fdIK7T.rst @@ -0,0 +1,2 @@ +Fix a possible overflow when a class inherits from an absurd number of +super-classes. Reported by Valery Fedorenko. Patch by Bénédikt Tran. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2024-12-11-14-32-22.gh-issue-127809.0W8khe.rst b/Misc/NEWS.d/next/Core_and_Builtins/2024-12-11-14-32-22.gh-issue-127809.0W8khe.rst new file mode 100644 index 00000000000000..19c8cc6e99c8c5 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2024-12-11-14-32-22.gh-issue-127809.0W8khe.rst @@ -0,0 +1,2 @@ +Fix an issue where the experimental JIT may infer an incorrect result type +for exponentiation (``**`` and ``**=``), leading to bugs or crashes. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2024-12-15-19-51-54.gh-issue-127970.vdUp-y.rst b/Misc/NEWS.d/next/Core_and_Builtins/2024-12-15-19-51-54.gh-issue-127970.vdUp-y.rst new file mode 100644 index 00000000000000..e4dc7b5fe032d6 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2024-12-15-19-51-54.gh-issue-127970.vdUp-y.rst @@ -0,0 +1,6 @@ +We now use the location of the ``libpython`` runtime library used in the current +proccess to determine :data:`sys.base_prefix` on all platforms implementing the +`dladdr `_ +function defined by the UNIX standard — this includes Linux, Android, macOS, +iOS, FreeBSD, etc. This was already the case on Windows and macOS Framework +builds. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2024-12-15-21-11-26.gh-issue-66409.wv109z.rst b/Misc/NEWS.d/next/Core_and_Builtins/2024-12-15-21-11-26.gh-issue-66409.wv109z.rst new file mode 100644 index 00000000000000..0d70ad06c97968 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2024-12-15-21-11-26.gh-issue-66409.wv109z.rst @@ -0,0 +1,3 @@ +During the :ref:`path initialization `, we now check if +``base_exec_prefix`` is the same as ``base_prefix`` before falling back to +searching the Python interpreter directory. diff --git a/Misc/NEWS.d/next/Library/2024-11-24-14-53-35.gh-issue-127196.8CBkUa.rst b/Misc/NEWS.d/next/Library/2024-11-24-14-53-35.gh-issue-127196.8CBkUa.rst new file mode 100644 index 00000000000000..471f64d185deab --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-11-24-14-53-35.gh-issue-127196.8CBkUa.rst @@ -0,0 +1,2 @@ +Fix crash when dict with keys in invalid encoding were passed to several +functions in ``_interpreters`` module. diff --git a/Misc/NEWS.d/next/Library/2024-12-03-14-45-16.gh-issue-98188.GX9i2b.rst b/Misc/NEWS.d/next/Library/2024-12-03-14-45-16.gh-issue-98188.GX9i2b.rst new file mode 100644 index 00000000000000..30ab8cfc3f0bc6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-12-03-14-45-16.gh-issue-98188.GX9i2b.rst @@ -0,0 +1,3 @@ +Fix an issue in :meth:`email.message.Message.get_payload` where data +cannot be decoded if the Content Transfer Encoding mechanism contains +trailing whitespaces or additional junk text. Patch by Hui Liu. diff --git a/Misc/NEWS.d/next/Library/2024-12-17-15-23-40.gh-issue-41872.31LjKY.rst b/Misc/NEWS.d/next/Library/2024-12-17-15-23-40.gh-issue-41872.31LjKY.rst new file mode 100644 index 00000000000000..b807dcb284c248 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-12-17-15-23-40.gh-issue-41872.31LjKY.rst @@ -0,0 +1,3 @@ +Fix quick extraction of module docstrings from a file in :mod:`pydoc`. +It now supports docstrings with single quotes, escape sequences, +raw string literals, and other Python syntax. diff --git a/Misc/NEWS.d/next/Library/2024-12-27-16-28-57.gh-issue-128302.2GMvyl.rst b/Misc/NEWS.d/next/Library/2024-12-27-16-28-57.gh-issue-128302.2GMvyl.rst new file mode 100644 index 00000000000000..56e2fe6f85f4bf --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-12-27-16-28-57.gh-issue-128302.2GMvyl.rst @@ -0,0 +1,3 @@ +Allow :meth:`!xml.dom.xmlbuilder.DOMParser.parse` to correctly handle +:class:`!xml.dom.xmlbuilder.DOMInputSource` instances that only have a +:attr:`!systemId` attribute set. diff --git a/Misc/NEWS.d/next/Library/2024-12-29-13-49-46.gh-issue-128302.psRpPN.rst b/Misc/NEWS.d/next/Library/2024-12-29-13-49-46.gh-issue-128302.psRpPN.rst new file mode 100644 index 00000000000000..98c07297b06f8a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-12-29-13-49-46.gh-issue-128302.psRpPN.rst @@ -0,0 +1,2 @@ +Fix :meth:`!xml.dom.xmlbuilder.DOMEntityResolver.resolveEntity`, which was +broken by the Python 3.0 transition. diff --git a/Misc/NEWS.d/next/Library/2025-01-05-11-46-14.gh-issue-128340.gKI0uU.rst b/Misc/NEWS.d/next/Library/2025-01-05-11-46-14.gh-issue-128340.gKI0uU.rst new file mode 100644 index 00000000000000..790400a19f334b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2025-01-05-11-46-14.gh-issue-128340.gKI0uU.rst @@ -0,0 +1 @@ +Add internal thread safe handle to be used in :meth:`asyncio.loop.call_soon_threadsafe` for thread safe cancellation. diff --git a/Misc/NEWS.d/next/Library/2025-01-06-18-41-08.gh-issue-128552.fV-f8j.rst b/Misc/NEWS.d/next/Library/2025-01-06-18-41-08.gh-issue-128552.fV-f8j.rst new file mode 100644 index 00000000000000..83816f775da9c5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2025-01-06-18-41-08.gh-issue-128552.fV-f8j.rst @@ -0,0 +1 @@ +Fix cyclic garbage introduced by :meth:`asyncio.loop.create_task` and :meth:`asyncio.TaskGroup.create_task` holding a reference to the created task if it is eager. diff --git a/Misc/NEWS.d/next/Library/2025-01-06-21-35-00.gh-issue-128559.6fxcDM.rst b/Misc/NEWS.d/next/Library/2025-01-06-21-35-00.gh-issue-128559.6fxcDM.rst new file mode 100644 index 00000000000000..7f9380de17761b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2025-01-06-21-35-00.gh-issue-128559.6fxcDM.rst @@ -0,0 +1 @@ +Improved import time of :mod:`asyncio`. diff --git a/Misc/NEWS.d/next/Library/2025-01-09-12-06-52.gh-issue-128661.ixx_0z.rst b/Misc/NEWS.d/next/Library/2025-01-09-12-06-52.gh-issue-128661.ixx_0z.rst new file mode 100644 index 00000000000000..6c52b3dcc0ed00 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2025-01-09-12-06-52.gh-issue-128661.ixx_0z.rst @@ -0,0 +1,2 @@ +Fixes :func:`typing.evaluate_forward_ref` not showing deprecation when +``type_params`` arg is not passed. diff --git a/Misc/NEWS.d/next/macOS/2024-12-22-08-54-30.gh-issue-127592.iyuFCC.rst b/Misc/NEWS.d/next/macOS/2024-12-22-08-54-30.gh-issue-127592.iyuFCC.rst new file mode 100644 index 00000000000000..dfe659294c712e --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2024-12-22-08-54-30.gh-issue-127592.iyuFCC.rst @@ -0,0 +1,2 @@ +Usage of the unified Apple System Log APIs was disabled when the minimum +macOS version is earlier than 10.12. diff --git a/Misc/stable_abi.toml b/Misc/stable_abi.toml index f9e51f0683c965..276526a1b6908e 100644 --- a/Misc/stable_abi.toml +++ b/Misc/stable_abi.toml @@ -2540,3 +2540,7 @@ added = '3.14' [function.PyType_Freeze] added = '3.14' +[function.Py_PACK_FULL_VERSION] + added = '3.14' +[function.Py_PACK_VERSION] + added = '3.14' diff --git a/Modules/Setup.stdlib.in b/Modules/Setup.stdlib.in index 52c0f883d383db..b7357f41768a2f 100644 --- a/Modules/Setup.stdlib.in +++ b/Modules/Setup.stdlib.in @@ -163,7 +163,7 @@ @MODULE__TESTBUFFER_TRUE@_testbuffer _testbuffer.c @MODULE__TESTINTERNALCAPI_TRUE@_testinternalcapi _testinternalcapi.c _testinternalcapi/test_lock.c _testinternalcapi/pytime.c _testinternalcapi/set.c _testinternalcapi/test_critical_sections.c @MODULE__TESTCAPI_TRUE@_testcapi _testcapimodule.c _testcapi/vectorcall.c _testcapi/heaptype.c _testcapi/abstract.c _testcapi/unicode.c _testcapi/dict.c _testcapi/set.c _testcapi/list.c _testcapi/tuple.c _testcapi/getargs.c _testcapi/datetime.c _testcapi/docstring.c _testcapi/mem.c _testcapi/watchers.c _testcapi/long.c _testcapi/float.c _testcapi/complex.c _testcapi/numbers.c _testcapi/structmember.c _testcapi/exceptions.c _testcapi/code.c _testcapi/buffer.c _testcapi/pyatomic.c _testcapi/run.c _testcapi/file.c _testcapi/codec.c _testcapi/immortal.c _testcapi/gc.c _testcapi/hash.c _testcapi/time.c _testcapi/bytes.c _testcapi/object.c _testcapi/monitoring.c _testcapi/config.c -@MODULE__TESTLIMITEDCAPI_TRUE@_testlimitedcapi _testlimitedcapi.c _testlimitedcapi/abstract.c _testlimitedcapi/bytearray.c _testlimitedcapi/bytes.c _testlimitedcapi/codec.c _testlimitedcapi/complex.c _testlimitedcapi/dict.c _testlimitedcapi/eval.c _testlimitedcapi/float.c _testlimitedcapi/heaptype_relative.c _testlimitedcapi/list.c _testlimitedcapi/long.c _testlimitedcapi/object.c _testlimitedcapi/pyos.c _testlimitedcapi/set.c _testlimitedcapi/sys.c _testlimitedcapi/tuple.c _testlimitedcapi/unicode.c _testlimitedcapi/vectorcall_limited.c +@MODULE__TESTLIMITEDCAPI_TRUE@_testlimitedcapi _testlimitedcapi.c _testlimitedcapi/abstract.c _testlimitedcapi/bytearray.c _testlimitedcapi/bytes.c _testlimitedcapi/codec.c _testlimitedcapi/complex.c _testlimitedcapi/dict.c _testlimitedcapi/eval.c _testlimitedcapi/float.c _testlimitedcapi/heaptype_relative.c _testlimitedcapi/list.c _testlimitedcapi/long.c _testlimitedcapi/object.c _testlimitedcapi/pyos.c _testlimitedcapi/set.c _testlimitedcapi/sys.c _testlimitedcapi/tuple.c _testlimitedcapi/unicode.c _testlimitedcapi/vectorcall_limited.c _testlimitedcapi/version.c @MODULE__TESTCLINIC_TRUE@_testclinic _testclinic.c @MODULE__TESTCLINIC_LIMITED_TRUE@_testclinic_limited _testclinic_limited.c diff --git a/Modules/_abc.c b/Modules/_abc.c index 4f4b24b035db4a..d6a953b336025d 100644 --- a/Modules/_abc.c +++ b/Modules/_abc.c @@ -67,6 +67,8 @@ typedef struct { uint64_t _abc_negative_cache_version; } _abc_data; +#define _abc_data_CAST(op) ((_abc_data *)(op)) + static inline uint64_t get_cache_version(_abc_data *impl) { @@ -88,8 +90,9 @@ set_cache_version(_abc_data *impl, uint64_t version) } static int -abc_data_traverse(_abc_data *self, visitproc visit, void *arg) +abc_data_traverse(PyObject *op, visitproc visit, void *arg) { + _abc_data *self = _abc_data_CAST(op); Py_VISIT(Py_TYPE(self)); Py_VISIT(self->_abc_registry); Py_VISIT(self->_abc_cache); @@ -98,8 +101,9 @@ abc_data_traverse(_abc_data *self, visitproc visit, void *arg) } static int -abc_data_clear(_abc_data *self) +abc_data_clear(PyObject *op) { + _abc_data *self = _abc_data_CAST(op); Py_CLEAR(self->_abc_registry); Py_CLEAR(self->_abc_cache); Py_CLEAR(self->_abc_negative_cache); @@ -107,7 +111,7 @@ abc_data_clear(_abc_data *self) } static void -abc_data_dealloc(_abc_data *self) +abc_data_dealloc(PyObject *self) { PyObject_GC_UnTrack(self); PyTypeObject *tp = Py_TYPE(self); @@ -212,7 +216,7 @@ _destroy(PyObject *setweakref, PyObject *objweakref) } static PyMethodDef _destroy_def = { - "_destroy", (PyCFunction) _destroy, METH_O + "_destroy", _destroy, METH_O }; static int @@ -964,7 +968,7 @@ _abcmodule_clear(PyObject *module) static void _abcmodule_free(void *module) { - _abcmodule_clear((PyObject *)module); + (void)_abcmodule_clear((PyObject *)module); } static PyModuleDef_Slot _abcmodule_slots[] = { diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c index b8b184af04a7cb..48f0ef95934fa4 100644 --- a/Modules/_asynciomodule.c +++ b/Modules/_asynciomodule.c @@ -3772,11 +3772,20 @@ _asyncio_all_tasks_impl(PyObject *module, PyObject *loop) llist_for_each_safe(node, &state->asyncio_tasks_head) { TaskObj *task = llist_data(node, TaskObj, task_node); - if (PyList_Append(tasks, (PyObject *)task) < 0) { - Py_DECREF(tasks); - Py_DECREF(loop); - err = 1; - break; + // The linked list holds borrowed references to task + // as such it is possible that the task is concurrently + // deallocated while added to this list. + // To protect against concurrent deallocations, + // we first try to incref the task which would fail + // if it is concurrently getting deallocated in another thread, + // otherwise it gets added to the list. + if (_Py_TryIncref((PyObject *)task)) { + if (_PyList_AppendTakeRef((PyListObject *)tasks, (PyObject *)task) < 0) { + Py_DECREF(tasks); + Py_DECREF(loop); + err = 1; + break; + } } } ASYNCIO_STATE_UNLOCK(state); diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c index c564813036e504..0def463c7d8b9e 100644 --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -30,7 +30,6 @@ #endif #include -#include "pycore_long.h" // _PyLong_IsZero() #include "pycore_pystate.h" // _PyThreadState_GET() #include "pycore_typeobject.h" #include "complexobject.h" @@ -2323,38 +2322,42 @@ static PyObject * dec_from_long(decimal_state *state, PyTypeObject *type, PyObject *v, const mpd_context_t *ctx, uint32_t *status) { - PyObject *dec; - PyLongObject *l = (PyLongObject *)v; + PyObject *dec = PyDecType_New(state, type); - dec = PyDecType_New(state, type); if (dec == NULL) { return NULL; } - if (_PyLong_IsZero(l)) { - _dec_settriple(dec, MPD_POS, 0, 0); - return dec; - } - - uint8_t sign = _PyLong_IsNegative(l) ? MPD_NEG : MPD_POS; + PyLongExport export_long; - if (_PyLong_IsCompact(l)) { - _dec_settriple(dec, sign, l->long_value.ob_digit[0], 0); - mpd_qfinalize(MPD(dec), ctx, status); - return dec; + if (PyLong_Export(v, &export_long) == -1) { + Py_DECREF(dec); + return NULL; } - size_t len = _PyLong_DigitCount(l); + if (export_long.digits) { + const PyLongLayout *layout = PyLong_GetNativeLayout(); + uint32_t base = (uint32_t)1 << layout->bits_per_digit; + uint8_t sign = export_long.negative ? MPD_NEG : MPD_POS; + Py_ssize_t len = export_long.ndigits; -#if PYLONG_BITS_IN_DIGIT == 30 - mpd_qimport_u32(MPD(dec), l->long_value.ob_digit, len, sign, PyLong_BASE, - ctx, status); -#elif PYLONG_BITS_IN_DIGIT == 15 - mpd_qimport_u16(MPD(dec), l->long_value.ob_digit, len, sign, PyLong_BASE, - ctx, status); -#else - #error "PYLONG_BITS_IN_DIGIT should be 15 or 30" -#endif + assert(layout->bits_per_digit <= 32); + assert(layout->digits_order == -1); + assert(layout->digit_endianness == (PY_LITTLE_ENDIAN ? -1 : 1)); + assert(layout->digit_size == 2 || layout->digit_size == 4); + if (layout->digit_size == 4) { + mpd_qimport_u32(MPD(dec), export_long.digits, len, sign, + base, ctx, status); + } + else { + mpd_qimport_u16(MPD(dec), export_long.digits, len, sign, + base, ctx, status); + } + PyLong_FreeExport(&export_long); + } + else { + mpd_qset_i64(MPD(dec), export_long.value, ctx, status); + } return dec; } diff --git a/Modules/_interpretersmodule.c b/Modules/_interpretersmodule.c index a36823c4bb982b..fcd0baf696f943 100644 --- a/Modules/_interpretersmodule.c +++ b/Modules/_interpretersmodule.c @@ -459,7 +459,12 @@ _run_in_interpreter(PyInterpreterState *interp, // Prep and switch interpreters. if (_PyXI_Enter(&session, interp, shareables) < 0) { - assert(!PyErr_Occurred()); + if (PyErr_Occurred()) { + // If an error occured at this step, it means that interp + // was not prepared and switched. + return -1; + } + // Now, apply the error from another interpreter: PyObject *excinfo = _PyXI_ApplyError(session.error); if (excinfo != NULL) { *p_excinfo = excinfo; diff --git a/Modules/_sre/sre.c b/Modules/_sre/sre.c index 36f542ddb4df2b..d0025dd21e045b 100644 --- a/Modules/_sre/sre.c +++ b/Modules/_sre/sre.c @@ -395,6 +395,11 @@ static struct PyModuleDef sremodule; static PyObject*pattern_new_match(_sremodulestate *, PatternObject*, SRE_STATE*, Py_ssize_t); static PyObject *pattern_scanner(_sremodulestate *, PatternObject *, PyObject *, Py_ssize_t, Py_ssize_t); +#define _PatternObject_CAST(op) ((PatternObject *)(op)) +#define _MatchObject_CAST(op) ((MatchObject *)(op)) +#define _TemplateObject_CAST(op) ((TemplateObject *)(op)) +#define _ScannerObject_CAST(op) ((ScannerObject *)(op)) + /*[clinic input] module _sre class _sre.SRE_Pattern "PatternObject *" "get_sre_module_state_by_class(tp)->Pattern_Type" @@ -699,8 +704,9 @@ pattern_error(Py_ssize_t status) } static int -pattern_traverse(PatternObject *self, visitproc visit, void *arg) +pattern_traverse(PyObject *op, visitproc visit, void *arg) { + PatternObject *self = _PatternObject_CAST(op); Py_VISIT(Py_TYPE(self)); Py_VISIT(self->groupindex); Py_VISIT(self->indexgroup); @@ -712,8 +718,9 @@ pattern_traverse(PatternObject *self, visitproc visit, void *arg) } static int -pattern_clear(PatternObject *self) +pattern_clear(PyObject *op) { + PatternObject *self = _PatternObject_CAST(op); Py_CLEAR(self->groupindex); Py_CLEAR(self->indexgroup); Py_CLEAR(self->pattern); @@ -724,13 +731,13 @@ pattern_clear(PatternObject *self) } static void -pattern_dealloc(PatternObject* self) +pattern_dealloc(PyObject *self) { PyTypeObject *tp = Py_TYPE(self); - PyObject_GC_UnTrack(self); - if (self->weakreflist != NULL) { - PyObject_ClearWeakRefs((PyObject *) self); + PatternObject *obj = _PatternObject_CAST(self); + if (obj->weakreflist != NULL) { + PyObject_ClearWeakRefs(self); } (void)pattern_clear(self); tp->tp_free(self); @@ -1497,7 +1504,7 @@ _sre_SRE_Pattern__fail_after_impl(PatternObject *self, int count, #endif /* Py_DEBUG */ static PyObject * -pattern_repr(PatternObject *obj) +pattern_repr(PyObject *self) { static const struct { const char *name; @@ -1512,6 +1519,8 @@ pattern_repr(PatternObject *obj) {"re.DEBUG", SRE_FLAG_DEBUG}, {"re.ASCII", SRE_FLAG_ASCII}, }; + + PatternObject *obj = _PatternObject_CAST(self); PyObject *result = NULL; PyObject *flag_items; size_t i; @@ -1579,8 +1588,9 @@ PyDoc_STRVAR(pattern_doc, "Compiled regular expression object."); /* PatternObject's 'groupindex' method. */ static PyObject * -pattern_groupindex(PatternObject *self, void *Py_UNUSED(ignored)) +pattern_groupindex(PyObject *op, void *Py_UNUSED(ignored)) { + PatternObject *self = _PatternObject_CAST(op); if (self->groupindex == NULL) return PyDict_New(); return PyDictProxy_New(self->groupindex); @@ -2245,8 +2255,9 @@ _validate(PatternObject *self) /* match methods */ static int -match_traverse(MatchObject *self, visitproc visit, void *arg) +match_traverse(PyObject *op, visitproc visit, void *arg) { + MatchObject *self = _MatchObject_CAST(op); Py_VISIT(Py_TYPE(self)); Py_VISIT(self->string); Py_VISIT(self->regs); @@ -2255,8 +2266,9 @@ match_traverse(MatchObject *self, visitproc visit, void *arg) } static int -match_clear(MatchObject *self) +match_clear(PyObject *op) { + MatchObject *self = _MatchObject_CAST(op); Py_CLEAR(self->string); Py_CLEAR(self->regs); Py_CLEAR(self->pattern); @@ -2264,10 +2276,9 @@ match_clear(MatchObject *self) } static void -match_dealloc(MatchObject* self) +match_dealloc(PyObject *self) { PyTypeObject *tp = Py_TYPE(self); - PyObject_GC_UnTrack(self); (void)match_clear(self); tp->tp_free(self); @@ -2376,8 +2387,9 @@ _sre_SRE_Match_expand_impl(MatchObject *self, PyObject *template) } static PyObject* -match_group(MatchObject* self, PyObject* args) +match_group(PyObject *op, PyObject* args) { + MatchObject *self = _MatchObject_CAST(op); PyObject* result; Py_ssize_t i, size; @@ -2411,8 +2423,9 @@ match_group(MatchObject* self, PyObject* args) } static PyObject* -match_getitem(MatchObject* self, PyObject* name) +match_getitem(PyObject *op, PyObject* name) { + MatchObject *self = _MatchObject_CAST(op); return match_getslice(self, name, Py_None); } @@ -2654,16 +2667,18 @@ PyDoc_STRVAR(match_group_doc, For 0 returns the entire match."); static PyObject * -match_lastindex_get(MatchObject *self, void *Py_UNUSED(ignored)) +match_lastindex_get(PyObject *op, void *Py_UNUSED(ignored)) { + MatchObject *self = _MatchObject_CAST(op); if (self->lastindex >= 0) return PyLong_FromSsize_t(self->lastindex); Py_RETURN_NONE; } static PyObject * -match_lastgroup_get(MatchObject *self, void *Py_UNUSED(ignored)) +match_lastgroup_get(PyObject *op, void *Py_UNUSED(ignored)) { + MatchObject *self = _MatchObject_CAST(op); if (self->pattern->indexgroup && self->lastindex >= 0 && self->lastindex < PyTuple_GET_SIZE(self->pattern->indexgroup)) @@ -2676,8 +2691,9 @@ match_lastgroup_get(MatchObject *self, void *Py_UNUSED(ignored)) } static PyObject * -match_regs_get(MatchObject *self, void *Py_UNUSED(ignored)) +match_regs_get(PyObject *op, void *Py_UNUSED(ignored)) { + MatchObject *self = _MatchObject_CAST(op); if (self->regs) { return Py_NewRef(self->regs); } else @@ -2780,27 +2796,29 @@ pattern_new_match(_sremodulestate* module_state, /* scanner methods (experimental) */ static int -scanner_traverse(ScannerObject *self, visitproc visit, void *arg) +scanner_traverse(PyObject *op, visitproc visit, void *arg) { + ScannerObject *self = _ScannerObject_CAST(op); Py_VISIT(Py_TYPE(self)); Py_VISIT(self->pattern); return 0; } static int -scanner_clear(ScannerObject *self) +scanner_clear(PyObject *op) { + ScannerObject *self = _ScannerObject_CAST(op); Py_CLEAR(self->pattern); return 0; } static void -scanner_dealloc(ScannerObject* self) +scanner_dealloc(PyObject *self) { PyTypeObject *tp = Py_TYPE(self); - PyObject_GC_UnTrack(self); - state_fini(&self->state); + ScannerObject *scanner = _ScannerObject_CAST(self); + state_fini(&scanner->state); (void)scanner_clear(self); tp->tp_free(self); Py_DECREF(tp); @@ -2957,8 +2975,9 @@ pattern_scanner(_sremodulestate *module_state, /* template methods */ static int -template_traverse(TemplateObject *self, visitproc visit, void *arg) +template_traverse(PyObject *op, visitproc visit, void *arg) { + TemplateObject *self = _TemplateObject_CAST(op); Py_VISIT(Py_TYPE(self)); Py_VISIT(self->literal); for (Py_ssize_t i = 0, n = Py_SIZE(self); i < n; i++) { @@ -2968,8 +2987,9 @@ template_traverse(TemplateObject *self, visitproc visit, void *arg) } static int -template_clear(TemplateObject *self) +template_clear(PyObject *op) { + TemplateObject *self = _TemplateObject_CAST(op); Py_CLEAR(self->literal); for (Py_ssize_t i = 0, n = Py_SIZE(self); i < n; i++) { Py_CLEAR(self->items[i].literal); @@ -2978,10 +2998,9 @@ template_clear(TemplateObject *self) } static void -template_dealloc(TemplateObject *self) +template_dealloc(PyObject *self) { PyTypeObject *tp = Py_TYPE(self); - PyObject_GC_UnTrack(self); (void)template_clear(self); tp->tp_free(self); @@ -3056,8 +3075,10 @@ expand_template(TemplateObject *self, MatchObject *match) static Py_hash_t -pattern_hash(PatternObject *self) +pattern_hash(PyObject *op) { + PatternObject *self = _PatternObject_CAST(op); + Py_hash_t hash, hash2; hash = PyObject_Hash(self->pattern); @@ -3148,7 +3169,7 @@ static PyMethodDef pattern_methods[] = { }; static PyGetSetDef pattern_getset[] = { - {"groupindex", (getter)pattern_groupindex, (setter)NULL, + {"groupindex", pattern_groupindex, NULL, "A dictionary mapping group names to group numbers."}, {NULL} /* Sentinel */ }; @@ -3166,9 +3187,9 @@ static PyMemberDef pattern_members[] = { }; static PyType_Slot pattern_slots[] = { - {Py_tp_dealloc, (destructor)pattern_dealloc}, - {Py_tp_repr, (reprfunc)pattern_repr}, - {Py_tp_hash, (hashfunc)pattern_hash}, + {Py_tp_dealloc, pattern_dealloc}, + {Py_tp_repr, pattern_repr}, + {Py_tp_hash, pattern_hash}, {Py_tp_doc, (void *)pattern_doc}, {Py_tp_richcompare, pattern_richcompare}, {Py_tp_methods, pattern_methods}, @@ -3189,7 +3210,7 @@ static PyType_Spec pattern_spec = { }; static PyMethodDef match_methods[] = { - {"group", (PyCFunction) match_group, METH_VARARGS, match_group_doc}, + {"group", match_group, METH_VARARGS, match_group_doc}, _SRE_SRE_MATCH_START_METHODDEF _SRE_SRE_MATCH_END_METHODDEF _SRE_SRE_MATCH_SPAN_METHODDEF @@ -3204,11 +3225,11 @@ static PyMethodDef match_methods[] = { }; static PyGetSetDef match_getset[] = { - {"lastindex", (getter)match_lastindex_get, (setter)NULL, + {"lastindex", match_lastindex_get, NULL, "The integer index of the last matched capturing group."}, - {"lastgroup", (getter)match_lastgroup_get, (setter)NULL, + {"lastgroup", match_lastgroup_get, NULL, "The name of the last matched capturing group."}, - {"regs", (getter)match_regs_get, (setter)NULL}, + {"regs", match_regs_get, NULL, NULL}, {NULL} }; diff --git a/Modules/_ssl.c b/Modules/_ssl.c index 05dd5fb38bcf40..cb3634bf44f893 100644 --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -4377,7 +4377,7 @@ _ssl__SSLContext_load_dh_params_impl(PySSLContext *self, PyObject *filepath) FILE *f; DH *dh; - f = _Py_fopen_obj(filepath, "rb"); + f = Py_fopen(filepath, "rb"); if (f == NULL) return NULL; diff --git a/Modules/_ssl/debughelpers.c b/Modules/_ssl/debughelpers.c index 9c87f8b4d21e68..318c045a0eec3c 100644 --- a/Modules/_ssl/debughelpers.c +++ b/Modules/_ssl/debughelpers.c @@ -180,8 +180,8 @@ _PySSLContext_set_keylog_filename(PySSLContext *self, PyObject *arg, void *c) { return 0; } - /* _Py_fopen_obj() also checks that arg is of proper type. */ - fp = _Py_fopen_obj(arg, "a" PY_STDIOTEXTMODE); + /* Py_fopen() also checks that arg is of proper type. */ + fp = Py_fopen(arg, "a" PY_STDIOTEXTMODE); if (fp == NULL) return -1; diff --git a/Modules/_testcapi/clinic/file.c.h b/Modules/_testcapi/clinic/file.c.h new file mode 100644 index 00000000000000..fddbf48071bd3b --- /dev/null +++ b/Modules/_testcapi/clinic/file.c.h @@ -0,0 +1,37 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +#include "pycore_modsupport.h" // _PyArg_CheckPositional() + +PyDoc_STRVAR(_testcapi_py_fopen__doc__, +"py_fopen($module, path, mode, /)\n" +"--\n" +"\n" +"Call Py_fopen(), fread(256) and Py_fclose(). Return read bytes."); + +#define _TESTCAPI_PY_FOPEN_METHODDEF \ + {"py_fopen", _PyCFunction_CAST(_testcapi_py_fopen), METH_FASTCALL, _testcapi_py_fopen__doc__}, + +static PyObject * +_testcapi_py_fopen_impl(PyObject *module, PyObject *path, const char *mode, + Py_ssize_t mode_length); + +static PyObject * +_testcapi_py_fopen(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + PyObject *path; + const char *mode; + Py_ssize_t mode_length; + + if (!_PyArg_ParseStack(args, nargs, "Oz#:py_fopen", + &path, &mode, &mode_length)) { + goto exit; + } + return_value = _testcapi_py_fopen_impl(module, path, mode, mode_length); + +exit: + return return_value; +} +/*[clinic end generated code: output=c4dc92400306c3eb input=a9049054013a1b77]*/ diff --git a/Modules/_testcapi/file.c b/Modules/_testcapi/file.c index 634563f6ea12cb..d15173fc7959e5 100644 --- a/Modules/_testcapi/file.c +++ b/Modules/_testcapi/file.c @@ -1,8 +1,45 @@ +// clinic/file.c.h uses internal pycore_modsupport.h API +#define PYTESTCAPI_NEED_INTERNAL_API + #include "parts.h" #include "util.h" +#include "clinic/file.c.h" + +/*[clinic input] +module _testcapi +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=6361033e795369fc]*/ + +/*[clinic input] +_testcapi.py_fopen + + path: object + mode: str(zeroes=True, accept={robuffer, str, NoneType}) + / + +Call Py_fopen(), fread(256) and Py_fclose(). Return read bytes. +[clinic start generated code]*/ +static PyObject * +_testcapi_py_fopen_impl(PyObject *module, PyObject *path, const char *mode, + Py_ssize_t mode_length) +/*[clinic end generated code: output=69840d0cfd8b7fbb input=f3a579dd7eb60926]*/ +{ + NULLABLE(path); + FILE *fp = Py_fopen(path, mode); + if (fp == NULL) { + return NULL; + } + + char buffer[256]; + size_t size = fread(buffer, 1, Py_ARRAY_LENGTH(buffer), fp); + Py_fclose(fp); + + return PyBytes_FromStringAndSize(buffer, size); +} static PyMethodDef test_methods[] = { + _TESTCAPI_PY_FOPEN_METHODDEF {NULL}, }; diff --git a/Modules/_testcapi/object.c b/Modules/_testcapi/object.c index 3af5429ef00985..841410c52b3ce2 100644 --- a/Modules/_testcapi/object.c +++ b/Modules/_testcapi/object.c @@ -15,7 +15,7 @@ call_pyobject_print(PyObject *self, PyObject * args) return NULL; } - fp = _Py_fopen_obj(filename, "w+"); + fp = Py_fopen(filename, "w+"); if (Py_IsTrue(print_raw)) { flags = Py_PRINT_RAW; @@ -41,7 +41,7 @@ pyobject_print_null(PyObject *self, PyObject *args) return NULL; } - fp = _Py_fopen_obj(filename, "w+"); + fp = Py_fopen(filename, "w+"); if (PyObject_Print(NULL, fp, 0) < 0) { fclose(fp); @@ -72,7 +72,7 @@ pyobject_print_noref_object(PyObject *self, PyObject *args) return NULL; } - fp = _Py_fopen_obj(filename, "w+"); + fp = Py_fopen(filename, "w+"); if (PyObject_Print(test_string, fp, 0) < 0){ fclose(fp); @@ -103,7 +103,7 @@ pyobject_print_os_error(PyObject *self, PyObject *args) } // open file in read mode to induce OSError - fp = _Py_fopen_obj(filename, "r"); + fp = Py_fopen(filename, "r"); if (PyObject_Print(test_string, fp, 0) < 0) { fclose(fp); diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index f737250ac29d57..a0a1f8af6710a3 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -1744,7 +1744,7 @@ pymarshal_write_long_to_file(PyObject* self, PyObject *args) &value, &filename, &version)) return NULL; - fp = _Py_fopen_obj(filename, "wb"); + fp = Py_fopen(filename, "wb"); if (fp == NULL) { PyErr_SetFromErrno(PyExc_OSError); return NULL; @@ -1769,7 +1769,7 @@ pymarshal_write_object_to_file(PyObject* self, PyObject *args) &obj, &filename, &version)) return NULL; - fp = _Py_fopen_obj(filename, "wb"); + fp = Py_fopen(filename, "wb"); if (fp == NULL) { PyErr_SetFromErrno(PyExc_OSError); return NULL; @@ -1793,7 +1793,7 @@ pymarshal_read_short_from_file(PyObject* self, PyObject *args) if (!PyArg_ParseTuple(args, "O:pymarshal_read_short_from_file", &filename)) return NULL; - fp = _Py_fopen_obj(filename, "rb"); + fp = Py_fopen(filename, "rb"); if (fp == NULL) { PyErr_SetFromErrno(PyExc_OSError); return NULL; @@ -1818,7 +1818,7 @@ pymarshal_read_long_from_file(PyObject* self, PyObject *args) if (!PyArg_ParseTuple(args, "O:pymarshal_read_long_from_file", &filename)) return NULL; - fp = _Py_fopen_obj(filename, "rb"); + fp = Py_fopen(filename, "rb"); if (fp == NULL) { PyErr_SetFromErrno(PyExc_OSError); return NULL; @@ -1840,7 +1840,7 @@ pymarshal_read_last_object_from_file(PyObject* self, PyObject *args) if (!PyArg_ParseTuple(args, "O:pymarshal_read_last_object_from_file", &filename)) return NULL; - FILE *fp = _Py_fopen_obj(filename, "rb"); + FILE *fp = Py_fopen(filename, "rb"); if (fp == NULL) { PyErr_SetFromErrno(PyExc_OSError); return NULL; @@ -1863,7 +1863,7 @@ pymarshal_read_object_from_file(PyObject* self, PyObject *args) if (!PyArg_ParseTuple(args, "O:pymarshal_read_object_from_file", &filename)) return NULL; - FILE *fp = _Py_fopen_obj(filename, "rb"); + FILE *fp = Py_fopen(filename, "rb"); if (fp == NULL) { PyErr_SetFromErrno(PyExc_OSError); return NULL; @@ -3415,6 +3415,26 @@ test_atexit(PyObject *self, PyObject *Py_UNUSED(args)) Py_RETURN_NONE; } +static PyObject* +code_offset_to_line(PyObject* self, PyObject* const* args, Py_ssize_t nargsf) +{ + Py_ssize_t nargs = _PyVectorcall_NARGS(nargsf); + if (nargs != 2) { + PyErr_SetString(PyExc_TypeError, "code_offset_to_line takes 2 arguments"); + return NULL; + } + int offset; + if (PyLong_AsInt32(args[1], &offset) < 0) { + return NULL; + } + PyCodeObject *code = (PyCodeObject *)args[0]; + if (!PyCode_Check(code)) { + PyErr_SetString(PyExc_TypeError, "first arg must be a code object"); + return NULL; + } + return PyLong_FromInt32(PyCode_Addr2Line(code, offset)); +} + static PyMethodDef TestMethods[] = { {"set_errno", set_errno, METH_VARARGS}, {"test_config", test_config, METH_NOARGS}, @@ -3557,6 +3577,7 @@ static PyMethodDef TestMethods[] = { {"finalize_thread_hang", finalize_thread_hang, METH_O, NULL}, {"type_freeze", type_freeze, METH_VARARGS}, {"test_atexit", test_atexit, METH_NOARGS}, + {"code_offset_to_line", _PyCFunction_CAST(code_offset_to_line), METH_FASTCALL}, {NULL, NULL} /* sentinel */ }; diff --git a/Modules/_testlimitedcapi.c b/Modules/_testlimitedcapi.c index ba83a23117b2a5..bcc69a339ec5c4 100644 --- a/Modules/_testlimitedcapi.c +++ b/Modules/_testlimitedcapi.c @@ -83,5 +83,8 @@ PyInit__testlimitedcapi(void) if (_PyTestLimitedCAPI_Init_VectorcallLimited(mod) < 0) { return NULL; } + if (_PyTestLimitedCAPI_Init_Version(mod) < 0) { + return NULL; + } return mod; } diff --git a/Modules/_testlimitedcapi/clinic/version.c.h b/Modules/_testlimitedcapi/clinic/version.c.h new file mode 100644 index 00000000000000..096c7dd528b332 --- /dev/null +++ b/Modules/_testlimitedcapi/clinic/version.c.h @@ -0,0 +1,93 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +PyDoc_STRVAR(_testlimitedcapi_pack_full_version__doc__, +"pack_full_version($module, major, minor, micro, level, serial, /)\n" +"--\n" +"\n"); + +#define _TESTLIMITEDCAPI_PACK_FULL_VERSION_METHODDEF \ + {"pack_full_version", (PyCFunction)(void(*)(void))_testlimitedcapi_pack_full_version, METH_FASTCALL, _testlimitedcapi_pack_full_version__doc__}, + +static PyObject * +_testlimitedcapi_pack_full_version_impl(PyObject *module, int major, + int minor, int micro, int level, + int serial); + +static PyObject * +_testlimitedcapi_pack_full_version(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + int major; + int minor; + int micro; + int level; + int serial; + + if (nargs != 5) { + PyErr_Format(PyExc_TypeError, "pack_full_version expected 5 arguments, got %zd", nargs); + goto exit; + } + major = PyLong_AsInt(args[0]); + if (major == -1 && PyErr_Occurred()) { + goto exit; + } + minor = PyLong_AsInt(args[1]); + if (minor == -1 && PyErr_Occurred()) { + goto exit; + } + micro = PyLong_AsInt(args[2]); + if (micro == -1 && PyErr_Occurred()) { + goto exit; + } + level = PyLong_AsInt(args[3]); + if (level == -1 && PyErr_Occurred()) { + goto exit; + } + serial = PyLong_AsInt(args[4]); + if (serial == -1 && PyErr_Occurred()) { + goto exit; + } + return_value = _testlimitedcapi_pack_full_version_impl(module, major, minor, micro, level, serial); + +exit: + return return_value; +} + +PyDoc_STRVAR(_testlimitedcapi_pack_version__doc__, +"pack_version($module, major, minor, /)\n" +"--\n" +"\n"); + +#define _TESTLIMITEDCAPI_PACK_VERSION_METHODDEF \ + {"pack_version", (PyCFunction)(void(*)(void))_testlimitedcapi_pack_version, METH_FASTCALL, _testlimitedcapi_pack_version__doc__}, + +static PyObject * +_testlimitedcapi_pack_version_impl(PyObject *module, int major, int minor); + +static PyObject * +_testlimitedcapi_pack_version(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + int major; + int minor; + + if (nargs != 2) { + PyErr_Format(PyExc_TypeError, "pack_version expected 2 arguments, got %zd", nargs); + goto exit; + } + major = PyLong_AsInt(args[0]); + if (major == -1 && PyErr_Occurred()) { + goto exit; + } + minor = PyLong_AsInt(args[1]); + if (minor == -1 && PyErr_Occurred()) { + goto exit; + } + return_value = _testlimitedcapi_pack_version_impl(module, major, minor); + +exit: + return return_value; +} +/*[clinic end generated code: output=aed3e226da77f2d2 input=a9049054013a1b77]*/ diff --git a/Modules/_testlimitedcapi/parts.h b/Modules/_testlimitedcapi/parts.h index 4107b150c5b4e0..56d566b66565a3 100644 --- a/Modules/_testlimitedcapi/parts.h +++ b/Modules/_testlimitedcapi/parts.h @@ -40,5 +40,6 @@ int _PyTestLimitedCAPI_Init_Sys(PyObject *module); int _PyTestLimitedCAPI_Init_Tuple(PyObject *module); int _PyTestLimitedCAPI_Init_Unicode(PyObject *module); int _PyTestLimitedCAPI_Init_VectorcallLimited(PyObject *module); +int _PyTestLimitedCAPI_Init_Version(PyObject *module); #endif // Py_TESTLIMITEDCAPI_PARTS_H diff --git a/Modules/_testlimitedcapi/version.c b/Modules/_testlimitedcapi/version.c new file mode 100644 index 00000000000000..57cd6e4e928ea3 --- /dev/null +++ b/Modules/_testlimitedcapi/version.c @@ -0,0 +1,77 @@ +/* Test version macros in the limited API */ + +#include "pyconfig.h" // Py_GIL_DISABLED +#ifndef Py_GIL_DISABLED +# define Py_LIMITED_API 0x030e0000 // Added in 3.14 +#endif + +#include "parts.h" +#include "clinic/version.c.h" +#include + +/*[clinic input] +module _testlimitedcapi +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=2700057f9c1135ba]*/ + +/*[clinic input] +_testlimitedcapi.pack_full_version + + major: int + minor: int + micro: int + level: int + serial: int + / +[clinic start generated code]*/ + +static PyObject * +_testlimitedcapi_pack_full_version_impl(PyObject *module, int major, + int minor, int micro, int level, + int serial) +/*[clinic end generated code: output=b87a1e9805648861 input=2a304423be61d2ac]*/ +{ + uint32_t macro_result = Py_PACK_FULL_VERSION( + major, minor, micro, level, serial); +#undef Py_PACK_FULL_VERSION + uint32_t func_result = Py_PACK_FULL_VERSION( + major, minor, micro, level, serial); + + assert(macro_result == func_result); + return PyLong_FromUnsignedLong((unsigned long)func_result); +} + +/*[clinic input] +_testlimitedcapi.pack_version + + major: int + minor: int + / +[clinic start generated code]*/ + +static PyObject * +_testlimitedcapi_pack_version_impl(PyObject *module, int major, int minor) +/*[clinic end generated code: output=771247bbd06e7883 input=3e39e9dcbc09e86a]*/ +{ + uint32_t macro_result = Py_PACK_VERSION(major, minor); +#undef Py_PACK_VERSION + uint32_t func_result = Py_PACK_VERSION(major, minor); + + assert(macro_result == func_result); + return PyLong_FromUnsignedLong((unsigned long)func_result); +} + +static PyMethodDef TestMethods[] = { + _TESTLIMITEDCAPI_PACK_FULL_VERSION_METHODDEF + _TESTLIMITEDCAPI_PACK_VERSION_METHODDEF + {NULL}, +}; + +int +_PyTestLimitedCAPI_Init_Version(PyObject *m) +{ + if (PyModule_AddFunctions(m, TestMethods) < 0) { + return -1; + } + return 0; +} diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c index 2cbdfeb09b95ae..d19ae326bd6b48 100644 --- a/Modules/_threadmodule.c +++ b/Modules/_threadmodule.c @@ -1414,6 +1414,10 @@ local_new(PyTypeObject *type, PyObject *args, PyObject *kw) return NULL; } + // gh-128691: Use deferred reference counting for thread-locals to avoid + // contention on the shared object. + _PyObject_SetDeferredRefcount((PyObject *)self); + self->args = Py_XNewRef(args); self->kw = Py_XNewRef(kw); diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h index 48f77a782f2a17..96bf21dced92f0 100644 --- a/Modules/clinic/posixmodule.c.h +++ b/Modules/clinic/posixmodule.c.h @@ -309,7 +309,7 @@ os_access(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *k return return_value; } -#if defined(HAVE_TTYNAME) +#if defined(HAVE_TTYNAME_R) PyDoc_STRVAR(os_ttyname__doc__, "ttyname($module, fd, /)\n" @@ -342,7 +342,7 @@ os_ttyname(PyObject *module, PyObject *arg) return return_value; } -#endif /* defined(HAVE_TTYNAME) */ +#endif /* defined(HAVE_TTYNAME_R) */ #if defined(HAVE_CTERMID) @@ -13140,4 +13140,4 @@ os__emscripten_debugger(PyObject *module, PyObject *Py_UNUSED(ignored)) #ifndef OS__EMSCRIPTEN_DEBUGGER_METHODDEF #define OS__EMSCRIPTEN_DEBUGGER_METHODDEF #endif /* !defined(OS__EMSCRIPTEN_DEBUGGER_METHODDEF) */ -/*[clinic end generated code: output=cc338f1f7d76a29e input=a9049054013a1b77]*/ +/*[clinic end generated code: output=34cb96bd07bcef90 input=a9049054013a1b77]*/ diff --git a/Modules/getpath.c b/Modules/getpath.c index 18ddfaf8dbce1a..2d3c9757298d16 100644 --- a/Modules/getpath.c +++ b/Modules/getpath.c @@ -17,10 +17,13 @@ #endif #ifdef __APPLE__ -# include # include #endif +#ifdef HAVE_DLFCN_H +# include +#endif + /* Reference the precompiled getpath.py */ #include "Python/frozen_modules/getpath.h" @@ -803,36 +806,25 @@ progname_to_dict(PyObject *dict, const char *key) static int library_to_dict(PyObject *dict, const char *key) { +/* macOS framework builds do not link against a libpython dynamic library, but + instead link against a macOS Framework. */ +#if defined(Py_ENABLE_SHARED) || defined(WITH_NEXT_FRAMEWORK) + #ifdef MS_WINDOWS -#ifdef Py_ENABLE_SHARED extern HMODULE PyWin_DLLhModule; if (PyWin_DLLhModule) { return winmodule_to_dict(dict, key, PyWin_DLLhModule); } #endif -#elif defined(WITH_NEXT_FRAMEWORK) - static char modPath[MAXPATHLEN + 1]; - static int modPathInitialized = -1; - if (modPathInitialized < 0) { - modPathInitialized = 0; - - /* On Mac OS X we have a special case if we're running from a framework. - This is because the python home should be set relative to the library, - which is in the framework, not relative to the executable, which may - be outside of the framework. Except when we're in the build - directory... */ - Dl_info pythonInfo; - if (dladdr(&Py_Initialize, &pythonInfo)) { - if (pythonInfo.dli_fname) { - strncpy(modPath, pythonInfo.dli_fname, MAXPATHLEN); - modPathInitialized = 1; - } - } - } - if (modPathInitialized > 0) { - return decode_to_dict(dict, key, modPath); + +#if HAVE_DLADDR + Dl_info libpython_info; + if (dladdr(&Py_Initialize, &libpython_info) && libpython_info.dli_fname) { + return decode_to_dict(dict, key, libpython_info.dli_fname); } #endif +#endif + return PyDict_SetItemString(dict, key, Py_None) == 0; } diff --git a/Modules/getpath.py b/Modules/getpath.py index c34101e720851d..be2210345afbda 100644 --- a/Modules/getpath.py +++ b/Modules/getpath.py @@ -625,6 +625,8 @@ def search_up(prefix, *landmarks, test=isfile): # gh-100320: Our PYDs are assumed to be relative to the Lib directory # (that is, prefix) rather than the executable (that is, executable_dir) exec_prefix = prefix + if not exec_prefix and prefix and isdir(joinpath(prefix, PLATSTDLIB_LANDMARK)): + exec_prefix = prefix if not exec_prefix and executable_dir: exec_prefix = search_up(executable_dir, PLATSTDLIB_LANDMARK, test=isdir) if not exec_prefix and EXEC_PREFIX: diff --git a/Modules/main.c b/Modules/main.c index 3bf2241f2837a3..5bb1de2d04d30c 100644 --- a/Modules/main.c +++ b/Modules/main.c @@ -370,7 +370,7 @@ pymain_run_file_obj(PyObject *program_name, PyObject *filename, return pymain_exit_err_print(); } - FILE *fp = _Py_fopen_obj(filename, "rb"); + FILE *fp = Py_fopen(filename, "rb"); if (fp == NULL) { // Ignore the OSError PyErr_Clear(); @@ -465,7 +465,7 @@ pymain_run_startup(PyConfig *config, int *exitcode) goto error; } - FILE *fp = _Py_fopen_obj(startup, "r"); + FILE *fp = Py_fopen(startup, "r"); if (fp == NULL) { int save_errno = errno; PyErr_Clear(); diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 151d469983fafb..bb8d698bfed375 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3347,7 +3347,7 @@ os_access_impl(PyObject *module, path_t *path, int mode, int dir_fd, #endif -#ifdef HAVE_TTYNAME +#ifdef HAVE_TTYNAME_R /*[clinic input] os.ttyname diff --git a/Objects/capsule.c b/Objects/capsule.c index 28965e0f21b7a0..16ae65905ef5ac 100644 --- a/Objects/capsule.c +++ b/Objects/capsule.c @@ -18,6 +18,8 @@ typedef struct { } PyCapsule; +#define _PyCapsule_CAST(op) ((PyCapsule *)(op)) + static int _is_legal_capsule(PyObject *op, const char *invalid_capsule) @@ -284,7 +286,7 @@ PyCapsule_Import(const char *name, int no_block) static void capsule_dealloc(PyObject *op) { - PyCapsule *capsule = (PyCapsule *)op; + PyCapsule *capsule = _PyCapsule_CAST(op); PyObject_GC_UnTrack(op); if (capsule->destructor) { capsule->destructor(op); @@ -296,7 +298,7 @@ capsule_dealloc(PyObject *op) static PyObject * capsule_repr(PyObject *o) { - PyCapsule *capsule = (PyCapsule *)o; + PyCapsule *capsule = _PyCapsule_CAST(o); const char *name; const char *quote; @@ -314,28 +316,27 @@ capsule_repr(PyObject *o) static int -capsule_traverse(PyCapsule *capsule, visitproc visit, void *arg) +capsule_traverse(PyObject *self, visitproc visit, void *arg) { // Capsule object is only tracked by the GC // if _PyCapsule_SetTraverse() is called, but // this can still be manually triggered by gc.get_referents() - + PyCapsule *capsule = _PyCapsule_CAST(self); if (capsule->traverse_func != NULL) { - return capsule->traverse_func((PyObject*)capsule, visit, arg); + return capsule->traverse_func(self, visit, arg); } - return 0; } static int -capsule_clear(PyCapsule *capsule) +capsule_clear(PyObject *self) { // Capsule object is only tracked by the GC // if _PyCapsule_SetTraverse() is called + PyCapsule *capsule = _PyCapsule_CAST(self); assert(capsule->clear_func != NULL); - - return capsule->clear_func((PyObject*)capsule); + return capsule->clear_func(self); } @@ -358,8 +359,8 @@ PyTypeObject PyCapsule_Type = { .tp_dealloc = capsule_dealloc, .tp_repr = capsule_repr, .tp_doc = PyCapsule_Type__doc__, - .tp_traverse = (traverseproc)capsule_traverse, - .tp_clear = (inquiry)capsule_clear, + .tp_traverse = capsule_traverse, + .tp_clear = capsule_clear, }; diff --git a/Objects/descrobject.c b/Objects/descrobject.c index 4eccd1704eb95a..1852118359f014 100644 --- a/Objects/descrobject.c +++ b/Objects/descrobject.c @@ -1508,6 +1508,8 @@ PyWrapper_New(PyObject *d, PyObject *self) /* A built-in 'property' type */ +#define _propertyobject_CAST(op) ((propertyobject *)(op)) + /* class property(object): @@ -1911,8 +1913,9 @@ property_init_impl(propertyobject *self, PyObject *fget, PyObject *fset, } static PyObject * -property_get__name__(propertyobject *prop, void *Py_UNUSED(ignored)) +property_get__name__(PyObject *op, void *Py_UNUSED(ignored)) { + propertyobject *prop = _propertyobject_CAST(op); PyObject *name; if (property_name(prop, &name) < 0) { return NULL; @@ -1925,16 +1928,17 @@ property_get__name__(propertyobject *prop, void *Py_UNUSED(ignored)) } static int -property_set__name__(propertyobject *prop, PyObject *value, - void *Py_UNUSED(ignored)) +property_set__name__(PyObject *op, PyObject *value, void *Py_UNUSED(ignored)) { + propertyobject *prop = _propertyobject_CAST(op); Py_XSETREF(prop->prop_name, Py_XNewRef(value)); return 0; } static PyObject * -property_get___isabstractmethod__(propertyobject *prop, void *closure) +property_get___isabstractmethod__(PyObject *op, void *closure) { + propertyobject *prop = _propertyobject_CAST(op); int res = _PyObject_IsAbstract(prop->prop_get); if (res == -1) { return NULL; @@ -1962,9 +1966,8 @@ property_get___isabstractmethod__(propertyobject *prop, void *closure) } static PyGetSetDef property_getsetlist[] = { - {"__name__", (getter)property_get__name__, (setter)property_set__name__}, - {"__isabstractmethod__", - (getter)property_get___isabstractmethod__, NULL, + {"__name__", property_get__name__, property_set__name__, NULL, NULL}, + {"__isabstractmethod__", property_get___isabstractmethod__, NULL, NULL, NULL}, {NULL} /* Sentinel */ diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 7f95b519561e68..680846f1c0b865 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -2860,7 +2860,7 @@ vectorcall_maybe(PyThreadState *tstate, PyObject *name, */ static int -tail_contains(PyObject *tuple, int whence, PyObject *o) +tail_contains(PyObject *tuple, Py_ssize_t whence, PyObject *o) { Py_ssize_t j, size; size = PyTuple_GET_SIZE(tuple); @@ -2923,7 +2923,7 @@ check_duplicates(PyObject *tuple) */ static void -set_mro_error(PyObject **to_merge, Py_ssize_t to_merge_size, int *remain) +set_mro_error(PyObject **to_merge, Py_ssize_t to_merge_size, Py_ssize_t *remain) { Py_ssize_t i, n, off; char buf[1000]; @@ -2978,13 +2978,13 @@ pmerge(PyObject *acc, PyObject **to_merge, Py_ssize_t to_merge_size) { int res = 0; Py_ssize_t i, j, empty_cnt; - int *remain; + Py_ssize_t *remain; /* remain stores an index into each sublist of to_merge. remain[i] is the index of the next base in to_merge[i] that is not included in acc. */ - remain = PyMem_New(int, to_merge_size); + remain = PyMem_New(Py_ssize_t, to_merge_size); if (remain == NULL) { PyErr_NoMemory(); return -1; diff --git a/PC/python3dll.c b/PC/python3dll.c index 8657ddb9fa5155..84b3c735240b73 100755 --- a/PC/python3dll.c +++ b/PC/python3dll.c @@ -81,6 +81,8 @@ EXPORT_FUNC(Py_Main) EXPORT_FUNC(Py_MakePendingCalls) EXPORT_FUNC(Py_NewInterpreter) EXPORT_FUNC(Py_NewRef) +EXPORT_FUNC(Py_PACK_FULL_VERSION) +EXPORT_FUNC(Py_PACK_VERSION) EXPORT_FUNC(Py_REFCNT) EXPORT_FUNC(Py_ReprEnter) EXPORT_FUNC(Py_ReprLeave) diff --git a/PCbuild/_testlimitedcapi.vcxproj b/PCbuild/_testlimitedcapi.vcxproj index 846e027e10c7fa..0ea5edba3aa9a7 100644 --- a/PCbuild/_testlimitedcapi.vcxproj +++ b/PCbuild/_testlimitedcapi.vcxproj @@ -112,6 +112,7 @@ + diff --git a/PCbuild/_testlimitedcapi.vcxproj.filters b/PCbuild/_testlimitedcapi.vcxproj.filters index 57be2e2fc5b950..b379090eb599f5 100644 --- a/PCbuild/_testlimitedcapi.vcxproj.filters +++ b/PCbuild/_testlimitedcapi.vcxproj.filters @@ -28,6 +28,7 @@ + diff --git a/Programs/test_frozenmain.h b/Programs/test_frozenmain.h index a0007830e8cbc0..4f6933ac0ddcd6 100644 --- a/Programs/test_frozenmain.h +++ b/Programs/test_frozenmain.h @@ -1,37 +1,37 @@ // Auto-generated by Programs/freeze_test_frozenmain.py unsigned char M_test_frozenmain[] = { 227,0,0,0,0,0,0,0,0,0,0,0,0,9,0,0, - 0,0,0,0,0,243,170,0,0,0,149,0,90,0,80,0, - 71,0,112,0,90,0,80,0,71,1,112,1,89,2,32,0, - 80,1,50,1,0,0,0,0,0,0,30,0,89,2,32,0, - 80,2,89,0,78,6,0,0,0,0,0,0,0,0,0,0, - 0,0,0,0,0,0,0,0,50,2,0,0,0,0,0,0, - 30,0,89,1,78,8,0,0,0,0,0,0,0,0,0,0, - 0,0,0,0,0,0,0,0,32,0,50,0,0,0,0,0, - 0,0,80,3,2,0,0,0,112,5,80,4,16,0,68,21, - 0,0,28,0,112,6,89,2,32,0,80,5,89,6,12,0, - 80,6,89,5,89,6,2,0,0,0,12,0,48,4,50,1, - 0,0,0,0,0,0,30,0,73,23,0,0,9,0,30,0, - 80,0,34,0,41,7,78,122,18,70,114,111,122,101,110,32, - 72,101,108,108,111,32,87,111,114,108,100,122,8,115,121,115, - 46,97,114,103,118,218,6,99,111,110,102,105,103,41,5,218, - 12,112,114,111,103,114,97,109,95,110,97,109,101,218,10,101, - 120,101,99,117,116,97,98,108,101,218,15,117,115,101,95,101, - 110,118,105,114,111,110,109,101,110,116,218,17,99,111,110,102, - 105,103,117,114,101,95,99,95,115,116,100,105,111,218,14,98, - 117,102,102,101,114,101,100,95,115,116,100,105,111,122,7,99, - 111,110,102,105,103,32,122,2,58,32,41,7,218,3,115,121, - 115,218,17,95,116,101,115,116,105,110,116,101,114,110,97,108, - 99,97,112,105,218,5,112,114,105,110,116,218,4,97,114,103, - 118,218,11,103,101,116,95,99,111,110,102,105,103,115,114,2, - 0,0,0,218,3,107,101,121,169,0,243,0,0,0,0,218, - 18,116,101,115,116,95,102,114,111,122,101,110,109,97,105,110, - 46,112,121,218,8,60,109,111,100,117,108,101,62,114,17,0, - 0,0,1,0,0,0,115,94,0,0,0,240,3,1,1,1, - 243,8,0,1,11,219,0,24,225,0,5,208,6,26,212,0, - 27,217,0,5,128,106,144,35,151,40,145,40,212,0,27,216, - 9,26,215,9,38,210,9,38,211,9,40,168,24,209,9,50, - 128,6,244,2,6,12,2,128,67,241,14,0,5,10,136,71, - 144,67,144,53,152,2,152,54,160,35,153,59,152,45,208,10, - 40,214,4,41,243,15,6,12,2,114,15,0,0,0, + 0,0,0,0,0,243,168,0,0,0,149,0,91,0,81,0, + 72,0,113,0,91,0,81,0,72,1,113,1,90,2,33,0, + 81,1,51,1,0,0,0,0,0,0,31,0,90,2,33,0, + 81,2,90,0,79,6,0,0,0,0,0,0,0,0,0,0, + 0,0,0,0,0,0,0,0,51,2,0,0,0,0,0,0, + 31,0,90,1,79,8,0,0,0,0,0,0,0,0,0,0, + 0,0,0,0,0,0,0,0,33,0,51,0,0,0,0,0, + 0,0,81,3,2,0,0,0,113,5,81,4,16,0,69,20, + 0,0,113,6,90,2,33,0,81,5,90,6,12,0,81,6, + 90,5,90,6,2,0,0,0,12,0,49,4,51,1,0,0, + 0,0,0,0,31,0,74,22,0,0,9,0,30,0,81,0, + 35,0,41,7,78,122,18,70,114,111,122,101,110,32,72,101, + 108,108,111,32,87,111,114,108,100,122,8,115,121,115,46,97, + 114,103,118,218,6,99,111,110,102,105,103,41,5,218,12,112, + 114,111,103,114,97,109,95,110,97,109,101,218,10,101,120,101, + 99,117,116,97,98,108,101,218,15,117,115,101,95,101,110,118, + 105,114,111,110,109,101,110,116,218,17,99,111,110,102,105,103, + 117,114,101,95,99,95,115,116,100,105,111,218,14,98,117,102, + 102,101,114,101,100,95,115,116,100,105,111,122,7,99,111,110, + 102,105,103,32,122,2,58,32,41,7,218,3,115,121,115,218, + 17,95,116,101,115,116,105,110,116,101,114,110,97,108,99,97, + 112,105,218,5,112,114,105,110,116,218,4,97,114,103,118,218, + 11,103,101,116,95,99,111,110,102,105,103,115,114,2,0,0, + 0,218,3,107,101,121,169,0,243,0,0,0,0,218,18,116, + 101,115,116,95,102,114,111,122,101,110,109,97,105,110,46,112, + 121,218,8,60,109,111,100,117,108,101,62,114,17,0,0,0, + 1,0,0,0,115,94,0,0,0,240,3,1,1,1,243,8, + 0,1,11,219,0,24,225,0,5,208,6,26,212,0,27,217, + 0,5,128,106,144,35,151,40,145,40,212,0,27,216,9,26, + 215,9,38,210,9,38,211,9,40,168,24,209,9,50,128,6, + 243,2,6,12,2,128,67,241,14,0,5,10,136,71,144,67, + 144,53,152,2,152,54,160,35,153,59,152,45,208,10,40,214, + 4,41,243,15,6,12,2,114,15,0,0,0, }; diff --git a/Python/bytecodes.c b/Python/bytecodes.c index 602cf7f47b812b..8bab4ea16b629b 100644 --- a/Python/bytecodes.c +++ b/Python/bytecodes.c @@ -60,6 +60,8 @@ #define specializing #define split #define replicate(TIMES) +#define tier1 +#define no_save_ip // Dummy variables for stack effects. static PyObject *value, *value1, *value2, *left, *right, *res, *sum, *prod, *sub; @@ -336,9 +338,18 @@ dummy_func( res = PyStackRef_NULL; } - macro(END_FOR) = POP_TOP; + no_save_ip inst(END_FOR, (value -- )) { + /* Don't update instr_ptr, so that POP_ITER sees + * the FOR_ITER as the previous instruction. + * This has the benign side effect that if value is + * finalized it will see the location as the FOR_ITER's. + */ + PyStackRef_CLOSE(value); + } + + macro(POP_ITER) = POP_TOP; - tier1 inst(INSTRUMENTED_END_FOR, (receiver, value -- receiver)) { + no_save_ip tier1 inst(INSTRUMENTED_END_FOR, (receiver, value -- receiver)) { /* Need to create a fake StopIteration error here, * to conform to PEP 380 */ if (PyStackRef_GenCheck(receiver)) { @@ -350,6 +361,11 @@ dummy_func( DECREF_INPUTS(); } + tier1 inst(INSTRUMENTED_POP_ITER, (iter -- )) { + INSTRUMENTED_JUMP(prev_instr, this_instr+1, PY_MONITORING_EVENT_BRANCH_RIGHT); + PyStackRef_CLOSE(iter); + } + pure inst(END_SEND, (receiver, value -- val)) { (void)receiver; val = value; @@ -514,6 +530,8 @@ dummy_func( pure op(_BINARY_OP_MULTIPLY_INT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Multiply((PyLongObject *)left_o, (PyLongObject *)right_o); @@ -527,6 +545,8 @@ dummy_func( pure op(_BINARY_OP_ADD_INT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Add((PyLongObject *)left_o, (PyLongObject *)right_o); @@ -540,6 +560,8 @@ dummy_func( pure op(_BINARY_OP_SUBTRACT_INT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Subtract((PyLongObject *)left_o, (PyLongObject *)right_o); @@ -577,6 +599,8 @@ dummy_func( pure op(_BINARY_OP_MULTIPLY_FLOAT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = @@ -591,6 +615,8 @@ dummy_func( pure op(_BINARY_OP_ADD_FLOAT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = @@ -605,6 +631,8 @@ dummy_func( pure op(_BINARY_OP_SUBTRACT_FLOAT, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = @@ -634,6 +662,8 @@ dummy_func( pure op(_BINARY_OP_ADD_UNICODE, (left, right -- res)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = PyUnicode_Concat(left_o, right_o); @@ -656,6 +686,8 @@ dummy_func( op(_BINARY_OP_INPLACE_ADD_UNICODE, (left, right --)) { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); int next_oparg; #if TIER_ONE @@ -2448,7 +2480,7 @@ dummy_func( }; specializing op(_SPECIALIZE_COMPARE_OP, (counter/1, left, right -- left, right)) { - #if ENABLE_SPECIALIZATION + #if ENABLE_SPECIALIZATION_FT if (ADAPTIVE_COUNTER_TRIGGERS(counter)) { next_instr = this_instr; _Py_Specialize_CompareOp(left, right, next_instr, oparg); @@ -2924,10 +2956,8 @@ dummy_func( /* iterator ended normally */ assert(next_instr[oparg].op.code == END_FOR || next_instr[oparg].op.code == INSTRUMENTED_END_FOR); - PyStackRef_CLOSE(iter); - STACK_SHRINK(1); - /* Jump forward oparg, then skip following END_FOR and POP_TOP instruction */ - JUMPBY(oparg + 2); + /* Jump forward oparg, then skip following END_FOR */ + JUMPBY(oparg + 1); DISPATCH(); } next = PyStackRef_FromPyObjectSteal(next_o); @@ -2957,12 +2987,14 @@ dummy_func( macro(FOR_ITER) = _SPECIALIZE_FOR_ITER + _FOR_ITER; + inst(INSTRUMENTED_FOR_ITER, (unused/1 -- )) { _PyStackRef iter_stackref = TOP(); PyObject *iter = PyStackRef_AsPyObjectBorrow(iter_stackref); PyObject *next = (*Py_TYPE(iter)->tp_iternext)(iter); if (next != NULL) { PUSH(PyStackRef_FromPyObjectSteal(next)); + INSTRUMENTED_JUMP(this_instr, next_instr, PY_MONITORING_EVENT_BRANCH_LEFT); } else { if (_PyErr_Occurred(tstate)) { @@ -2976,14 +3008,12 @@ dummy_func( /* iterator ended normally */ assert(next_instr[oparg].op.code == END_FOR || next_instr[oparg].op.code == INSTRUMENTED_END_FOR); - STACK_SHRINK(1); - PyStackRef_CLOSE(iter_stackref); - /* Skip END_FOR and POP_TOP */ - _Py_CODEUNIT *target = next_instr + oparg + 2; - INSTRUMENTED_JUMP(this_instr, target, PY_MONITORING_EVENT_BRANCH_RIGHT); + /* Skip END_FOR */ + JUMPBY(oparg + 1); } } + op(_ITER_CHECK_LIST, (iter -- iter)) { EXIT_IF(Py_TYPE(PyStackRef_AsPyObjectBorrow(iter)) != &PyListIter_Type); } @@ -3002,10 +3032,8 @@ dummy_func( Py_DECREF(seq); } #endif - PyStackRef_CLOSE(iter); - STACK_SHRINK(1); - /* Jump forward oparg, then skip following END_FOR and POP_TOP instructions */ - JUMPBY(oparg + 2); + /* Jump forward oparg, then skip following END_FOR instruction */ + JUMPBY(oparg + 1); DISPATCH(); } } @@ -3054,10 +3082,8 @@ dummy_func( it->it_seq = NULL; Py_DECREF(seq); } - PyStackRef_CLOSE(iter); - STACK_SHRINK(1); - /* Jump forward oparg, then skip following END_FOR and POP_TOP instructions */ - JUMPBY(oparg + 2); + /* Jump forward oparg, then skip following END_FOR instruction */ + JUMPBY(oparg + 1); DISPATCH(); } } @@ -3098,10 +3124,8 @@ dummy_func( assert(Py_TYPE(r) == &PyRangeIter_Type); STAT_INC(FOR_ITER, hit); if (r->len <= 0) { - STACK_SHRINK(1); - PyStackRef_CLOSE(iter); - // Jump over END_FOR and POP_TOP instructions. - JUMPBY(oparg + 2); + // Jump over END_FOR instruction. + JUMPBY(oparg + 1); DISPATCH(); } } @@ -4779,7 +4803,8 @@ dummy_func( } inst(INSTRUMENTED_NOT_TAKEN, ( -- )) { - INSTRUMENTED_JUMP(this_instr, next_instr, PY_MONITORING_EVENT_BRANCH_LEFT); + (void)this_instr; // INSTRUMENTED_JUMP requires this_instr + INSTRUMENTED_JUMP(prev_instr, next_instr, PY_MONITORING_EVENT_BRANCH_LEFT); } macro(INSTRUMENTED_JUMP_BACKWARD) = diff --git a/Python/ceval_macros.h b/Python/ceval_macros.h index f15633fa467376..c37e1cf3afa60e 100644 --- a/Python/ceval_macros.h +++ b/Python/ceval_macros.h @@ -363,7 +363,7 @@ do { \ next_instr = dest; \ } else { \ _PyFrame_SetStackPointer(frame, stack_pointer); \ - next_instr = _Py_call_instrumentation_jump(tstate, event, frame, src, dest); \ + next_instr = _Py_call_instrumentation_jump(this_instr, tstate, event, frame, src, dest); \ stack_pointer = _PyFrame_GetStackPointer(frame); \ if (next_instr == NULL) { \ next_instr = (dest)+1; \ diff --git a/Python/codegen.c b/Python/codegen.c index 7432415b17414e..61707ba677097c 100644 --- a/Python/codegen.c +++ b/Python/codegen.c @@ -406,13 +406,7 @@ codegen_addop_j(instr_sequence *seq, location loc, assert(IS_JUMP_TARGET_LABEL(target)); assert(OPCODE_HAS_JUMP(opcode) || IS_BLOCK_PUSH_OPCODE(opcode)); assert(!IS_ASSEMBLER_OPCODE(opcode)); - if (_PyInstructionSequence_Addop(seq, opcode, target.id, loc) != SUCCESS) { - return ERROR; - } - if (IS_CONDITIONAL_JUMP_OPCODE(opcode) || opcode == FOR_ITER) { - return _PyInstructionSequence_Addop(seq, NOT_TAKEN, 0, NO_LOCATION); - } - return SUCCESS; + return _PyInstructionSequence_Addop(seq, opcode, target.id, loc); } #define ADDOP_JUMP(C, LOC, OP, O) \ @@ -2018,7 +2012,7 @@ codegen_for(compiler *c, stmt_ty s) * but a non-generator will jump to a later instruction. */ ADDOP(c, NO_LOCATION, END_FOR); - ADDOP(c, NO_LOCATION, POP_TOP); + ADDOP(c, NO_LOCATION, POP_ITER); _PyCompile_PopFBlock(c, COMPILE_FBLOCK_FOR_LOOP, start); @@ -4283,7 +4277,7 @@ codegen_sync_comprehension_generator(compiler *c, location loc, * but a non-generator will jump to a later instruction. */ ADDOP(c, NO_LOCATION, END_FOR); - ADDOP(c, NO_LOCATION, POP_TOP); + ADDOP(c, NO_LOCATION, POP_ITER); } return SUCCESS; diff --git a/Python/context.c b/Python/context.c index 95aa82206270f9..f30b59b9443bbf 100644 --- a/Python/context.c +++ b/Python/context.c @@ -419,6 +419,9 @@ class _contextvars.Context "PyContext *" "&PyContext_Type" /*[clinic end generated code: output=da39a3ee5e6b4b0d input=bdf87f8e0cb580e8]*/ +#define _PyContext_CAST(op) ((PyContext *)(op)) + + static inline PyContext * _context_alloc(void) { @@ -513,28 +516,30 @@ context_tp_new(PyTypeObject *type, PyObject *args, PyObject *kwds) } static int -context_tp_clear(PyContext *self) +context_tp_clear(PyObject *op) { + PyContext *self = _PyContext_CAST(op); Py_CLEAR(self->ctx_prev); Py_CLEAR(self->ctx_vars); return 0; } static int -context_tp_traverse(PyContext *self, visitproc visit, void *arg) +context_tp_traverse(PyObject *op, visitproc visit, void *arg) { + PyContext *self = _PyContext_CAST(op); Py_VISIT(self->ctx_prev); Py_VISIT(self->ctx_vars); return 0; } static void -context_tp_dealloc(PyContext *self) +context_tp_dealloc(PyObject *self) { _PyObject_GC_UNTRACK(self); - - if (self->ctx_weakreflist != NULL) { - PyObject_ClearWeakRefs((PyObject*)self); + PyContext *ctx = _PyContext_CAST(self); + if (ctx->ctx_weakreflist != NULL) { + PyObject_ClearWeakRefs(self); } (void)context_tp_clear(self); @@ -542,8 +547,9 @@ context_tp_dealloc(PyContext *self) } static PyObject * -context_tp_iter(PyContext *self) +context_tp_iter(PyObject *op) { + PyContext *self = _PyContext_CAST(op); return _PyHamt_NewIterKeys(self->ctx_vars); } @@ -575,18 +581,20 @@ context_tp_richcompare(PyObject *v, PyObject *w, int op) } static Py_ssize_t -context_tp_len(PyContext *self) +context_tp_len(PyObject *op) { + PyContext *self = _PyContext_CAST(op); return _PyHamt_Len(self->ctx_vars); } static PyObject * -context_tp_subscript(PyContext *self, PyObject *key) +context_tp_subscript(PyObject *op, PyObject *key) { if (context_check_key_type(key)) { return NULL; } PyObject *val = NULL; + PyContext *self = _PyContext_CAST(op); int found = _PyHamt_Find(self->ctx_vars, key, &val); if (found < 0) { return NULL; @@ -599,12 +607,13 @@ context_tp_subscript(PyContext *self, PyObject *key) } static int -context_tp_contains(PyContext *self, PyObject *key) +context_tp_contains(PyObject *op, PyObject *key) { if (context_check_key_type(key)) { return -1; } PyObject *val = NULL; + PyContext *self = _PyContext_CAST(op); return _PyHamt_Find(self->ctx_vars, key, &val); } @@ -701,7 +710,7 @@ _contextvars_Context_copy_impl(PyContext *self) static PyObject * -context_run(PyContext *self, PyObject *const *args, +context_run(PyObject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyThreadState *ts = _PyThreadState_GET(); @@ -712,14 +721,14 @@ context_run(PyContext *self, PyObject *const *args, return NULL; } - if (_PyContext_Enter(ts, (PyObject *)self)) { + if (_PyContext_Enter(ts, self)) { return NULL; } PyObject *call_result = _PyObject_VectorcallTstate( ts, args[0], args + 1, nargs - 1, kwnames); - if (_PyContext_Exit(ts, (PyObject *)self)) { + if (_PyContext_Exit(ts, self)) { Py_XDECREF(call_result); return NULL; } @@ -739,21 +748,12 @@ static PyMethodDef PyContext_methods[] = { }; static PySequenceMethods PyContext_as_sequence = { - 0, /* sq_length */ - 0, /* sq_concat */ - 0, /* sq_repeat */ - 0, /* sq_item */ - 0, /* sq_slice */ - 0, /* sq_ass_item */ - 0, /* sq_ass_slice */ - (objobjproc)context_tp_contains, /* sq_contains */ - 0, /* sq_inplace_concat */ - 0, /* sq_inplace_repeat */ + .sq_contains = context_tp_contains }; static PyMappingMethods PyContext_as_mapping = { - (lenfunc)context_tp_len, /* mp_length */ - (binaryfunc)context_tp_subscript, /* mp_subscript */ + .mp_length = context_tp_len, + .mp_subscript = context_tp_subscript }; PyTypeObject PyContext_Type = { @@ -763,13 +763,13 @@ PyTypeObject PyContext_Type = { .tp_methods = PyContext_methods, .tp_as_mapping = &PyContext_as_mapping, .tp_as_sequence = &PyContext_as_sequence, - .tp_iter = (getiterfunc)context_tp_iter, - .tp_dealloc = (destructor)context_tp_dealloc, + .tp_iter = context_tp_iter, + .tp_dealloc = context_tp_dealloc, .tp_getattro = PyObject_GenericGetAttr, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, .tp_richcompare = context_tp_richcompare, - .tp_traverse = (traverseproc)context_tp_traverse, - .tp_clear = (inquiry)context_tp_clear, + .tp_traverse = context_tp_traverse, + .tp_clear = context_tp_clear, .tp_new = context_tp_new, .tp_weaklistoffset = offsetof(PyContext, ctx_weakreflist), .tp_hash = PyObject_HashNotImplemented, @@ -909,6 +909,9 @@ class _contextvars.ContextVar "PyContextVar *" "&PyContextVar_Type" /*[clinic end generated code: output=da39a3ee5e6b4b0d input=445da935fa8883c3]*/ +#define _PyContextVar_CAST(op) ((PyContextVar *)(op)) + + static PyObject * contextvar_tp_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { @@ -926,8 +929,9 @@ contextvar_tp_new(PyTypeObject *type, PyObject *args, PyObject *kwds) } static int -contextvar_tp_clear(PyContextVar *self) +contextvar_tp_clear(PyObject *op) { + PyContextVar *self = _PyContextVar_CAST(op); Py_CLEAR(self->var_name); Py_CLEAR(self->var_default); #ifndef Py_GIL_DISABLED @@ -939,15 +943,16 @@ contextvar_tp_clear(PyContextVar *self) } static int -contextvar_tp_traverse(PyContextVar *self, visitproc visit, void *arg) +contextvar_tp_traverse(PyObject *op, visitproc visit, void *arg) { + PyContextVar *self = _PyContextVar_CAST(op); Py_VISIT(self->var_name); Py_VISIT(self->var_default); return 0; } static void -contextvar_tp_dealloc(PyContextVar *self) +contextvar_tp_dealloc(PyObject *self) { PyObject_GC_UnTrack(self); (void)contextvar_tp_clear(self); @@ -955,14 +960,16 @@ contextvar_tp_dealloc(PyContextVar *self) } static Py_hash_t -contextvar_tp_hash(PyContextVar *self) +contextvar_tp_hash(PyObject *op) { + PyContextVar *self = _PyContextVar_CAST(op); return self->var_hash; } static PyObject * -contextvar_tp_repr(PyContextVar *self) +contextvar_tp_repr(PyObject *op) { + PyContextVar *self = _PyContextVar_CAST(op); // Estimation based on the shortest name and default value, // but maximize the pointer size. // "" @@ -1106,15 +1113,15 @@ PyTypeObject PyContextVar_Type = { sizeof(PyContextVar), .tp_methods = PyContextVar_methods, .tp_members = PyContextVar_members, - .tp_dealloc = (destructor)contextvar_tp_dealloc, + .tp_dealloc = contextvar_tp_dealloc, .tp_getattro = PyObject_GenericGetAttr, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, - .tp_traverse = (traverseproc)contextvar_tp_traverse, - .tp_clear = (inquiry)contextvar_tp_clear, + .tp_traverse = contextvar_tp_traverse, + .tp_clear = contextvar_tp_clear, .tp_new = contextvar_tp_new, .tp_free = PyObject_GC_Del, - .tp_hash = (hashfunc)contextvar_tp_hash, - .tp_repr = (reprfunc)contextvar_tp_repr, + .tp_hash = contextvar_tp_hash, + .tp_repr = contextvar_tp_repr, }; @@ -1129,6 +1136,9 @@ class _contextvars.Token "PyContextToken *" "&PyContextToken_Type" /*[clinic end generated code: output=da39a3ee5e6b4b0d input=338a5e2db13d3f5b]*/ +#define _PyContextToken_CAST(op) ((PyContextToken *)(op)) + + static PyObject * token_tp_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { @@ -1138,8 +1148,9 @@ token_tp_new(PyTypeObject *type, PyObject *args, PyObject *kwds) } static int -token_tp_clear(PyContextToken *self) +token_tp_clear(PyObject *op) { + PyContextToken *self = _PyContextToken_CAST(op); Py_CLEAR(self->tok_ctx); Py_CLEAR(self->tok_var); Py_CLEAR(self->tok_oldval); @@ -1147,8 +1158,9 @@ token_tp_clear(PyContextToken *self) } static int -token_tp_traverse(PyContextToken *self, visitproc visit, void *arg) +token_tp_traverse(PyObject *op, visitproc visit, void *arg) { + PyContextToken *self = _PyContextToken_CAST(op); Py_VISIT(self->tok_ctx); Py_VISIT(self->tok_var); Py_VISIT(self->tok_oldval); @@ -1156,7 +1168,7 @@ token_tp_traverse(PyContextToken *self, visitproc visit, void *arg) } static void -token_tp_dealloc(PyContextToken *self) +token_tp_dealloc(PyObject *self) { PyObject_GC_UnTrack(self); (void)token_tp_clear(self); @@ -1164,8 +1176,9 @@ token_tp_dealloc(PyContextToken *self) } static PyObject * -token_tp_repr(PyContextToken *self) +token_tp_repr(PyObject *op) { + PyContextToken *self = _PyContextToken_CAST(op); PyUnicodeWriter *writer = PyUnicodeWriter_Create(0); if (writer == NULL) { return NULL; @@ -1195,14 +1208,16 @@ token_tp_repr(PyContextToken *self) } static PyObject * -token_get_var(PyContextToken *self, void *Py_UNUSED(ignored)) +token_get_var(PyObject *op, void *Py_UNUSED(ignored)) { + PyContextToken *self = _PyContextToken_CAST(op); return Py_NewRef(self->tok_var);; } static PyObject * -token_get_old_value(PyContextToken *self, void *Py_UNUSED(ignored)) +token_get_old_value(PyObject *op, void *Py_UNUSED(ignored)) { + PyContextToken *self = _PyContextToken_CAST(op); if (self->tok_oldval == NULL) { return get_token_missing(); } @@ -1211,8 +1226,8 @@ token_get_old_value(PyContextToken *self, void *Py_UNUSED(ignored)) } static PyGetSetDef PyContextTokenType_getsetlist[] = { - {"var", (getter)token_get_var, NULL, NULL}, - {"old_value", (getter)token_get_old_value, NULL, NULL}, + {"var", token_get_var, NULL, NULL}, + {"old_value", token_get_old_value, NULL, NULL}, {NULL} }; @@ -1228,15 +1243,15 @@ PyTypeObject PyContextToken_Type = { sizeof(PyContextToken), .tp_methods = PyContextTokenType_methods, .tp_getset = PyContextTokenType_getsetlist, - .tp_dealloc = (destructor)token_tp_dealloc, + .tp_dealloc = token_tp_dealloc, .tp_getattro = PyObject_GenericGetAttr, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, - .tp_traverse = (traverseproc)token_tp_traverse, - .tp_clear = (inquiry)token_tp_clear, + .tp_traverse = token_tp_traverse, + .tp_clear = token_tp_clear, .tp_new = token_tp_new, .tp_free = PyObject_GC_Del, .tp_hash = PyObject_HashNotImplemented, - .tp_repr = (reprfunc)token_tp_repr, + .tp_repr = token_tp_repr, }; static PyContextToken * @@ -1270,7 +1285,7 @@ context_token_missing_tp_repr(PyObject *self) } static void -context_token_missing_tp_dealloc(_PyContextTokenMissing *Py_UNUSED(self)) +context_token_missing_tp_dealloc(PyObject *Py_UNUSED(self)) { #ifdef Py_DEBUG /* The singleton is statically allocated. */ @@ -1285,7 +1300,7 @@ PyTypeObject _PyContextTokenMissing_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "Token.MISSING", sizeof(_PyContextTokenMissing), - .tp_dealloc = (destructor)context_token_missing_tp_dealloc, + .tp_dealloc = context_token_missing_tp_dealloc, .tp_getattro = PyObject_GenericGetAttr, .tp_flags = Py_TPFLAGS_DEFAULT, .tp_repr = context_token_missing_tp_repr, diff --git a/Python/errors.c b/Python/errors.c index 2d362c1864ffff..b6ac2f767a283b 100644 --- a/Python/errors.c +++ b/Python/errors.c @@ -1981,7 +1981,7 @@ _PyErr_ProgramDecodedTextObject(PyObject *filename, int lineno, const char* enco return NULL; } - FILE *fp = _Py_fopen_obj(filename, "r" PY_STDIOTEXTMODE); + FILE *fp = Py_fopen(filename, "r" PY_STDIOTEXTMODE); if (fp == NULL) { PyErr_Clear(); return NULL; diff --git a/Python/executor_cases.c.h b/Python/executor_cases.c.h index f7374d52705960..e40fa88be89172 100644 --- a/Python/executor_cases.c.h +++ b/Python/executor_cases.c.h @@ -411,6 +411,20 @@ break; } + case _END_FOR: { + _PyStackRef value; + value = stack_pointer[-1]; + /* Don't update instr_ptr, so that POP_ITER sees + * the FOR_ITER as the previous instruction. + * This has the benign side effect that if value is + * finalized it will see the location as the FOR_ITER's. + */ + PyStackRef_CLOSE(value); + stack_pointer += -1; + assert(WITHIN_STACK_BOUNDS()); + break; + } + case _END_SEND: { _PyStackRef value; _PyStackRef receiver; @@ -624,6 +638,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Multiply((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); @@ -644,6 +660,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Add((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); @@ -664,6 +682,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Subtract((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); @@ -724,6 +744,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval * @@ -745,6 +767,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval + @@ -766,6 +790,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval - @@ -805,6 +831,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = PyUnicode_Concat(left_o, right_o); PyStackRef_CLOSE_SPECIALIZED(left, _PyUnicode_ExactDealloc); @@ -824,6 +852,8 @@ left = stack_pointer[-2]; PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); int next_oparg; #if TIER_ONE assert(next_instr->op.code == STORE_FAST); diff --git a/Python/fileutils.c b/Python/fileutils.c index 81276651f6df44..6bc3a44c3c1313 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -1748,8 +1748,10 @@ _Py_wfopen(const wchar_t *path, const wchar_t *mode) } -/* Open a file. Call _wfopen() on Windows, or encode the path to the filesystem - encoding and call fopen() otherwise. +/* Open a file. + + On Windows, if 'path' is a Unicode string, call _wfopen(). Otherwise, encode + the path to the filesystem encoding and call fopen(). Return the new file object on success. Raise an exception and return NULL on error. @@ -1762,32 +1764,32 @@ _Py_wfopen(const wchar_t *path, const wchar_t *mode) Release the GIL to call _wfopen() or fopen(). The caller must hold the GIL. */ FILE* -_Py_fopen_obj(PyObject *path, const char *mode) +Py_fopen(PyObject *path, const char *mode) { - FILE *f; - int async_err = 0; -#ifdef MS_WINDOWS - wchar_t wmode[10]; - int usize; - assert(PyGILState_Check()); if (PySys_Audit("open", "Osi", path, mode, 0) < 0) { return NULL; } - if (!PyUnicode_Check(path)) { - PyErr_Format(PyExc_TypeError, - "str file path expected under Windows, got %R", - Py_TYPE(path)); + + FILE *f; + int async_err = 0; + int saved_errno; +#ifdef MS_WINDOWS + PyObject *unicode; + if (!PyUnicode_FSDecoder(path, &unicode)) { return NULL; } - wchar_t *wpath = PyUnicode_AsWideCharString(path, NULL); - if (wpath == NULL) + wchar_t *wpath = PyUnicode_AsWideCharString(unicode, NULL); + Py_DECREF(unicode); + if (wpath == NULL) { return NULL; + } - usize = MultiByteToWideChar(CP_ACP, 0, mode, -1, - wmode, Py_ARRAY_LENGTH(wmode)); + wchar_t wmode[10]; + int usize = MultiByteToWideChar(CP_ACP, 0, mode, -1, + wmode, Py_ARRAY_LENGTH(wmode)); if (usize == 0) { PyErr_SetFromWindowsErr(0); PyMem_Free(wpath); @@ -1796,26 +1798,20 @@ _Py_fopen_obj(PyObject *path, const char *mode) do { Py_BEGIN_ALLOW_THREADS + _Py_BEGIN_SUPPRESS_IPH f = _wfopen(wpath, wmode); + _Py_END_SUPPRESS_IPH Py_END_ALLOW_THREADS } while (f == NULL && errno == EINTR && !(async_err = PyErr_CheckSignals())); - int saved_errno = errno; + saved_errno = errno; PyMem_Free(wpath); #else PyObject *bytes; - const char *path_bytes; - - assert(PyGILState_Check()); - - if (!PyUnicode_FSConverter(path, &bytes)) - return NULL; - path_bytes = PyBytes_AS_STRING(bytes); - - if (PySys_Audit("open", "Osi", path, mode, 0) < 0) { - Py_DECREF(bytes); + if (!PyUnicode_FSConverter(path, &bytes)) { return NULL; } + const char *path_bytes = PyBytes_AS_STRING(bytes); do { Py_BEGIN_ALLOW_THREADS @@ -1823,11 +1819,13 @@ _Py_fopen_obj(PyObject *path, const char *mode) Py_END_ALLOW_THREADS } while (f == NULL && errno == EINTR && !(async_err = PyErr_CheckSignals())); - int saved_errno = errno; + saved_errno = errno; Py_DECREF(bytes); #endif - if (async_err) + + if (async_err) { return NULL; + } if (f == NULL) { errno = saved_errno; @@ -1842,6 +1840,27 @@ _Py_fopen_obj(PyObject *path, const char *mode) return f; } + +// Deprecated alias to Py_fopen() kept for backward compatibility +FILE* +_Py_fopen_obj(PyObject *path, const char *mode) +{ + return Py_fopen(path, mode); +} + + +// Call fclose(). +// +// On Windows, files opened by Py_fopen() in the Python DLL must be closed by +// the Python DLL to use the same C runtime version. Otherwise, calling +// fclose() directly can cause undefined behavior. +int +Py_fclose(FILE *file) +{ + return fclose(file); +} + + /* Read count bytes from fd into buf. On success, return the number of read bytes, it can be lower than count. diff --git a/Python/flowgraph.c b/Python/flowgraph.c index 64df6290de06ba..24561c1ee04db9 100644 --- a/Python/flowgraph.c +++ b/Python/flowgraph.c @@ -4,6 +4,7 @@ #include "Python.h" #include "pycore_flowgraph.h" #include "pycore_compile.h" +#include "pycore_intrinsics.h" #include "pycore_pymem.h" // _PyMem_IsPtrFreed() #include "pycore_opcode_utils.h" @@ -522,14 +523,15 @@ no_redundant_jumps(cfg_builder *g) { static int normalize_jumps_in_block(cfg_builder *g, basicblock *b) { cfg_instr *last = basicblock_last_instr(b); - if (last == NULL || !is_jump(last) || - IS_UNCONDITIONAL_JUMP_OPCODE(last->i_opcode)) { + if (last == NULL || !IS_CONDITIONAL_JUMP_OPCODE(last->i_opcode)) { return SUCCESS; } assert(!IS_ASSEMBLER_OPCODE(last->i_opcode)); bool is_forward = last->i_target->b_visited == 0; if (is_forward) { + RETURN_IF_ERROR( + basicblock_addop(b, NOT_TAKEN, 0, last->i_loc)); return SUCCESS; } @@ -557,10 +559,6 @@ normalize_jumps_in_block(cfg_builder *g, basicblock *b) { if (backwards_jump == NULL) { return ERROR; } - assert(b->b_next->b_iused > 0); - assert(b->b_next->b_instr[0].i_opcode == NOT_TAKEN); - b->b_next->b_instr[0].i_opcode = NOP; - b->b_next->b_instr[0].i_loc = NO_LOCATION; RETURN_IF_ERROR( basicblock_addop(backwards_jump, NOT_TAKEN, 0, last->i_loc)); RETURN_IF_ERROR( @@ -1877,6 +1875,12 @@ optimize_basic_block(PyObject *const_cache, basicblock *bb, PyObject *consts) continue; } break; + case CALL_INTRINSIC_1: + // for _ in (*foo, *bar) -> for _ in [*foo, *bar] + if (oparg == INTRINSIC_LIST_TO_TUPLE && nextop == GET_ITER) { + INSTR_SET_OP0(inst, NOP); + } + break; } } diff --git a/Python/generated_cases.c.h b/Python/generated_cases.c.h index 98743c27c38524..7028ba52faae96 100644 --- a/Python/generated_cases.c.h +++ b/Python/generated_cases.c.h @@ -80,6 +80,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval + @@ -116,6 +118,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Add((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); @@ -151,6 +155,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = PyUnicode_Concat(left_o, right_o); PyStackRef_CLOSE_SPECIALIZED(left, _PyUnicode_ExactDealloc); @@ -185,6 +191,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyUnicode_CheckExact(left_o)); + assert(PyUnicode_CheckExact(right_o)); int next_oparg; #if TIER_ONE assert(next_instr->op.code == STORE_FAST); @@ -247,6 +255,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval * @@ -283,6 +293,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Multiply((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); @@ -318,6 +330,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyFloat_CheckExact(left_o)); + assert(PyFloat_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); double dres = ((PyFloatObject *)left_o)->ob_fval - @@ -354,6 +368,8 @@ { PyObject *left_o = PyStackRef_AsPyObjectBorrow(left); PyObject *right_o = PyStackRef_AsPyObjectBorrow(right); + assert(PyLong_CheckExact(left_o)); + assert(PyLong_CheckExact(right_o)); STAT_INC(BINARY_OP, hit); PyObject *res_o = _PyLong_Subtract((PyLongObject *)left_o, (PyLongObject *)right_o); PyStackRef_CLOSE_SPECIALIZED(right, _PyLong_ExactDealloc); @@ -3229,7 +3245,7 @@ left = stack_pointer[-2]; uint16_t counter = read_u16(&this_instr[1].cache); (void)counter; - #if ENABLE_SPECIALIZATION + #if ENABLE_SPECIALIZATION_FT if (ADAPTIVE_COUNTER_TRIGGERS(counter)) { next_instr = this_instr; _PyFrame_SetStackPointer(frame, stack_pointer); @@ -3768,11 +3784,15 @@ } TARGET(END_FOR) { - frame->instr_ptr = next_instr; next_instr += 1; INSTRUCTION_STATS(END_FOR); _PyStackRef value; value = stack_pointer[-1]; + /* Don't update instr_ptr, so that POP_ITER sees + * the FOR_ITER as the previous instruction. + * This has the benign side effect that if value is + * finalized it will see the location as the FOR_ITER's. + */ PyStackRef_CLOSE(value); stack_pointer += -1; assert(WITHIN_STACK_BOUNDS()); @@ -3957,10 +3977,8 @@ /* iterator ended normally */ assert(next_instr[oparg].op.code == END_FOR || next_instr[oparg].op.code == INSTRUMENTED_END_FOR); - PyStackRef_CLOSE(iter); - STACK_SHRINK(1); - /* Jump forward oparg, then skip following END_FOR and POP_TOP instruction */ - JUMPBY(oparg + 2); + /* Jump forward oparg, then skip following END_FOR */ + JUMPBY(oparg + 1); DISPATCH(); } next = PyStackRef_FromPyObjectSteal(next_o); @@ -4048,10 +4066,8 @@ Py_DECREF(seq); } #endif - PyStackRef_CLOSE(iter); - STACK_SHRINK(1); - /* Jump forward oparg, then skip following END_FOR and POP_TOP instructions */ - JUMPBY(oparg + 2); + /* Jump forward oparg, then skip following END_FOR instruction */ + JUMPBY(oparg + 1); DISPATCH(); } } @@ -4091,10 +4107,8 @@ assert(Py_TYPE(r) == &PyRangeIter_Type); STAT_INC(FOR_ITER, hit); if (r->len <= 0) { - STACK_SHRINK(1); - PyStackRef_CLOSE(iter); - // Jump over END_FOR and POP_TOP instructions. - JUMPBY(oparg + 2); + // Jump over END_FOR instruction. + JUMPBY(oparg + 1); DISPATCH(); } } @@ -4141,10 +4155,8 @@ it->it_seq = NULL; Py_DECREF(seq); } - PyStackRef_CLOSE(iter); - STACK_SHRINK(1); - /* Jump forward oparg, then skip following END_FOR and POP_TOP instructions */ - JUMPBY(oparg + 2); + /* Jump forward oparg, then skip following END_FOR instruction */ + JUMPBY(oparg + 1); DISPATCH(); } } @@ -4572,7 +4584,7 @@ } TARGET(INSTRUMENTED_END_FOR) { - _Py_CODEUNIT* const this_instr = frame->instr_ptr = next_instr; + _Py_CODEUNIT* const this_instr = next_instr; (void)this_instr; next_instr += 1; INSTRUCTION_STATS(INSTRUMENTED_END_FOR); @@ -4636,6 +4648,7 @@ stack_pointer = _PyFrame_GetStackPointer(frame); if (next != NULL) { PUSH(PyStackRef_FromPyObjectSteal(next)); + INSTRUMENTED_JUMP(this_instr, next_instr, PY_MONITORING_EVENT_BRANCH_LEFT); } else { if (_PyErr_Occurred(tstate)) { @@ -4653,11 +4666,8 @@ /* iterator ended normally */ assert(next_instr[oparg].op.code == END_FOR || next_instr[oparg].op.code == INSTRUMENTED_END_FOR); - STACK_SHRINK(1); - PyStackRef_CLOSE(iter_stackref); - /* Skip END_FOR and POP_TOP */ - _Py_CODEUNIT *target = next_instr + oparg + 2; - INSTRUMENTED_JUMP(this_instr, target, PY_MONITORING_EVENT_BRANCH_RIGHT); + /* Skip END_FOR */ + JUMPBY(oparg + 1); } DISPATCH(); } @@ -4764,11 +4774,28 @@ } TARGET(INSTRUMENTED_NOT_TAKEN) { + _Py_CODEUNIT* const prev_instr = frame->instr_ptr; _Py_CODEUNIT* const this_instr = frame->instr_ptr = next_instr; (void)this_instr; next_instr += 1; INSTRUCTION_STATS(INSTRUMENTED_NOT_TAKEN); - INSTRUMENTED_JUMP(this_instr, next_instr, PY_MONITORING_EVENT_BRANCH_LEFT); + (void)this_instr; // INSTRUMENTED_JUMP requires this_instr + INSTRUMENTED_JUMP(prev_instr, next_instr, PY_MONITORING_EVENT_BRANCH_LEFT); + DISPATCH(); + } + + TARGET(INSTRUMENTED_POP_ITER) { + _Py_CODEUNIT* const prev_instr = frame->instr_ptr; + _Py_CODEUNIT* const this_instr = frame->instr_ptr = next_instr; + (void)this_instr; + next_instr += 1; + INSTRUCTION_STATS(INSTRUMENTED_POP_ITER); + _PyStackRef iter; + iter = stack_pointer[-1]; + INSTRUMENTED_JUMP(prev_instr, this_instr+1, PY_MONITORING_EVENT_BRANCH_RIGHT); + PyStackRef_CLOSE(iter); + stack_pointer += -1; + assert(WITHIN_STACK_BOUNDS()); DISPATCH(); } @@ -6693,6 +6720,18 @@ DISPATCH(); } + TARGET(POP_ITER) { + frame->instr_ptr = next_instr; + next_instr += 1; + INSTRUCTION_STATS(POP_ITER); + _PyStackRef value; + value = stack_pointer[-1]; + PyStackRef_CLOSE(value); + stack_pointer += -1; + assert(WITHIN_STACK_BOUNDS()); + DISPATCH(); + } + TARGET(POP_JUMP_IF_FALSE) { _Py_CODEUNIT* const this_instr = frame->instr_ptr = next_instr; (void)this_instr; diff --git a/Python/hamt.c b/Python/hamt.c index cfd211f4541446..ed43a0449d7a01 100644 --- a/Python/hamt.c +++ b/Python/hamt.c @@ -319,6 +319,8 @@ typedef struct { Py_ssize_t a_count; } PyHamtNode_Array; +#define _PyHamtNode_Array_CAST(op) ((PyHamtNode_Array *)(op)) + typedef struct { PyObject_VAR_HEAD @@ -326,6 +328,8 @@ typedef struct { PyObject *c_array[1]; } PyHamtNode_Collision; +#define _PyHamtNode_Collision_CAST(op) ((PyHamtNode_Collision *)(op)) + static PyHamtObject * hamt_alloc(void); @@ -479,6 +483,8 @@ _hamt_dump_ident(PyUnicodeWriter *writer, int level) #endif /* Py_DEBUG */ /////////////////////////////////// Bitmap Node +#define _PyHamtNode_Bitmap_CAST(op) ((PyHamtNode_Bitmap *)(op)) + static PyHamtNode * hamt_node_bitmap_new(Py_ssize_t size) @@ -1083,30 +1089,27 @@ hamt_node_bitmap_find(PyHamtNode_Bitmap *self, } static int -hamt_node_bitmap_traverse(PyHamtNode_Bitmap *self, visitproc visit, void *arg) +hamt_node_bitmap_traverse(PyObject *op, visitproc visit, void *arg) { /* Bitmap's tp_traverse */ - - Py_ssize_t i; - - for (i = Py_SIZE(self); --i >= 0; ) { + PyHamtNode_Bitmap *self = _PyHamtNode_Bitmap_CAST(op); + for (Py_ssize_t i = Py_SIZE(self); --i >= 0;) { Py_VISIT(self->b_array[i]); } - return 0; } static void -hamt_node_bitmap_dealloc(PyHamtNode_Bitmap *self) +hamt_node_bitmap_dealloc(PyObject *self) { /* Bitmap's tp_dealloc */ - Py_ssize_t len = Py_SIZE(self); - Py_ssize_t i; + PyHamtNode_Bitmap *node = _PyHamtNode_Bitmap_CAST(self); + Py_ssize_t i, len = Py_SIZE(self); - if (Py_SIZE(self) == 0) { + if (len == 0) { /* The empty node is statically allocated. */ - assert(self == &_Py_SINGLETON(hamt_bitmap_node_empty)); + assert(node == &_Py_SINGLETON(hamt_bitmap_node_empty)); #ifdef Py_DEBUG _Py_FatalRefcountError("deallocating the empty hamt node bitmap singleton"); #else @@ -1120,11 +1123,11 @@ hamt_node_bitmap_dealloc(PyHamtNode_Bitmap *self) if (len > 0) { i = len; while (--i >= 0) { - Py_XDECREF(self->b_array[i]); + Py_XDECREF(node->b_array[i]); } } - Py_TYPE(self)->tp_free((PyObject *)self); + Py_TYPE(self)->tp_free(self); Py_TRASHCAN_END } @@ -1489,38 +1492,30 @@ hamt_node_collision_find(PyHamtNode_Collision *self, static int -hamt_node_collision_traverse(PyHamtNode_Collision *self, - visitproc visit, void *arg) +hamt_node_collision_traverse(PyObject *op, visitproc visit, void *arg) { /* Collision's tp_traverse */ - - Py_ssize_t i; - - for (i = Py_SIZE(self); --i >= 0; ) { + PyHamtNode_Collision *self = _PyHamtNode_Collision_CAST(op); + for (Py_ssize_t i = Py_SIZE(self); --i >= 0; ) { Py_VISIT(self->c_array[i]); } - return 0; } static void -hamt_node_collision_dealloc(PyHamtNode_Collision *self) +hamt_node_collision_dealloc(PyObject *self) { /* Collision's tp_dealloc */ - Py_ssize_t len = Py_SIZE(self); - PyObject_GC_UnTrack(self); Py_TRASHCAN_BEGIN(self, hamt_node_collision_dealloc) - if (len > 0) { - + PyHamtNode_Collision *node = _PyHamtNode_Collision_CAST(self); while (--len >= 0) { - Py_XDECREF(self->c_array[len]); + Py_XDECREF(node->c_array[len]); } } - - Py_TYPE(self)->tp_free((PyObject *)self); + Py_TYPE(self)->tp_free(self); Py_TRASHCAN_END } @@ -1868,35 +1863,27 @@ hamt_node_array_find(PyHamtNode_Array *self, } static int -hamt_node_array_traverse(PyHamtNode_Array *self, - visitproc visit, void *arg) +hamt_node_array_traverse(PyObject *op, visitproc visit, void *arg) { /* Array's tp_traverse */ - - Py_ssize_t i; - - for (i = 0; i < HAMT_ARRAY_NODE_SIZE; i++) { + PyHamtNode_Array *self = _PyHamtNode_Array_CAST(op); + for (Py_ssize_t i = 0; i < HAMT_ARRAY_NODE_SIZE; i++) { Py_VISIT(self->a_array[i]); } - return 0; } static void -hamt_node_array_dealloc(PyHamtNode_Array *self) +hamt_node_array_dealloc(PyObject *self) { /* Array's tp_dealloc */ - - Py_ssize_t i; - PyObject_GC_UnTrack(self); Py_TRASHCAN_BEGIN(self, hamt_node_array_dealloc) - - for (i = 0; i < HAMT_ARRAY_NODE_SIZE; i++) { - Py_XDECREF(self->a_array[i]); + PyHamtNode_Array *obj = _PyHamtNode_Array_CAST(self); + for (Py_ssize_t i = 0; i < HAMT_ARRAY_NODE_SIZE; i++) { + Py_XDECREF(obj->a_array[i]); } - - Py_TYPE(self)->tp_free((PyObject *)self); + Py_TYPE(self)->tp_free(self); Py_TRASHCAN_END } @@ -2605,6 +2592,8 @@ static PyObject * hamt_dump(PyHamtObject *self); #endif +#define _PyHamtObject_CAST(op) ((PyHamtObject *)(op)) + static PyObject * hamt_tp_new(PyTypeObject *type, PyObject *args, PyObject *kwds) @@ -2613,24 +2602,27 @@ hamt_tp_new(PyTypeObject *type, PyObject *args, PyObject *kwds) } static int -hamt_tp_clear(PyHamtObject *self) +hamt_tp_clear(PyObject *op) { + PyHamtObject *self = _PyHamtObject_CAST(op); Py_CLEAR(self->h_root); return 0; } static int -hamt_tp_traverse(PyHamtObject *self, visitproc visit, void *arg) +hamt_tp_traverse(PyObject *op, visitproc visit, void *arg) { + PyHamtObject *self = _PyHamtObject_CAST(op); Py_VISIT(self->h_root); return 0; } static void -hamt_tp_dealloc(PyHamtObject *self) +hamt_tp_dealloc(PyObject *self) { - if (self == _empty_hamt) { + PyHamtObject *obj = _PyHamtObject_CAST(self); + if (obj == _empty_hamt) { /* The empty one is statically allocated. */ #ifdef Py_DEBUG _Py_FatalRefcountError("deallocating the empty hamt singleton"); @@ -2640,8 +2632,8 @@ hamt_tp_dealloc(PyHamtObject *self) } PyObject_GC_UnTrack(self); - if (self->h_weakreflist != NULL) { - PyObject_ClearWeakRefs((PyObject*)self); + if (obj->h_weakreflist != NULL) { + PyObject_ClearWeakRefs(self); } (void)hamt_tp_clear(self); Py_TYPE(self)->tp_free(self); @@ -2673,16 +2665,18 @@ hamt_tp_richcompare(PyObject *v, PyObject *w, int op) } static int -hamt_tp_contains(PyHamtObject *self, PyObject *key) +hamt_tp_contains(PyObject *op, PyObject *key) { PyObject *val; + PyHamtObject *self = _PyHamtObject_CAST(op); return _PyHamt_Find(self, key, &val); } static PyObject * -hamt_tp_subscript(PyHamtObject *self, PyObject *key) +hamt_tp_subscript(PyObject *op, PyObject *key) { PyObject *val; + PyHamtObject *self = _PyHamtObject_CAST(op); hamt_find_t res = hamt_find(self, key, &val); switch (res) { case F_ERROR: @@ -2698,19 +2692,21 @@ hamt_tp_subscript(PyHamtObject *self, PyObject *key) } static Py_ssize_t -hamt_tp_len(PyHamtObject *self) +hamt_tp_len(PyObject *op) { + PyHamtObject *self = _PyHamtObject_CAST(op); return _PyHamt_Len(self); } static PyObject * -hamt_tp_iter(PyHamtObject *self) +hamt_tp_iter(PyObject *op) { + PyHamtObject *self = _PyHamtObject_CAST(op); return _PyHamt_NewIterKeys(self); } static PyObject * -hamt_py_set(PyHamtObject *self, PyObject *args) +hamt_py_set(PyObject *op, PyObject *args) { PyObject *key; PyObject *val; @@ -2719,11 +2715,12 @@ hamt_py_set(PyHamtObject *self, PyObject *args) return NULL; } + PyHamtObject *self = _PyHamtObject_CAST(op); return (PyObject *)_PyHamt_Assoc(self, key, val); } static PyObject * -hamt_py_get(PyHamtObject *self, PyObject *args) +hamt_py_get(PyObject *op, PyObject *args) { PyObject *key; PyObject *def = NULL; @@ -2733,6 +2730,7 @@ hamt_py_get(PyHamtObject *self, PyObject *args) } PyObject *val = NULL; + PyHamtObject *self = _PyHamtObject_CAST(op); hamt_find_t res = hamt_find(self, key, &val); switch (res) { case F_ERROR: @@ -2750,67 +2748,63 @@ hamt_py_get(PyHamtObject *self, PyObject *args) } static PyObject * -hamt_py_delete(PyHamtObject *self, PyObject *key) +hamt_py_delete(PyObject *op, PyObject *key) { + PyHamtObject *self = _PyHamtObject_CAST(op); return (PyObject *)_PyHamt_Without(self, key); } static PyObject * -hamt_py_items(PyHamtObject *self, PyObject *args) +hamt_py_items(PyObject *op, PyObject *args) { + PyHamtObject *self = _PyHamtObject_CAST(op); return _PyHamt_NewIterItems(self); } static PyObject * -hamt_py_values(PyHamtObject *self, PyObject *args) +hamt_py_values(PyObject *op, PyObject *args) { + PyHamtObject *self = _PyHamtObject_CAST(op); return _PyHamt_NewIterValues(self); } static PyObject * -hamt_py_keys(PyHamtObject *self, PyObject *Py_UNUSED(args)) +hamt_py_keys(PyObject *op, PyObject *Py_UNUSED(args)) { + PyHamtObject *self = _PyHamtObject_CAST(op); return _PyHamt_NewIterKeys(self); } #ifdef Py_DEBUG static PyObject * -hamt_py_dump(PyHamtObject *self, PyObject *Py_UNUSED(args)) +hamt_py_dump(PyObject *op, PyObject *Py_UNUSED(args)) { + PyHamtObject *self = _PyHamtObject_CAST(op); return hamt_dump(self); } #endif static PyMethodDef PyHamt_methods[] = { - {"set", _PyCFunction_CAST(hamt_py_set), METH_VARARGS, NULL}, - {"get", _PyCFunction_CAST(hamt_py_get), METH_VARARGS, NULL}, - {"delete", _PyCFunction_CAST(hamt_py_delete), METH_O, NULL}, - {"items", _PyCFunction_CAST(hamt_py_items), METH_NOARGS, NULL}, - {"keys", _PyCFunction_CAST(hamt_py_keys), METH_NOARGS, NULL}, - {"values", _PyCFunction_CAST(hamt_py_values), METH_NOARGS, NULL}, + {"set", hamt_py_set, METH_VARARGS, NULL}, + {"get", hamt_py_get, METH_VARARGS, NULL}, + {"delete", hamt_py_delete, METH_O, NULL}, + {"items", hamt_py_items, METH_NOARGS, NULL}, + {"keys", hamt_py_keys, METH_NOARGS, NULL}, + {"values", hamt_py_values, METH_NOARGS, NULL}, #ifdef Py_DEBUG - {"__dump__", _PyCFunction_CAST(hamt_py_dump), METH_NOARGS, NULL}, + {"__dump__", hamt_py_dump, METH_NOARGS, NULL}, #endif {NULL, NULL} }; static PySequenceMethods PyHamt_as_sequence = { - 0, /* sq_length */ - 0, /* sq_concat */ - 0, /* sq_repeat */ - 0, /* sq_item */ - 0, /* sq_slice */ - 0, /* sq_ass_item */ - 0, /* sq_ass_slice */ - (objobjproc)hamt_tp_contains, /* sq_contains */ - 0, /* sq_inplace_concat */ - 0, /* sq_inplace_repeat */ + .sq_contains = hamt_tp_contains, }; static PyMappingMethods PyHamt_as_mapping = { - (lenfunc)hamt_tp_len, /* mp_length */ - (binaryfunc)hamt_tp_subscript, /* mp_subscript */ + .mp_length = hamt_tp_len, + .mp_subscript = hamt_tp_subscript, }; PyTypeObject _PyHamt_Type = { @@ -2820,13 +2814,13 @@ PyTypeObject _PyHamt_Type = { .tp_methods = PyHamt_methods, .tp_as_mapping = &PyHamt_as_mapping, .tp_as_sequence = &PyHamt_as_sequence, - .tp_iter = (getiterfunc)hamt_tp_iter, - .tp_dealloc = (destructor)hamt_tp_dealloc, + .tp_iter = hamt_tp_iter, + .tp_dealloc = hamt_tp_dealloc, .tp_getattro = PyObject_GenericGetAttr, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, .tp_richcompare = hamt_tp_richcompare, - .tp_traverse = (traverseproc)hamt_tp_traverse, - .tp_clear = (inquiry)hamt_tp_clear, + .tp_traverse = hamt_tp_traverse, + .tp_clear = hamt_tp_clear, .tp_new = hamt_tp_new, .tp_weaklistoffset = offsetof(PyHamtObject, h_weakreflist), .tp_hash = PyObject_HashNotImplemented, @@ -2841,10 +2835,10 @@ PyTypeObject _PyHamt_ArrayNode_Type = { "hamt_array_node", sizeof(PyHamtNode_Array), 0, - .tp_dealloc = (destructor)hamt_node_array_dealloc, + .tp_dealloc = hamt_node_array_dealloc, .tp_getattro = PyObject_GenericGetAttr, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, - .tp_traverse = (traverseproc)hamt_node_array_traverse, + .tp_traverse = hamt_node_array_traverse, .tp_free = PyObject_GC_Del, .tp_hash = PyObject_HashNotImplemented, }; @@ -2854,10 +2848,10 @@ PyTypeObject _PyHamt_BitmapNode_Type = { "hamt_bitmap_node", sizeof(PyHamtNode_Bitmap) - sizeof(PyObject *), sizeof(PyObject *), - .tp_dealloc = (destructor)hamt_node_bitmap_dealloc, + .tp_dealloc = hamt_node_bitmap_dealloc, .tp_getattro = PyObject_GenericGetAttr, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, - .tp_traverse = (traverseproc)hamt_node_bitmap_traverse, + .tp_traverse = hamt_node_bitmap_traverse, .tp_free = PyObject_GC_Del, .tp_hash = PyObject_HashNotImplemented, }; @@ -2867,10 +2861,10 @@ PyTypeObject _PyHamt_CollisionNode_Type = { "hamt_collision_node", sizeof(PyHamtNode_Collision) - sizeof(PyObject *), sizeof(PyObject *), - .tp_dealloc = (destructor)hamt_node_collision_dealloc, + .tp_dealloc = hamt_node_collision_dealloc, .tp_getattro = PyObject_GenericGetAttr, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, - .tp_traverse = (traverseproc)hamt_node_collision_traverse, + .tp_traverse = hamt_node_collision_traverse, .tp_free = PyObject_GC_Del, .tp_hash = PyObject_HashNotImplemented, }; diff --git a/Python/import.c b/Python/import.c index a9282dde633959..b3648e24d0e064 100644 --- a/Python/import.c +++ b/Python/import.c @@ -4688,7 +4688,7 @@ _imp_create_dynamic_impl(PyObject *module, PyObject *spec, PyObject *file) * code relies on fp still being open. */ FILE *fp; if (file != NULL) { - fp = _Py_fopen_obj(info.filename, "r"); + fp = Py_fopen(info.filename, "r"); if (fp == NULL) { goto finally; } diff --git a/Python/instrumentation.c b/Python/instrumentation.c index e4255bfad8c41a..17e5346be5ed3d 100644 --- a/Python/instrumentation.c +++ b/Python/instrumentation.c @@ -14,6 +14,7 @@ #include "pycore_namespace.h" #include "pycore_object.h" #include "pycore_opcode_metadata.h" // IS_VALID_OPCODE, _PyOpcode_Caches +#include "pycore_opcode_utils.h" // IS_CONDITIONAL_JUMP_OPCODE #include "pycore_pyatomic_ft_wrappers.h" // FT_ATOMIC_STORE_UINTPTR_RELEASE #include "pycore_pyerrors.h" #include "pycore_pystate.h" // _PyInterpreterState_GET() @@ -95,8 +96,10 @@ static const int8_t EVENT_FOR_OPCODE[256] = { [INSTRUMENTED_POP_JUMP_IF_TRUE] = PY_MONITORING_EVENT_BRANCH_RIGHT, [INSTRUMENTED_POP_JUMP_IF_NONE] = PY_MONITORING_EVENT_BRANCH_RIGHT, [INSTRUMENTED_POP_JUMP_IF_NOT_NONE] = PY_MONITORING_EVENT_BRANCH_RIGHT, - [FOR_ITER] = PY_MONITORING_EVENT_BRANCH_RIGHT, - [INSTRUMENTED_FOR_ITER] = PY_MONITORING_EVENT_BRANCH_RIGHT, + [FOR_ITER] = PY_MONITORING_EVENT_BRANCH_LEFT, + [INSTRUMENTED_FOR_ITER] = PY_MONITORING_EVENT_BRANCH_LEFT, + [POP_ITER] = PY_MONITORING_EVENT_BRANCH_RIGHT, + [INSTRUMENTED_POP_ITER] = PY_MONITORING_EVENT_BRANCH_RIGHT, [END_FOR] = PY_MONITORING_EVENT_STOP_ITERATION, [INSTRUMENTED_END_FOR] = PY_MONITORING_EVENT_STOP_ITERATION, [END_SEND] = PY_MONITORING_EVENT_STOP_ITERATION, @@ -119,6 +122,7 @@ static const uint8_t DE_INSTRUMENT[256] = { [INSTRUMENTED_POP_JUMP_IF_NONE] = POP_JUMP_IF_NONE, [INSTRUMENTED_POP_JUMP_IF_NOT_NONE] = POP_JUMP_IF_NOT_NONE, [INSTRUMENTED_FOR_ITER] = FOR_ITER, + [INSTRUMENTED_POP_ITER] = POP_ITER, [INSTRUMENTED_END_FOR] = END_FOR, [INSTRUMENTED_END_SEND] = END_SEND, [INSTRUMENTED_LOAD_SUPER_ATTR] = LOAD_SUPER_ATTR, @@ -156,6 +160,8 @@ static const uint8_t INSTRUMENTED_OPCODES[256] = { [INSTRUMENTED_END_SEND] = INSTRUMENTED_END_SEND, [FOR_ITER] = INSTRUMENTED_FOR_ITER, [INSTRUMENTED_FOR_ITER] = INSTRUMENTED_FOR_ITER, + [POP_ITER] = INSTRUMENTED_POP_ITER, + [INSTRUMENTED_POP_ITER] = INSTRUMENTED_POP_ITER, [LOAD_SUPER_ATTR] = INSTRUMENTED_LOAD_SUPER_ATTR, [INSTRUMENTED_LOAD_SUPER_ATTR] = INSTRUMENTED_LOAD_SUPER_ATTR, [NOT_TAKEN] = INSTRUMENTED_NOT_TAKEN, @@ -1077,8 +1083,8 @@ static const char *const event_names [] = { static int call_instrumentation_vector( - PyThreadState *tstate, int event, - _PyInterpreterFrame *frame, _Py_CODEUNIT *instr, Py_ssize_t nargs, PyObject *args[]) + _Py_CODEUNIT *instr, PyThreadState *tstate, int event, + _PyInterpreterFrame *frame, _Py_CODEUNIT *arg2, Py_ssize_t nargs, PyObject *args[]) { if (tstate->tracing) { return 0; @@ -1091,17 +1097,13 @@ call_instrumentation_vector( int offset = (int)(instr - _PyFrame_GetBytecode(frame)); /* Offset visible to user should be the offset in bytes, as that is the * convention for APIs involving code offsets. */ - int bytes_offset = offset * (int)sizeof(_Py_CODEUNIT); - if (event == PY_MONITORING_EVENT_BRANCH_LEFT) { - assert(EVENT_FOR_OPCODE[_Py_GetBaseCodeUnit(code, offset-2).op.code] == PY_MONITORING_EVENT_BRANCH_RIGHT); - bytes_offset -= 4; - } - PyObject *offset_obj = PyLong_FromLong(bytes_offset); - if (offset_obj == NULL) { + int bytes_arg2 = (int)(arg2 - _PyFrame_GetBytecode(frame)) * (int)sizeof(_Py_CODEUNIT); + PyObject *arg2_obj = PyLong_FromLong(bytes_arg2); + if (arg2_obj == NULL) { return -1; } assert(args[2] == NULL); - args[2] = offset_obj; + args[2] = arg2_obj; PyInterpreterState *interp = tstate->interp; uint8_t tools = get_tools_for_instruction(code, interp, offset, event); size_t nargsf = (size_t) nargs | PY_VECTORCALL_ARGUMENTS_OFFSET; @@ -1139,7 +1141,7 @@ call_instrumentation_vector( } } } - Py_DECREF(offset_obj); + Py_DECREF(arg2_obj); return err; } @@ -1149,7 +1151,7 @@ _Py_call_instrumentation( _PyInterpreterFrame *frame, _Py_CODEUNIT *instr) { PyObject *args[3] = { NULL, NULL, NULL }; - return call_instrumentation_vector(tstate, event, frame, instr, 2, args); + return call_instrumentation_vector(instr, tstate, event, frame, instr, 2, args); } int @@ -1158,7 +1160,7 @@ _Py_call_instrumentation_arg( _PyInterpreterFrame *frame, _Py_CODEUNIT *instr, PyObject *arg) { PyObject *args[4] = { NULL, NULL, NULL, arg }; - return call_instrumentation_vector(tstate, event, frame, instr, 3, args); + return call_instrumentation_vector(instr, tstate, event, frame, instr, 3, args); } int @@ -1167,34 +1169,34 @@ _Py_call_instrumentation_2args( _PyInterpreterFrame *frame, _Py_CODEUNIT *instr, PyObject *arg0, PyObject *arg1) { PyObject *args[5] = { NULL, NULL, NULL, arg0, arg1 }; - return call_instrumentation_vector(tstate, event, frame, instr, 4, args); + return call_instrumentation_vector(instr, tstate, event, frame, instr, 4, args); } _Py_CODEUNIT * _Py_call_instrumentation_jump( - PyThreadState *tstate, int event, - _PyInterpreterFrame *frame, _Py_CODEUNIT *instr, _Py_CODEUNIT *target) + _Py_CODEUNIT *instr, PyThreadState *tstate, int event, + _PyInterpreterFrame *frame, _Py_CODEUNIT *src, _Py_CODEUNIT *dest) { assert(event == PY_MONITORING_EVENT_JUMP || event == PY_MONITORING_EVENT_BRANCH_RIGHT || event == PY_MONITORING_EVENT_BRANCH_LEFT); - assert(frame->instr_ptr == instr); - int to = (int)(target - _PyFrame_GetBytecode(frame)); + int to = (int)(dest - _PyFrame_GetBytecode(frame)); PyObject *to_obj = PyLong_FromLong(to * (int)sizeof(_Py_CODEUNIT)); if (to_obj == NULL) { return NULL; } PyObject *args[4] = { NULL, NULL, NULL, to_obj }; - int err = call_instrumentation_vector(tstate, event, frame, instr, 3, args); + _Py_CODEUNIT *instr_ptr = frame->instr_ptr; + int err = call_instrumentation_vector(instr, tstate, event, frame, src, 3, args); Py_DECREF(to_obj); if (err) { return NULL; } - if (frame->instr_ptr != instr) { + if (frame->instr_ptr != instr_ptr) { /* The callback has caused a jump (by setting the line number) */ return frame->instr_ptr; } - return target; + return dest; } static void @@ -1204,7 +1206,7 @@ call_instrumentation_vector_protected( { assert(_PyErr_Occurred(tstate)); PyObject *exc = _PyErr_GetRaisedException(tstate); - int err = call_instrumentation_vector(tstate, event, frame, instr, nargs, args); + int err = call_instrumentation_vector(instr, tstate, event, frame, instr, nargs, args); if (err) { Py_XDECREF(exc); } @@ -1496,9 +1498,10 @@ initialize_lines(PyCodeObject *code) case END_FOR: case END_SEND: case RESUME: + case POP_ITER: /* END_FOR cannot start a line, as it is skipped by FOR_ITER * END_SEND cannot start a line, as it is skipped by SEND - * RESUME must not be instrumented with INSTRUMENT_LINE */ + * RESUME and POP_ITER must not be instrumented with INSTRUMENT_LINE */ line_data[i].original_opcode = 0; break; default: @@ -1570,11 +1573,14 @@ initialize_lines(PyCodeObject *code) } assert(target >= 0); if (line_data[target].line_delta != NO_LINE) { - line_data[target].original_opcode = _Py_GetBaseCodeUnit(code, target).op.code; - if (line_data[target].line_delta == COMPUTED_LINE_LINENO_CHANGE) { - // If the line is a jump target, we are not sure if the line - // number changes, so we set it to COMPUTED_LINE. - line_data[target].line_delta = COMPUTED_LINE; + int opcode = _Py_GetBaseCodeUnit(code, target).op.code; + if (opcode != POP_ITER) { + line_data[target].original_opcode = opcode; + if (line_data[target].line_delta == COMPUTED_LINE_LINENO_CHANGE) { + // If the line is a jump target, we are not sure if the line + // number changes, so we set it to COMPUTED_LINE. + line_data[target].line_delta = COMPUTED_LINE; + } } } } @@ -2887,30 +2893,52 @@ branch_handler( _PyLegacyBranchEventHandler *self, PyObject *const *args, size_t nargsf, PyObject *kwnames ) { + // Find the other instrumented instruction and remove tool + // The spec (PEP 669) allows spurious events after a DISABLE, + // so a best effort is good enough. + assert(PyVectorcall_NARGS(nargsf) >= 3); + PyCodeObject *code = (PyCodeObject *)args[0]; + int src_offset = PyLong_AsLong(args[1]); + if (PyErr_Occurred()) { + return NULL; + } + _Py_CODEUNIT instr = _PyCode_CODE(code)[src_offset/2]; + if (!is_instrumented(instr.op.code)) { + /* Already disabled */ + return &_PyInstrumentation_DISABLE; + } PyObject *res = PyObject_Vectorcall(self->handler, args, nargsf, kwnames); if (res == &_PyInstrumentation_DISABLE) { - // Find the other instrumented instruction and remove tool - assert(PyVectorcall_NARGS(nargsf) >= 2); - PyObject *offset_obj = args[1]; - int bytes_offset = PyLong_AsLong(offset_obj); - if (PyErr_Occurred()) { - return NULL; - } - PyCodeObject *code = (PyCodeObject *)args[0]; - if (!PyCode_Check(code) || (bytes_offset & 1)) { - return res; - } - int offset = bytes_offset / 2; /* We need FOR_ITER and POP_JUMP_ to be the same size */ assert(INLINE_CACHE_ENTRIES_FOR_ITER == 1); - if (self->right) { - offset += 2; + int offset; + int other_event; + if (instr.op.code == FOR_ITER) { + if (self->right) { + offset = src_offset/2; + other_event = PY_MONITORING_EVENT_BRANCH_LEFT; + } + else { + // We don't know where the POP_ITER is, so + // we cannot de-instrument it. + return res; + } + } + else if (IS_CONDITIONAL_JUMP_OPCODE(instr.op.code)) { + if (self->right) { + offset = src_offset/2 + 2; + other_event = PY_MONITORING_EVENT_BRANCH_LEFT; + assert(_Py_GetBaseCodeUnit(code, offset).op.code == NOT_TAKEN); + } + else { + offset = src_offset/2; + other_event = PY_MONITORING_EVENT_BRANCH_RIGHT; + } } - if (offset >= Py_SIZE(code)) { + else { + // Orphaned NOT_TAKEN -- Jump removed by the compiler return res; } - int other_event = self->right ? - PY_MONITORING_EVENT_BRANCH_LEFT : PY_MONITORING_EVENT_BRANCH_RIGHT; LOCK_CODE(code); remove_tools(code, offset, other_event, 1 << self->tool_id); UNLOCK_CODE(); @@ -3013,15 +3041,30 @@ static PyObject * branchesiter_next(branchesiterator *bi) { int offset = bi->bi_offset; + int oparg = 0; while (offset < Py_SIZE(bi->bi_code)) { _Py_CODEUNIT inst = _Py_GetBaseCodeUnit(bi->bi_code, offset); - int next_offset = offset + _PyInstruction_GetLength(bi->bi_code, offset); - int event = EVENT_FOR_OPCODE[inst.op.code]; - if (event == PY_MONITORING_EVENT_BRANCH_RIGHT) { - /* Skip NOT_TAKEN */ - int not_taken = next_offset + 1; - bi->bi_offset = not_taken; - return int_triple(offset*2, not_taken*2, (next_offset + inst.op.arg)*2); + int next_offset = offset + 1 + _PyOpcode_Caches[inst.op.code]; + switch(inst.op.code) { + case EXTENDED_ARG: + oparg = (oparg << 8) | inst.op.arg; + break; + case FOR_ITER: + oparg = (oparg << 8) | inst.op.arg; + bi->bi_offset = next_offset; + int target = next_offset + oparg+2; // Skips END_FOR and POP_ITER + return int_triple(offset*2, next_offset*2, target*2); + case POP_JUMP_IF_FALSE: + case POP_JUMP_IF_TRUE: + case POP_JUMP_IF_NONE: + case POP_JUMP_IF_NOT_NONE: + oparg = (oparg << 8) | inst.op.arg; + /* Skip NOT_TAKEN */ + int not_taken = next_offset + 1; + bi->bi_offset = not_taken; + return int_triple(offset*2, not_taken*2, (next_offset + oparg)*2); + default: + oparg = 0; } offset = next_offset; } diff --git a/Python/modsupport.c b/Python/modsupport.c index 0fb7783345c78e..517dc971f88c87 100644 --- a/Python/modsupport.c +++ b/Python/modsupport.c @@ -648,3 +648,20 @@ PyModule_AddType(PyObject *module, PyTypeObject *type) return PyModule_AddObjectRef(module, name, (PyObject *)type); } + + +/* Exported functions for version helper macros */ + +#undef Py_PACK_FULL_VERSION +uint32_t +Py_PACK_FULL_VERSION(int x, int y, int z, int level, int serial) +{ + return _Py_PACK_FULL_VERSION(x, y, z, level, serial); +} + +#undef Py_PACK_VERSION +uint32_t +Py_PACK_VERSION(int x, int y) +{ + return Py_PACK_FULL_VERSION(x, y, 0, 0, 0); +} diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index 7f3fb9c9a63dd1..00d55d3a82edc1 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -29,6 +29,7 @@ static void *opcode_targets[256] = { &&TARGET_NOP, &&TARGET_NOT_TAKEN, &&TARGET_POP_EXCEPT, + &&TARGET_POP_ITER, &&TARGET_POP_TOP, &&TARGET_PUSH_EXC_INFO, &&TARGET_PUSH_NULL, @@ -147,7 +148,6 @@ static void *opcode_targets[256] = { &&_unknown_opcode, &&_unknown_opcode, &&_unknown_opcode, - &&_unknown_opcode, &&TARGET_RESUME, &&TARGET_BINARY_OP_ADD_FLOAT, &&TARGET_BINARY_OP_ADD_INT, @@ -234,8 +234,8 @@ static void *opcode_targets[256] = { &&_unknown_opcode, &&_unknown_opcode, &&_unknown_opcode, - &&_unknown_opcode, &&TARGET_INSTRUMENTED_END_FOR, + &&TARGET_INSTRUMENTED_POP_ITER, &&TARGET_INSTRUMENTED_END_SEND, &&TARGET_INSTRUMENTED_LOAD_SUPER_ATTR, &&TARGET_INSTRUMENTED_FOR_ITER, diff --git a/Python/optimizer.c b/Python/optimizer.c index 6a4d20fad76c15..52b3f0a84afedf 100644 --- a/Python/optimizer.c +++ b/Python/optimizer.c @@ -622,8 +622,14 @@ translate_bytecode_to_trace( goto done; } assert(opcode != ENTER_EXECUTOR && opcode != EXTENDED_ARG); - RESERVE_RAW(2, "_CHECK_VALIDITY_AND_SET_IP"); - ADD_TO_TRACE(_CHECK_VALIDITY_AND_SET_IP, 0, (uintptr_t)instr, target); + if (OPCODE_HAS_NO_SAVE_IP(opcode)) { + RESERVE_RAW(2, "_CHECK_VALIDITY"); + ADD_TO_TRACE(_CHECK_VALIDITY, 0, 0, target); + } + else { + RESERVE_RAW(2, "_CHECK_VALIDITY_AND_SET_IP"); + ADD_TO_TRACE(_CHECK_VALIDITY_AND_SET_IP, 0, (uintptr_t)instr, target); + } /* Special case the first instruction, * so that we can guarantee forward progress */ @@ -771,7 +777,7 @@ translate_bytecode_to_trace( uint32_t next_inst = target + 1 + INLINE_CACHE_ENTRIES_FOR_ITER + (oparg > 255); uint32_t jump_target = next_inst + oparg; assert(_Py_GetBaseCodeUnit(code, jump_target).op.code == END_FOR); - assert(_Py_GetBaseCodeUnit(code, jump_target+1).op.code == POP_TOP); + assert(_Py_GetBaseCodeUnit(code, jump_target+1).op.code == POP_ITER); } #endif break; diff --git a/Python/optimizer_bytecodes.c b/Python/optimizer_bytecodes.c index a14d119b7a1dec..86394480f76bb8 100644 --- a/Python/optimizer_bytecodes.c +++ b/Python/optimizer_bytecodes.c @@ -167,23 +167,56 @@ dummy_func(void) { } op(_BINARY_OP, (left, right -- res)) { - PyTypeObject *ltype = sym_get_type(left); - PyTypeObject *rtype = sym_get_type(right); - if (ltype != NULL && (ltype == &PyLong_Type || ltype == &PyFloat_Type) && - rtype != NULL && (rtype == &PyLong_Type || rtype == &PyFloat_Type)) - { - if (oparg != NB_TRUE_DIVIDE && oparg != NB_INPLACE_TRUE_DIVIDE && - ltype == &PyLong_Type && rtype == &PyLong_Type) { - /* If both inputs are ints and the op is not division the result is an int */ - res = sym_new_type(ctx, &PyLong_Type); + bool lhs_int = sym_matches_type(left, &PyLong_Type); + bool rhs_int = sym_matches_type(right, &PyLong_Type); + bool lhs_float = sym_matches_type(left, &PyFloat_Type); + bool rhs_float = sym_matches_type(right, &PyFloat_Type); + if (!((lhs_int || lhs_float) && (rhs_int || rhs_float))) { + // There's something other than an int or float involved: + res = sym_new_unknown(ctx); + } + else if (oparg == NB_POWER || oparg == NB_INPLACE_POWER) { + // This one's fun... the *type* of the result depends on the + // *values* being exponentiated. However, exponents with one + // constant part are reasonably common, so it's probably worth + // trying to infer some simple cases: + // - A: 1 ** 1 -> 1 (int ** int -> int) + // - B: 1 ** -1 -> 1.0 (int ** int -> float) + // - C: 1.0 ** 1 -> 1.0 (float ** int -> float) + // - D: 1 ** 1.0 -> 1.0 (int ** float -> float) + // - E: -1 ** 0.5 ~> 1j (int ** float -> complex) + // - F: 1.0 ** 1.0 -> 1.0 (float ** float -> float) + // - G: -1.0 ** 0.5 ~> 1j (float ** float -> complex) + if (rhs_float) { + // Case D, E, F, or G... can't know without the sign of the LHS + // or whether the RHS is whole, which isn't worth the effort: + res = sym_new_unknown(ctx); } - else { - /* For any other op combining ints/floats the result is a float */ + else if (lhs_float) { + // Case C: res = sym_new_type(ctx, &PyFloat_Type); } + else if (!sym_is_const(right)) { + // Case A or B... can't know without the sign of the RHS: + res = sym_new_unknown(ctx); + } + else if (_PyLong_IsNegative((PyLongObject *)sym_get_const(right))) { + // Case B: + res = sym_new_type(ctx, &PyFloat_Type); + } + else { + // Case A: + res = sym_new_type(ctx, &PyLong_Type); + } + } + else if (oparg == NB_TRUE_DIVIDE || oparg == NB_INPLACE_TRUE_DIVIDE) { + res = sym_new_type(ctx, &PyFloat_Type); + } + else if (lhs_int && rhs_int) { + res = sym_new_type(ctx, &PyLong_Type); } else { - res = sym_new_unknown(ctx); + res = sym_new_type(ctx, &PyFloat_Type); } } diff --git a/Python/optimizer_cases.c.h b/Python/optimizer_cases.c.h index 0fcf5e18ed5808..c72ae7b6281e80 100644 --- a/Python/optimizer_cases.c.h +++ b/Python/optimizer_cases.c.h @@ -115,6 +115,12 @@ break; } + case _END_FOR: { + stack_pointer += -1; + assert(WITHIN_STACK_BOUNDS()); + break; + } + case _END_SEND: { _Py_UopsSymbol *val; val = sym_new_not_null(ctx); @@ -2301,24 +2307,69 @@ _Py_UopsSymbol *res; right = stack_pointer[-1]; left = stack_pointer[-2]; - PyTypeObject *ltype = sym_get_type(left); - PyTypeObject *rtype = sym_get_type(right); - if (ltype != NULL && (ltype == &PyLong_Type || ltype == &PyFloat_Type) && - rtype != NULL && (rtype == &PyLong_Type || rtype == &PyFloat_Type)) - { - if (oparg != NB_TRUE_DIVIDE && oparg != NB_INPLACE_TRUE_DIVIDE && - ltype == &PyLong_Type && rtype == &PyLong_Type) { - /* If both inputs are ints and the op is not division the result is an int */ - res = sym_new_type(ctx, &PyLong_Type); + bool lhs_int = sym_matches_type(left, &PyLong_Type); + bool rhs_int = sym_matches_type(right, &PyLong_Type); + bool lhs_float = sym_matches_type(left, &PyFloat_Type); + bool rhs_float = sym_matches_type(right, &PyFloat_Type); + if (!((lhs_int || lhs_float) && (rhs_int || rhs_float))) { + // There's something other than an int or float involved: + res = sym_new_unknown(ctx); + } + else { + if (oparg == NB_POWER || oparg == NB_INPLACE_POWER) { + // This one's fun... the *type* of the result depends on the + // *values* being exponentiated. However, exponents with one + // constant part are reasonably common, so it's probably worth + // trying to infer some simple cases: + // - A: 1 ** 1 -> 1 (int ** int -> int) + // - B: 1 ** -1 -> 1.0 (int ** int -> float) + // - C: 1.0 ** 1 -> 1.0 (float ** int -> float) + // - D: 1 ** 1.0 -> 1.0 (int ** float -> float) + // - E: -1 ** 0.5 ~> 1j (int ** float -> complex) + // - F: 1.0 ** 1.0 -> 1.0 (float ** float -> float) + // - G: -1.0 ** 0.5 ~> 1j (float ** float -> complex) + if (rhs_float) { + // Case D, E, F, or G... can't know without the sign of the LHS + // or whether the RHS is whole, which isn't worth the effort: + res = sym_new_unknown(ctx); + } + else { + if (lhs_float) { + // Case C: + res = sym_new_type(ctx, &PyFloat_Type); + } + else { + if (!sym_is_const(right)) { + // Case A or B... can't know without the sign of the RHS: + res = sym_new_unknown(ctx); + } + else { + if (_PyLong_IsNegative((PyLongObject *)sym_get_const(right))) { + // Case B: + res = sym_new_type(ctx, &PyFloat_Type); + } + else { + // Case A: + res = sym_new_type(ctx, &PyLong_Type); + } + } + } + } } else { - /* For any other op combining ints/floats the result is a float */ - res = sym_new_type(ctx, &PyFloat_Type); + if (oparg == NB_TRUE_DIVIDE || oparg == NB_INPLACE_TRUE_DIVIDE) { + res = sym_new_type(ctx, &PyFloat_Type); + } + else { + if (lhs_int && rhs_int) { + res = sym_new_type(ctx, &PyLong_Type); + } + else { + res = sym_new_type(ctx, &PyFloat_Type); + } + } } } - else { - res = sym_new_unknown(ctx); - } stack_pointer[-2] = res; stack_pointer += -1; assert(WITHIN_STACK_BOUNDS()); diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 06418123d6dd9b..8a15a09a01dbf7 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -46,8 +46,25 @@ #if defined(__APPLE__) # include +# include # include -# include +// The os_log unified logging APIs were introduced in macOS 10.12, iOS 10.0, +// tvOS 10.0, and watchOS 3.0; +# if defined(TARGET_OS_IPHONE) && TARGET_OS_IPHONE +# define HAS_APPLE_SYSTEM_LOG 1 +# elif defined(TARGET_OS_OSX) && TARGET_OS_OSX +# if defined(MAC_OS_X_VERSION_10_12) && MAC_OS_X_VERSION_MIN_REQUIRED >= MAC_OS_X_VERSION_10_12 +# define HAS_APPLE_SYSTEM_LOG 1 +# else +# define HAS_APPLE_SYSTEM_LOG 0 +# endif +# else +# define HAS_APPLE_SYSTEM_LOG 0 +# endif + +# if HAS_APPLE_SYSTEM_LOG +# include +# endif #endif #ifdef HAVE_SIGNAL_H @@ -77,7 +94,7 @@ static PyStatus init_sys_streams(PyThreadState *tstate); #ifdef __ANDROID__ static PyStatus init_android_streams(PyThreadState *tstate); #endif -#if defined(__APPLE__) +#if defined(__APPLE__) && HAS_APPLE_SYSTEM_LOG static PyStatus init_apple_streams(PyThreadState *tstate); #endif static void wait_for_thread_shutdown(PyThreadState *tstate); @@ -1262,7 +1279,7 @@ init_interp_main(PyThreadState *tstate) return status; } #endif -#if defined(__APPLE__) +#if defined(__APPLE__) && HAS_APPLE_SYSTEM_LOG if (config->use_system_logger) { status = init_apple_streams(tstate); if (_PyStatus_EXCEPTION(status)) { @@ -2946,7 +2963,7 @@ init_android_streams(PyThreadState *tstate) #endif // __ANDROID__ -#if defined(__APPLE__) +#if defined(__APPLE__) && HAS_APPLE_SYSTEM_LOG static PyObject * apple_log_write_impl(PyObject *self, PyObject *args) @@ -2957,14 +2974,9 @@ apple_log_write_impl(PyObject *self, PyObject *args) return NULL; } - // Call the underlying Apple logging API. The os_log unified logging APIs - // were introduced in macOS 10.12, iOS 10.0, tvOS 10.0, and watchOS 3.0; - // this call is a no-op on older versions. - #if TARGET_OS_IPHONE || (TARGET_OS_OSX && MAC_OS_X_VERSION_MIN_REQUIRED >= MAC_OS_X_VERSION_10_12) // Pass the user-provided text through explicit %s formatting // to avoid % literals being interpreted as a formatting directive. os_log_with_type(OS_LOG_DEFAULT, logtype, "%s", text); - #endif Py_RETURN_NONE; } @@ -2999,7 +3011,6 @@ init_apple_streams(PyThreadState *tstate) if (result == NULL) { goto error; } - goto done; error: @@ -3013,7 +3024,7 @@ init_apple_streams(PyThreadState *tstate) return status; } -#endif // __APPLE__ +#endif // __APPLE__ && HAS_APPLE_SYSTEM_LOG static void diff --git a/Python/pythonrun.c b/Python/pythonrun.c index 31e065ff00d59a..0da26ad3f9b4bd 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -467,7 +467,7 @@ _PyRun_SimpleFileObject(FILE *fp, PyObject *filename, int closeit, fclose(fp); } - pyc_fp = _Py_fopen_obj(filename, "rb"); + pyc_fp = Py_fopen(filename, "rb"); if (pyc_fp == NULL) { fprintf(stderr, "python: Can't reopen .pyc file\n"); goto done; diff --git a/Python/specialize.c b/Python/specialize.c index c918c77779d20d..c9325c39210874 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -2480,23 +2480,23 @@ _Py_Specialize_CompareOp(_PyStackRef lhs_st, _PyStackRef rhs_st, _Py_CODEUNIT *i { PyObject *lhs = PyStackRef_AsPyObjectBorrow(lhs_st); PyObject *rhs = PyStackRef_AsPyObjectBorrow(rhs_st); + uint8_t specialized_op; - assert(ENABLE_SPECIALIZATION); + assert(ENABLE_SPECIALIZATION_FT); assert(_PyOpcode_Caches[COMPARE_OP] == INLINE_CACHE_ENTRIES_COMPARE_OP); // All of these specializations compute boolean values, so they're all valid // regardless of the fifth-lowest oparg bit. - _PyCompareOpCache *cache = (_PyCompareOpCache *)(instr + 1); if (Py_TYPE(lhs) != Py_TYPE(rhs)) { SPECIALIZATION_FAIL(COMPARE_OP, compare_op_fail_kind(lhs, rhs)); goto failure; } if (PyFloat_CheckExact(lhs)) { - instr->op.code = COMPARE_OP_FLOAT; + specialized_op = COMPARE_OP_FLOAT; goto success; } if (PyLong_CheckExact(lhs)) { if (_PyLong_IsCompact((PyLongObject *)lhs) && _PyLong_IsCompact((PyLongObject *)rhs)) { - instr->op.code = COMPARE_OP_INT; + specialized_op = COMPARE_OP_INT; goto success; } else { @@ -2511,19 +2511,16 @@ _Py_Specialize_CompareOp(_PyStackRef lhs_st, _PyStackRef rhs_st, _Py_CODEUNIT *i goto failure; } else { - instr->op.code = COMPARE_OP_STR; + specialized_op = COMPARE_OP_STR; goto success; } } SPECIALIZATION_FAIL(COMPARE_OP, compare_op_fail_kind(lhs, rhs)); failure: - STAT_INC(COMPARE_OP, failure); - instr->op.code = COMPARE_OP; - cache->counter = adaptive_counter_backoff(cache->counter); + unspecialize(instr); return; success: - STAT_INC(COMPARE_OP, success); - cache->counter = adaptive_counter_cooldown(); + specialize(instr, specialized_op); } #ifdef Py_STATS diff --git a/Python/sysmodule.c b/Python/sysmodule.c index d6719f9bb0af91..887591a681b25c 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -2356,7 +2356,7 @@ static PyObject * sys__dump_tracelets_impl(PyObject *module, PyObject *outpath) /*[clinic end generated code: output=a7fe265e2bc3b674 input=5bff6880cd28ffd1]*/ { - FILE *out = _Py_fopen_obj(outpath, "wb"); + FILE *out = Py_fopen(outpath, "wb"); if (out == NULL) { return NULL; } diff --git a/Tools/cases_generator/analyzer.py b/Tools/cases_generator/analyzer.py index eca851e6de87ae..679beca3ec3a9d 100644 --- a/Tools/cases_generator/analyzer.py +++ b/Tools/cases_generator/analyzer.py @@ -27,6 +27,7 @@ class Properties: oparg_and_1: bool = False const_oparg: int = -1 needs_prev: bool = False + no_save_ip: bool = False def dump(self, indent: str) -> None: simple_properties = self.__dict__.copy() @@ -60,6 +61,7 @@ def from_list(properties: list["Properties"]) -> "Properties": side_exit=any(p.side_exit for p in properties), pure=all(p.pure for p in properties), needs_prev=any(p.needs_prev for p in properties), + no_save_ip=all(p.no_save_ip for p in properties), ) @property @@ -87,6 +89,7 @@ def escapes(self) -> bool: has_free=False, side_exit=False, pure=True, + no_save_ip=False, ) @@ -596,6 +599,7 @@ def has_error_without_pop(op: parser.InstDef) -> bool: "_PyLong_CompactValue", "_PyLong_DigitCount", "_PyLong_IsCompact", + "_PyLong_IsNegative", "_PyLong_IsNonNegativeCompact", "_PyLong_IsZero", "_PyLong_Multiply", @@ -668,7 +672,7 @@ def check_escaping_calls(instr: parser.InstDef, escapes: dict[lexer.Token, tuple if tkn.kind == "IF": next(tkn_iter) in_if = 1 - if tkn.kind == "IDENTIFIER" and tkn.text in ("DEOPT_IF", "ERROR_IF"): + if tkn.kind == "IDENTIFIER" and tkn.text in ("DEOPT_IF", "ERROR_IF", "EXIT_IF"): next(tkn_iter) in_if = 1 elif tkn.kind == "LPAREN" and in_if: @@ -829,6 +833,7 @@ def compute_properties(op: parser.InstDef) -> Properties: and not has_free, has_free=has_free, pure="pure" in op.annotations, + no_save_ip="no_save_ip" in op.annotations, tier=tier_variable(op), needs_prev=variable_used(op, "prev_instr"), ) diff --git a/Tools/cases_generator/generators_common.py b/Tools/cases_generator/generators_common.py index d17617cab0266b..8df9a9cada92ae 100644 --- a/Tools/cases_generator/generators_common.py +++ b/Tools/cases_generator/generators_common.py @@ -583,6 +583,8 @@ def cflags(p: Properties) -> str: flags.append("HAS_ESCAPES_FLAG") if p.pure: flags.append("HAS_PURE_FLAG") + if p.no_save_ip: + flags.append("HAS_NO_SAVE_IP_FLAG") if p.oparg_and_1: flags.append("HAS_OPARG_AND_1_FLAG") if flags: diff --git a/Tools/cases_generator/lexer.py b/Tools/cases_generator/lexer.py index 37f96398ff175f..bee2a185745f4d 100644 --- a/Tools/cases_generator/lexer.py +++ b/Tools/cases_generator/lexer.py @@ -226,6 +226,7 @@ def choice(*opts: str) -> str: "replicate", "tier1", "tier2", + "no_save_ip", } __all__ = [] diff --git a/Tools/cases_generator/opcode_metadata_generator.py b/Tools/cases_generator/opcode_metadata_generator.py index 1a9849c0cbbb25..453db6905d6842 100644 --- a/Tools/cases_generator/opcode_metadata_generator.py +++ b/Tools/cases_generator/opcode_metadata_generator.py @@ -53,6 +53,7 @@ "PASSTHROUGH", "OPARG_AND_1", "ERROR_NO_POP", + "NO_SAVE_IP", ] @@ -285,8 +286,8 @@ def generate_metadata_table(analysis: Analysis, out: CWriter) -> None: table_size = 256 + len(analysis.pseudos) out.emit("struct opcode_metadata {\n") out.emit("uint8_t valid_entry;\n") - out.emit("int8_t instr_format;\n") - out.emit("int16_t flags;\n") + out.emit("uint8_t instr_format;\n") + out.emit("uint16_t flags;\n") out.emit("};\n\n") out.emit( f"extern const struct opcode_metadata _PyOpcode_opcode_metadata[{table_size}];\n" diff --git a/Tools/cases_generator/tier1_generator.py b/Tools/cases_generator/tier1_generator.py index fcdd3bdacd0e7a..40562da99b20ea 100644 --- a/Tools/cases_generator/tier1_generator.py +++ b/Tools/cases_generator/tier1_generator.py @@ -151,9 +151,12 @@ def generate_tier1( if inst.properties.needs_prev: out.emit(f"_Py_CODEUNIT* const prev_instr = frame->instr_ptr;\n") if needs_this and not inst.is_target: - out.emit(f"_Py_CODEUNIT* const this_instr = frame->instr_ptr = next_instr;\n") + if inst.properties.no_save_ip: + out.emit(f"_Py_CODEUNIT* const this_instr = next_instr;\n") + else: + out.emit(f"_Py_CODEUNIT* const this_instr = frame->instr_ptr = next_instr;\n") out.emit(unused_guard) - else: + elif not inst.properties.no_save_ip: out.emit(f"frame->instr_ptr = next_instr;\n") out.emit(f"next_instr += {inst.size};\n") out.emit(f"INSTRUCTION_STATS({name});\n") diff --git a/configure b/configure index 61ee51c4b36473..bb77c558abda5a 100755 --- a/configure +++ b/configure @@ -4595,7 +4595,7 @@ printf "%s\n" "$IPHONEOS_DEPLOYMENT_TARGET" >&6; } _host_ident=$host_cpu ;; *-*-emscripten) - _host_ident=$(emcc -dumpversion)-$host_cpu + _host_ident=$(emcc -dumpversion | cut -f1 -d-)-$host_cpu ;; wasm32-*-* | wasm64-*-*) _host_ident=$host_cpu @@ -9398,7 +9398,7 @@ fi printf %s "checking BOLT_COMMON_FLAGS... " >&6; } if test -z "${BOLT_COMMON_FLAGS}" then - BOLT_COMMON_FLAGS=-update-debug-sections + BOLT_COMMON_FLAGS=" -update-debug-sections -skip-funcs=_PyEval_EvalFrameDefault,sre_ucs1_match/1,sre_ucs2_match/1,sre_ucs4_match/1 " fi @@ -19001,6 +19001,12 @@ if test "x$ac_cv_func_ctermid" = xyes then : printf "%s\n" "#define HAVE_CTERMID 1" >>confdefs.h +fi +ac_fn_c_check_func "$LINENO" "dladdr" "ac_cv_func_dladdr" +if test "x$ac_cv_func_dladdr" = xyes +then : + printf "%s\n" "#define HAVE_DLADDR 1" >>confdefs.h + fi ac_fn_c_check_func "$LINENO" "dup" "ac_cv_func_dup" if test "x$ac_cv_func_dup" = xyes @@ -19986,10 +19992,10 @@ then : printf "%s\n" "#define HAVE_TRUNCATE 1" >>confdefs.h fi -ac_fn_c_check_func "$LINENO" "ttyname" "ac_cv_func_ttyname" -if test "x$ac_cv_func_ttyname" = xyes +ac_fn_c_check_func "$LINENO" "ttyname_r" "ac_cv_func_ttyname_r" +if test "x$ac_cv_func_ttyname_r" = xyes then : - printf "%s\n" "#define HAVE_TTYNAME 1" >>confdefs.h + printf "%s\n" "#define HAVE_TTYNAME_R 1" >>confdefs.h fi ac_fn_c_check_func "$LINENO" "umask" "ac_cv_func_umask" diff --git a/configure.ac b/configure.ac index 172e8a1a842010..653cd3f6c531b6 100644 --- a/configure.ac +++ b/configure.ac @@ -794,7 +794,7 @@ if test "$cross_compiling" = yes; then _host_ident=$host_cpu ;; *-*-emscripten) - _host_ident=$(emcc -dumpversion)-$host_cpu + _host_ident=$(emcc -dumpversion | cut -f1 -d-)-$host_cpu ;; wasm32-*-* | wasm64-*-*) _host_ident=$host_cpu @@ -2170,7 +2170,14 @@ if test -z "${BOLT_COMMON_FLAGS}" then AS_VAR_SET( [BOLT_COMMON_FLAGS], - [-update-debug-sections] + [m4_normalize(" + [-update-debug-sections] + + dnl At least LLVM 19.x doesn't support computed gotos in PIC compiled code. + dnl Exclude functions containing computed gotos. + dnl TODO this may be fixed in LLVM 20.x via https://github.com/llvm/llvm-project/pull/120267. + [-skip-funcs=_PyEval_EvalFrameDefault,sre_ucs1_match/1,sre_ucs2_match/1,sre_ucs4_match/1] + ")] ) fi @@ -5121,7 +5128,7 @@ fi # checks for library functions AC_CHECK_FUNCS([ \ accept4 alarm bind_textdomain_codeset chmod chown clock closefrom close_range confstr \ - copy_file_range ctermid dup dup3 execv explicit_bzero explicit_memset \ + copy_file_range ctermid dladdr dup dup3 execv explicit_bzero explicit_memset \ faccessat fchmod fchmodat fchown fchownat fdopendir fdwalk fexecve \ fork fork1 fpathconf fstatat ftime ftruncate futimens futimes futimesat \ gai_strerror getegid geteuid getgid getgrent getgrgid getgrgid_r \ @@ -5145,7 +5152,7 @@ AC_CHECK_FUNCS([ \ sigfillset siginterrupt sigpending sigrelse sigtimedwait sigwait \ sigwaitinfo snprintf splice strftime strlcpy strsignal symlinkat sync \ sysconf tcgetpgrp tcsetpgrp tempnam timegm times tmpfile \ - tmpnam tmpnam_r truncate ttyname umask uname unlinkat unlockpt utimensat utimes vfork \ + tmpnam tmpnam_r truncate ttyname_r umask uname unlinkat unlockpt utimensat utimes vfork \ wait wait3 wait4 waitid waitpid wcscoll wcsftime wcsxfrm wmemcmp writev \ ]) diff --git a/pyconfig.h.in b/pyconfig.h.in index d862966b7de4c8..aaf52168c3d39d 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -286,6 +286,9 @@ /* Define if you have the 'dirfd' function or macro. */ #undef HAVE_DIRFD +/* Define to 1 if you have the 'dladdr' function. */ +#undef HAVE_DLADDR + /* Define to 1 if you have the header file. */ #undef HAVE_DLFCN_H @@ -1506,8 +1509,8 @@ /* Define to 1 if you have the 'truncate' function. */ #undef HAVE_TRUNCATE -/* Define to 1 if you have the 'ttyname' function. */ -#undef HAVE_TTYNAME +/* Define to 1 if you have the 'ttyname_r' function. */ +#undef HAVE_TTYNAME_R /* Define to 1 if you don't have 'tm_zone' but do have the external array 'tzname'. */