Skip to content

Commit

Permalink
Merge branch 'main' into fixsendfamily
Browse files Browse the repository at this point in the history
* main:
  pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)
  pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)
  pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)
  pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)
  pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)
  pythongh-104240: return code unit metadata from codegen (python#104300)
  • Loading branch information
carljm committed May 9, 2023
2 parents f44e763 + 2866e03 commit e40071d
Show file tree
Hide file tree
Showing 36 changed files with 1,366 additions and 718 deletions.
11 changes: 11 additions & 0 deletions Doc/library/asyncio-task.rst
Original file line number Diff line number Diff line change
Expand Up @@ -527,6 +527,8 @@ Running Tasks Concurrently
and there is no running event loop.


.. _eager-task-factory:

Eager Task Factory
==================

Expand Down Expand Up @@ -1174,8 +1176,17 @@ Task Object

Return the coroutine object wrapped by the :class:`Task`.

.. note::

This will return ``None`` for Tasks which have already
completed eagerly. See the :ref:`Eager Task Factory <eager-task-factory>`.

.. versionadded:: 3.8

.. versionchanged:: 3.12

Newly added eager task execution means result may be ``None``.

.. method:: get_context()

Return the :class:`contextvars.Context` object
Expand Down
8 changes: 8 additions & 0 deletions Doc/library/dis.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1196,6 +1196,14 @@ iterations of the loop.

.. versionadded:: 3.12

.. opcode:: LOAD_FAST_AND_CLEAR (var_num)

Pushes a reference to the local ``co_varnames[var_num]`` onto the stack (or
pushes ``NULL`` onto the stack if the local variable has not been
initialized) and sets ``co_varnames[var_num]`` to ``NULL``.

.. versionadded:: 3.12

.. opcode:: STORE_FAST (var_num)

Stores ``STACK.pop()`` into the local ``co_varnames[var_num]``.
Expand Down
24 changes: 24 additions & 0 deletions Doc/whatsnew/3.12.rst
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,30 @@ New Features
In Python 3.14, the default will switch to ``'data'``.
(Contributed by Petr Viktorin in :pep:`706`.)

.. _whatsnew312-pep709:

PEP 709: Comprehension inlining
-------------------------------

Dictionary, list, and set comprehensions are now inlined, rather than creating a
new single-use function object for each execution of the comprehension. This
speeds up execution of a comprehension by up to 2x.

Comprehension iteration variables remain isolated; they don't overwrite a
variable of the same name in the outer scope, nor are they visible after the
comprehension. This isolation is now maintained via stack/locals manipulation,
not via separate function scope.

Inlining does result in a few visible behavior changes:

* There is no longer a separate frame for the comprehension in tracebacks,
and tracing/profiling no longer shows the comprehension as a function call.
* Calling :func:`locals` inside a comprehension now includes variables
from outside the comprehension, and no longer includes the synthetic ``.0``
variable for the comprehension "argument".

Contributed by Carl Meyer and Vladimir Matveev in :pep:`709`.

PEP 688: Making the buffer protocol accessible in Python
--------------------------------------------------------

Expand Down
1 change: 1 addition & 0 deletions Include/internal/pycore_code.h
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ struct callable_cache {
// Note that these all fit within a byte, as do combinations.
// Later, we will use the smaller numbers to differentiate the different
// kinds of locals (e.g. pos-only arg, varkwargs, local-only).
#define CO_FAST_HIDDEN 0x10
#define CO_FAST_LOCAL 0x20
#define CO_FAST_CELL 0x40
#define CO_FAST_FREE 0x80
Expand Down
3 changes: 3 additions & 0 deletions Include/internal/pycore_compile.h
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ typedef struct {
PyObject *u_varnames; /* local variables */
PyObject *u_cellvars; /* cell variables */
PyObject *u_freevars; /* free variables */
PyObject *u_fasthidden; /* dict; keys are names that are fast-locals only
temporarily within an inlined comprehension. When
value is True, treat as fast-local. */

Py_ssize_t u_argcount; /* number of arguments for block */
Py_ssize_t u_posonlyargcount; /* number of positional only arguments for block */
Expand Down
2 changes: 1 addition & 1 deletion Include/internal/pycore_flowgraph.h
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ _PyCfgInstruction* _PyCfg_BasicblockLastInstr(const _PyCfgBasicblock *b);
int _PyCfg_OptimizeCodeUnit(_PyCfgBuilder *g, PyObject *consts, PyObject *const_cache,
int code_flags, int nlocals, int nparams, int firstlineno);
int _PyCfg_Stackdepth(_PyCfgBasicblock *entryblock, int code_flags);
void _PyCfg_ConvertExceptionHandlersToNops(_PyCfgBasicblock *entryblock);
void _PyCfg_ConvertPseudoOps(_PyCfgBasicblock *entryblock);
int _PyCfg_ResolveJumps(_PyCfgBuilder *g);


Expand Down
13 changes: 7 additions & 6 deletions Include/internal/pycore_opcode.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Include/internal/pycore_symtable.h
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ typedef struct _symtable_entry {
unsigned ste_needs_class_closure : 1; /* for class scopes, true if a
closure over __class__
should be created */
unsigned ste_comp_inlined : 1; /* true if this comprehension is inlined */
unsigned ste_comp_iter_target : 1; /* true if visiting comprehension target */
int ste_comp_iter_expr; /* non-zero if visiting a comprehension range expression */
int ste_lineno; /* first line of block */
Expand Down
21 changes: 12 additions & 9 deletions Include/opcode.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion Lib/importlib/_bootstrap_external.py
Original file line number Diff line number Diff line change
Expand Up @@ -442,6 +442,7 @@ def _write_atomic(path, data, mode=0o666):
# Python 3.12b1 3526 (Add instrumentation support)
# Python 3.12b1 3527 (Add LOAD_SUPER_ATTR)
# Python 3.12b1 3528 (Add LOAD_SUPER_ATTR_METHOD specialization)
# Python 3.12b1 3529 (Inline list/dict/set comprehensions)

# Python 3.13 will start with 3550

Expand All @@ -458,7 +459,7 @@ def _write_atomic(path, data, mode=0o666):
# Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array
# in PC/launcher.c must also be updated.

MAGIC_NUMBER = (3528).to_bytes(2, 'little') + b'\r\n'
MAGIC_NUMBER = (3529).to_bytes(2, 'little') + b'\r\n'

_RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c

Expand Down
4 changes: 4 additions & 0 deletions Lib/opcode.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,8 @@ def pseudo_op(name, op, real_ops):
jrel_op('JUMP_BACKWARD', 140) # Number of words to skip (backwards)
name_op('LOAD_SUPER_ATTR', 141)
def_op('CALL_FUNCTION_EX', 142) # Flags
def_op('LOAD_FAST_AND_CLEAR', 143) # Local variable number
haslocal.append(143)

def_op('EXTENDED_ARG', 144)
EXTENDED_ARG = 144
Expand Down Expand Up @@ -268,6 +270,8 @@ def pseudo_op(name, op, real_ops):
pseudo_op('LOAD_ZERO_SUPER_METHOD', 264, ['LOAD_SUPER_ATTR'])
pseudo_op('LOAD_ZERO_SUPER_ATTR', 265, ['LOAD_SUPER_ATTR'])

pseudo_op('STORE_FAST_MAYBE_NULL', 266, ['STORE_FAST'])

MAX_PSEUDO_OPCODE = MIN_PSEUDO_OPCODE + len(_pseudo_ops) - 1

del def_op, name_op, jrel_op, jabs_op, pseudo_op
Expand Down
2 changes: 1 addition & 1 deletion Lib/test/support/bytecode_helper.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ def complete_insts_info(self, insts):
class CodegenTestCase(CompilationStepTestCase):

def generate_code(self, ast):
insts = compiler_codegen(ast, "my_file.py", 0)
insts, _ = compiler_codegen(ast, "my_file.py", 0)
return insts


Expand Down
24 changes: 6 additions & 18 deletions Lib/test/test_compile.py
Original file line number Diff line number Diff line change
Expand Up @@ -1352,14 +1352,11 @@ def test_multiline_list_comprehension(self):
and x != 50)]
""")
compiled_code, _ = self.check_positions_against_ast(snippet)
compiled_code = compiled_code.co_consts[0]
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'LIST_APPEND',
line=1, end_line=2, column=1, end_column=8, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=1, end_line=2, column=1, end_column=8, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
line=1, end_line=6, column=0, end_column=32, occurrence=1)

def test_multiline_async_list_comprehension(self):
snippet = textwrap.dedent("""\
Expand All @@ -1374,13 +1371,13 @@ async def f():
compiled_code, _ = self.check_positions_against_ast(snippet)
g = {}
eval(compiled_code, g)
compiled_code = g['f'].__code__.co_consts[1]
compiled_code = g['f'].__code__
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'LIST_APPEND',
line=2, end_line=3, column=5, end_column=12, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=2, end_line=3, column=5, end_column=12, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_CONST',
line=2, end_line=7, column=4, end_column=36, occurrence=1)

def test_multiline_set_comprehension(self):
Expand All @@ -1393,14 +1390,11 @@ def test_multiline_set_comprehension(self):
and x != 50)}
""")
compiled_code, _ = self.check_positions_against_ast(snippet)
compiled_code = compiled_code.co_consts[0]
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'SET_ADD',
line=1, end_line=2, column=1, end_column=8, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=1, end_line=2, column=1, end_column=8, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
line=1, end_line=6, column=0, end_column=32, occurrence=1)

def test_multiline_async_set_comprehension(self):
snippet = textwrap.dedent("""\
Expand All @@ -1415,13 +1409,13 @@ async def f():
compiled_code, _ = self.check_positions_against_ast(snippet)
g = {}
eval(compiled_code, g)
compiled_code = g['f'].__code__.co_consts[1]
compiled_code = g['f'].__code__
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'SET_ADD',
line=2, end_line=3, column=5, end_column=12, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=2, end_line=3, column=5, end_column=12, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_CONST',
line=2, end_line=7, column=4, end_column=36, occurrence=1)

def test_multiline_dict_comprehension(self):
Expand All @@ -1434,14 +1428,11 @@ def test_multiline_dict_comprehension(self):
and x != 50)}
""")
compiled_code, _ = self.check_positions_against_ast(snippet)
compiled_code = compiled_code.co_consts[0]
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'MAP_ADD',
line=1, end_line=2, column=1, end_column=7, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=1, end_line=2, column=1, end_column=7, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
line=1, end_line=6, column=0, end_column=32, occurrence=1)

def test_multiline_async_dict_comprehension(self):
snippet = textwrap.dedent("""\
Expand All @@ -1456,13 +1447,13 @@ async def f():
compiled_code, _ = self.check_positions_against_ast(snippet)
g = {}
eval(compiled_code, g)
compiled_code = g['f'].__code__.co_consts[1]
compiled_code = g['f'].__code__
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'MAP_ADD',
line=2, end_line=3, column=5, end_column=11, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=2, end_line=3, column=5, end_column=11, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_CONST',
line=2, end_line=7, column=4, end_column=36, occurrence=1)

def test_matchcase_sequence(self):
Expand Down Expand Up @@ -1711,9 +1702,6 @@ def test_column_offset_deduplication(self):
for source in [
"lambda: a",
"(a for b in c)",
"[a for b in c]",
"{a for b in c}",
"{a: b for c in d}",
]:
with self.subTest(source):
code = compile(f"{source}, {source}", "<test>", "eval")
Expand Down
7 changes: 5 additions & 2 deletions Lib/test/test_compiler_assemble.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ def complete_metadata(self, metadata, filename="myfile.py"):
metadata.setdefault(key, key)
for key in ['consts']:
metadata.setdefault(key, [])
for key in ['names', 'varnames', 'cellvars', 'freevars']:
for key in ['names', 'varnames', 'cellvars', 'freevars', 'fasthidden']:
metadata.setdefault(key, {})
for key in ['argcount', 'posonlyargcount', 'kwonlyargcount']:
metadata.setdefault(key, 0)
Expand All @@ -33,6 +33,9 @@ def assemble_test(self, insts, metadata, expected):

expected_metadata = {}
for key, value in metadata.items():
if key == "fasthidden":
# not exposed on code object
continue
if isinstance(value, list):
expected_metadata[key] = tuple(value)
elif isinstance(value, dict):
Expand All @@ -52,7 +55,7 @@ def test_simple_expr(self):
'filename' : 'avg.py',
'name' : 'avg',
'qualname' : 'stats.avg',
'consts' : [2],
'consts' : {2 : 0},
'argcount' : 2,
'varnames' : {'x' : 0, 'y' : 1},
}
Expand Down
Loading

0 comments on commit e40071d

Please sign in to comment.