Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate the interpreter #98831

Closed
gvanrossum opened this issue Oct 28, 2022 · 2 comments
Closed

Generate the interpreter #98831

gvanrossum opened this issue Oct 28, 2022 · 2 comments
Assignees
@gvanrossum
Labels
3.12 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement

Comments

@gvanrossum
Copy link
Member

gvanrossum commented Oct 28, 2022 •
edited by bedevere-bot
Loading

Overview

Over in faster-cpython/ideas we've been exploring the idea of generating the interpreter from a set of instruction definitions. This will eventually enable having multiple versions of the interpreter (e.g. with and without tracing enabled), and it will allow us to automatically combine instructions using a powerful notation (e.g. super(LOAD_FAST__LOAD_FAST) = LOAD_FAST + LOAD_FAST;). It will also let us auto-generate things like stack_effect().

We are planning to land at least an early version of this work in 3.12. We have a tentative grammar for instruction definition DSL -- which will undoubtedly undergo several iterations before we've settled. We have a first draft of the tooling ready for review (which currently reproduces the status quo) done.

Once the first version of the tooling has landed we expect to iterate quickly, using this issue an umbrella issue for our PRs to link to.

References
@gvanrossum gvanrossum added the type-feature A feature request or enhancement label Oct 28, 2022
@bedevere-bot bedevere-bot mentioned this issue Oct 28, 2022
Merged
@kumaraditya303 kumaraditya303 added interpreter-core (Objects, Python, Grammar, and Parser dirs) 3.12 bugs and security fixes labels Oct 29, 2022
@gvanrossum gvanrossum self-assigned this Oct 31, 2022
@gvanrossum gvanrossum changed the title Generating the interpreter Generate the interpreter Oct 31, 2022
@qexat qexat mentioned this issue Oct 31, 2022
Closed
gvanrossum added a commit that referenced this issue Nov 3, 2022
@gvanrossum
GH-98831: "Generate" the interpreter (#98830)
41bc101 
The switch cases (really TARGET(opcode) macros) have been moved from ceval.c to generated_cases.c.h. That file is generated from instruction definitions in bytecodes.c (which impersonates a C file so the C code it contains can be edited without custom support in e.g. VS Code).

The code generator lives in Tools/cases_generator (it has a README.md explaining how it works). The DSL used to describe the instructions is a work in progress, described in https://github.com/faster-cpython/ideas/blob/main/3.12/interpreter_definition.md.

This is surely a work-in-progress. An easy next step could be auto-generating super-instructions.

**IMPORTANT: Merge Conflicts**

If you get a merge conflict for instruction implementations in ceval.c, your best bet is to port your changes to bytecodes.c. That file looks almost the same as the original cases, except instead of `TARGET(NAME)` it uses `inst(NAME)`, and the trailing `DISPATCH()` call is omitted (the code generator adds it automatically).
@bedevere-bot bedevere-bot mentioned this issue Nov 3, 2022
Merged
gvanrossum added a commit that referenced this issue Nov 3, 2022
@gvanrossum
GH-98831: Remove redundant extract_cases.py script (GH-99065)
916af11 
Debt I owe from PR GH-98830.
This was referenced Nov 4, 2022
Merged
Merged
Merged
Merged
gvanrossum pushed a commit that referenced this issue Nov 4, 2022
@brandtbucher
GH-98831: Add regen-cases to regen-all (#99107)
e99380c
gvanrossum added a commit that referenced this issue Nov 4, 2022
@gvanrossum @brandtbucher
GH-98831: Auto-generate PREDICTED() macro calls (#99102)
c885623 
Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
@gvanrossum gvanrossum mentioned this issue Nov 4, 2022
Merged
gvanrossum pushed a commit that referenced this issue Nov 4, 2022
@markshannon
GH-98831: Add some macros definitions to bytecodes.c to reduce IDE wa…
7a020b8 
…rnings. (#99093)
@bedevere-bot bedevere-bot mentioned this issue Nov 5, 2022
Merged
gvanrossum added a commit that referenced this issue Nov 6, 2022
@gvanrossum @CAM-Gerlach
GH-98831: Implement super-instruction generation (#99084)
7dcd28e 
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
gvanrossum added a commit that referenced this issue Nov 8, 2022
@gvanrossum
GH-98831: Simple input-output stack effects for bytecodes.c (#99120)
f1a6546
This was referenced Nov 9, 2022
Merged
Merged
gvanrossum added a commit that referenced this issue Nov 10, 2022
@gvanrossum @brandtbucher
GH-98831: Remove all remaining DISPATCH() calls from bytecodes.c (#99271
694cdb2 
)

Also mark those opcodes that have no stack effect as such.

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
gvanrossum added a commit to gvanrossum/cpython that referenced this issue Nov 10, 2022
@gvanrossum @brandtbucher
pythonGH-98831: Remove all remaining DISPATCH() calls from bytecodes.c (
d094e42 
python#99271)

Also mark those opcodes that have no stack effect as such.

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
@bedevere-bot bedevere-bot mentioned this issue Nov 12, 2022
Closed
ethanfurman pushed a commit to ethanfurman/cpython that referenced this issue Nov 12, 2022
@gvanrossum @brandtbucher @ethanfurman
pythonGH-98831: Remove all remaining DISPATCH() calls from bytecodes.c (
5729ea9 
python#99271)

Also mark those opcodes that have no stack effect as such.

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
@bedevere-bot bedevere-bot mentioned this issue Nov 14, 2022
Merged
gvanrossum added a commit that referenced this issue Nov 16, 2022
@gvanrossum
GH-98831: Implement basic cache effects (#99313)
e37744f
@bedevere-bot bedevere-bot mentioned this issue Nov 16, 2022
Merged
@bedevere-bot bedevere-bot mentioned this issue Feb 3, 2023
Merged
iritkatriel added a commit that referenced this issue Feb 3, 2023
@iritkatriel
gh-98831: rewrite SEND, GET_YIELD_FROM_ITER, RETURN_GENERATOR in the …
04e06e2 
…instruction definition DSL (#101516)
iritkatriel added a commit to iritkatriel/cpython that referenced this issue Feb 3, 2023
@iritkatriel
pythongh-98831: rewrite SEND, GET_YIELD_FROM_ITER, RETURN_GENERATOR i…
77af074 
…n the instruction definition DSL (python#101516)
iritkatriel added a commit that referenced this issue Feb 3, 2023
@iritkatriel
gh-98831: rewrite MAKE_FUNCTION and BUILD_SLICE in the instruction de…
433fb3e 
…finition DSL (#101529)
@bedevere-bot bedevere-bot mentioned this issue Feb 6, 2023
Merged
iritkatriel added a commit that referenced this issue Feb 6, 2023
@iritkatriel
gh-98831: rewrite COPY and SWAP in the instruction definition DSL (#1…
3875276 
…01620)
This was referenced Feb 7, 2023
Merged
Merged
Merged
Merged
gvanrossum added a commit that referenced this issue Feb 7, 2023
@gvanrossum
gh-98831: Move DSL documentation here from ideas repo (#101629)
694e346
@bedevere-bot bedevere-bot mentioned this issue Feb 7, 2023
Merged
gvanrossum added a commit that referenced this issue Feb 7, 2023
@gvanrossum @iritkatriel
gh-98831: Modernize the FOR_ITER family of instructions (#101626)
d54b8d8 
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
iritkatriel added a commit that referenced this issue Feb 7, 2023
@iritkatriel
gh-98831: rewrite UNPACK_EX, UNPACK_SEQUENCE, UNPACK_SEQUENCE_TWO_TUP…
dec1ab0 
…LE in the instruction definition DSL (#101641)
@bedevere-bot bedevere-bot mentioned this issue Feb 7, 2023
Merged
@gvanrossum
Copy link
Member Author

gvanrossum commented Feb 7, 2023 •
edited
Loading

Once these PRs land, every opcode has cache/stack effect specifications, and I will disable the legacy syntax from the parser.

gvanrossum added a commit that referenced this issue Feb 7, 2023
@gvanrossum
gh-98831: Finish the UNPACK_SEQUENCE family (#101666)
aacbdb0 
New generator feature: Generate useful glue for output arrays, so you can just write values to the output array (no bounds checking). Rewrote UNPACK_SEQUENCE_TWO_TUPLE to use this, and also UNPACK_SEQUENCE_{TUPLE,LIST}.
gvanrossum added a commit that referenced this issue Feb 8, 2023
@gvanrossum
gh-98831: Modernize FORMAT_VALUE (#101628)
b2b85b5 
Generator update: support balanced parentheses and brackets in conditions and size expressions.
gvanrossum added a commit that referenced this issue Feb 8, 2023
@gvanrossum
gh-98831: Modernize CALL_FUNCTION_EX (#101627)
a9f0144 
New generator feature: Move CHECK_EVAL_BREAKER() call to just before DISPATCH().
@bedevere-bot bedevere-bot mentioned this issue Feb 8, 2023
Merged
gvanrossum added a commit that referenced this issue Feb 8, 2023
@gvanrossum
gh-98831: Modernize CALL and family (#101508)
616aec1 
Includes a slight improvement to `DECREF_INPUTS()`.
gvanrossum added a commit that referenced this issue Feb 9, 2023
@gvanrossum
gh-98831: Use opcode metadata for stack_effect() (#101704)
65b7b6b 
* Write output and metadata in a single run
  This halves the time to run the cases generator
  (most of the time goes into parsing the input).
* Declare or define opcode metadata based on NEED_OPCODE_TABLES
* Use generated metadata for stack_effect()
* compile.o depends on opcode_metadata.h
* Return -1 from _PyOpcode_num_popped/pushed for unknown opcode
carljm added a commit to carljm/cpython that referenced this issue Feb 9, 2023
@carljm
Merge branch 'main' into inlinecomp2
be3becc 
* main: (82 commits)
  pythongh-101670: typo fix in PyImport_ExtendInittab() (python#101723)
  pythonGH-99293: Document that `Py_TPFLAGS_VALID_VERSION_TAG` shouldn't be used. (#pythonGH-101736)
  no-issue: Add Dong-hee Na as the cjkcodecs codeowner (pythongh-101731)
  pythongh-101678: Merge math_1_to_whatever() and math_1() (python#101730)
  pythongh-101678: refactor the math module to use special functions from c11 (pythonGH-101679)
  pythongh-85984: Remove legacy Lib/pty.py code. (python#92365)
  pythongh-98831: Use opcode metadata for stack_effect() (python#101704)
  pythongh-101283: Version was just released, so should be changed in 3.11.3 (pythonGH-101719)
  pythongh-101283: Fix use of unbound variable (pythonGH-101712)
  pythongh-101283: Improved fallback logic for subprocess with shell=True on Windows (pythonGH-101286)
  pythongh-101277: Port more itertools static types to heap types (python#101304)
  pythongh-98831: Modernize CALL and family (python#101508)
  pythonGH-101696: invalidate type version tag in `_PyStaticType_Dealloc` (python#101697)
  pythongh-100221: Fix creating dirs in `make sharedinstall` (pythonGH-100329)
  pythongh-101670: typo fix in PyImport_AppendInittab() (pythonGH-101672)
  pythongh-101196: Make isdir/isfile/exists faster on Windows (pythonGH-101324)
  pythongh-101614: Don't treat python3_d.dll as a Python DLL when checking extension modules for incompatibility (pythonGH-101615)
  pythongh-100933: Improve `check_element` helper in `test_xml_etree` (python#100934)
  pythonGH-101578: Normalize the current exception (pythonGH-101607)
  pythongh-47937: Note that Popen attributes are read-only (python#93070)
  ...
@bedevere-bot bedevere-bot mentioned this issue Mar 13, 2023
Merged
gvanrossum added a commit that referenced this issue Mar 13, 2023
@gvanrossum
gh-98831: Use DECREF_INPUTS() more (#102409)
392f2ad
carljm added a commit to carljm/cpython that referenced this issue Mar 14, 2023
@carljm
Merge branch 'main' into inlinecomp2
6c5f269 
* main: (50 commits)
  pythongh-102674: Remove _specialization_stats from Lib/opcode.py (python#102685)
  pythongh-102660: Handle m_copy Specially for the sys and builtins Modules (pythongh-102661)
  pythongh-102354: change python3 to python in docs examples (python#102696)
  pythongh-81057: Add a CI Check for New Unsupported C Global Variables (pythongh-102506)
  pythonGH-94851: check unicode consistency of static strings in debug mode (python#102684)
  pythongh-100315: clarification to `__slots__` docs. (python#102621)
  pythonGH-100227: cleanup initialization of global interned dict (python#102682)
  doc: Remove a duplicate 'versionchanged' in library/asyncio-task (pythongh-102677)
  pythongh-102013: Add PyUnstable_GC_VisitObjects (python#102014)
  pythonGH-102670: Use sumprod() to simplify, speed up, and improve accuracy of statistics functions (pythonGH-102649)
  pythongh-102627: Replace address pointing toward malicious web page (python#102630)
  pythongh-98831: Use DECREF_INPUTS() more (python#102409)
  pythongh-101659: Avoid Allocation for Shared Exceptions in the _xxsubinterpreters Module (pythongh-102659)
  pythongh-101524: Fix the ChannelID tp_name (pythongh-102655)
  pythongh-102069: Fix `__weakref__` descriptor generation for custom dataclasses (python#102075)
  pythongh-98169 dataclasses.astuple support DefaultDict (python#98170)
  pythongh-102650: Remove duplicate include directives from multiple source files (python#102651)
  pythonGH-100987: Don't cache references to the names and consts array in `_PyEval_EvalFrameDefault`. (python#102640)
  pythongh-87092: refactor assemble() to a number of separate functions, which do not need the compiler struct (python#102562)
  pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (python#102631)
  ...
@gvanrossum
Copy link
Member Author

Let's close this issue, the interpreter is now definitely being generated. Subsequent changes may reference this (if small) or have their own issue.

Fidget-Spinner pushed a commit to Fidget-Spinner/cpython that referenced this issue Mar 27, 2023
@gvanrossum @Fidget-Spinner
pythongh-98831: Use DECREF_INPUTS() more (python#102409)
b99132d
warsaw pushed a commit to warsaw/cpython that referenced this issue Apr 11, 2023
@gvanrossum @warsaw
pythongh-98831: Use DECREF_INPUTS() more (python#102409)
9cf44ab
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gvanrossum gvanrossum

Labels
3.12 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gvanrossum @kumaraditya303