gh-104252: Immortalize Py_EMPTY_KEYS #104253

ericsnowcurrently · 2023-05-06T23:59:09Z

This was missed in gh-19474. It matters for with a per-interpreter GIL since PyDictKeysObject.dk_refcnt breaks isolation and leads to races.

cc @eduardo-elizondo

Issue: Immortalize Py_EMPTY_KEYS #104252

sunmy2019

There are other places with direct changes to dk_refcnt. They are not guarded.

cpython/Objects/dictobject.c

Line 639 in 4729383

dk->dk_refcnt = 1;

cpython/Objects/dictobject.c

Lines 822 to 828 in 4729383

    
               /* Since we copied the keys table we now have an extra reference 
        
                  in the system.  Manually call increment _Py_RefTotal to signal that 
        
                  we have it now; calling dictkeys_incref would be an error as 
        
                  keys->dk_refcnt is already set to 1 (after memcpy). */ 
        
           #ifdef Py_REF_DEBUG 
        
               _Py_IncRefTotal(_PyInterpreterState_GET()); 
        
           #endif

cpython/Objects/dictobject.c

Lines 1530 to 1538 in 4729383

    
                   // We can not use free_keys_object here because key's reference 
        
                   // are moved already. 
        
           #ifdef Py_REF_DEBUG 
        
                   _Py_DecRefTotal(_PyInterpreterState_GET()); 
        
           #endif 
        
                   if (oldkeys == Py_EMPTY_KEYS) { 
        
                       oldkeys->dk_refcnt--; 
        
                       assert(oldkeys->dk_refcnt > 0); 
        
                   }

Objects/dictobject.c

ericsnowcurrently · 2023-05-08T13:47:12Z

@sunmy2019, thanks for pointing out the other uses of dk_refcnt. I'll fix that.

[done]

Objects/dictobject.c

Include/internal/pycore_dict_state.h

Objects/dictobject.c

methane · 2023-05-09T01:56:53Z

Please remove these lines too:

cpython/Objects/dictobject.c

Line 849 in 58585a9

dictkeys_incref(Py_EMPTY_KEYS);

cpython/Objects/dictobject.c

Line 1347 in 58585a9

dictkeys_decref(interp, Py_EMPTY_KEYS);

And Move _Py_DecRefTotal() into if (oldkeys != Py_EMPTY_KEYS) block.

cpython/Objects/dictobject.c

Lines 1545 to 1548 in 58585a9

    
           #ifdef Py_REF_DEBUG 
        
                   _Py_DecRefTotal(_PyInterpreterState_GET()); 
        
           #endif 
        
                   if (oldkeys != Py_EMPTY_KEYS) {

ericsnowcurrently · 2023-05-09T03:35:51Z

Thanks for the feedback. I'll clean things up tomorrow.

bedevere-bot · 2023-05-09T18:51:10Z

🤖 New build scheduled with the buildbot fleet by @sunmy2019 for commit 5aabfd7 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

sunmy2019

LGTM! now it's much cleaner

ericsnowcurrently · 2023-05-10T13:42:49Z

Thanks for the reviews!

* main: pythonGH-102181: Improve specialization stats for SEND (pythonGH-102182) pythongh-103000: Optimise `dataclasses.asdict` for the common case (python#104364) pythongh-103538: Remove unused TK_AQUA code (pythonGH-103539) pythonGH-87695: Fix OSError from `pathlib.Path.glob()` (pythonGH-104292) pythongh-104263: Rely on Py_NAN and introduce Py_INFINITY (pythonGH-104202) pythongh-104010: Separate and improve docs for `typing.get_origin` and `typing.get_args` (python#104013) pythongh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic (python#104355) pythongh-103960: Dark mode: invert image brightness (python#103983) pythongh-104252: Immortalize Py_EMPTY_KEYS (pythongh-104253) pythongh-101819: Clean up _io windows console io after pythongh-104197 (python#104354) pythongh-101819: Harden _io init (python#104352) pythongh-103247: clear the module cache in a test in test_importlib/extensions/test_loader.py (pythonGH-104226) pythongh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format (python#103849) pythongh-74895: adjust tests to work on Solaris (python#104326) pythongh-101819: Refactor _io in preparation for module isolation (python#104334) pythongh-90953: Don't use deprecated AST nodes in clinic.py (python#104322) pythongh-102327: Extend docs for "url" and "headers" parameters to HTTPConnection.request() pythongh-104328: Fix typo in ``typing.Generic`` multiple inheritance error message (python#104335)

* main: (27 commits) pythongh-87849: fix SEND specialization family definition (pythonGH-104268) pythongh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic (python#104384) pythongh-101819: Adapt _io._Buffered* methods to Argument Clinic (python#104367) pythongh-101819: Refactor `_io` futher in preparation for module isolation (python#104369) pythongh-101819: Adapt _io.TextIOBase methods to Argument Clinic (python#104383) pythongh-101117: Improve accuracy of sqlite3.Cursor.rowcount docs (python#104287) pythonGH-92184: Convert os.altsep to '/' in filenames when creating ZipInfo objects (python#92185) pythongh-104357: fix inlined comprehensions that close over iteration var (python#104368) pythonGH-90208: Suppress OSError exceptions from `pathlib.Path.glob()` (pythonGH-104141) pythonGH-102181: Improve specialization stats for SEND (pythonGH-102182) pythongh-103000: Optimise `dataclasses.asdict` for the common case (python#104364) pythongh-103538: Remove unused TK_AQUA code (pythonGH-103539) pythonGH-87695: Fix OSError from `pathlib.Path.glob()` (pythonGH-104292) pythongh-104263: Rely on Py_NAN and introduce Py_INFINITY (pythonGH-104202) pythongh-104010: Separate and improve docs for `typing.get_origin` and `typing.get_args` (python#104013) pythongh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic (python#104355) pythongh-103960: Dark mode: invert image brightness (python#103983) pythongh-104252: Immortalize Py_EMPTY_KEYS (pythongh-104253) pythongh-101819: Clean up _io windows console io after pythongh-104197 (python#104354) pythongh-101819: Harden _io init (python#104352) ...

eduardo-elizondo · 2023-06-19T15:32:56Z

Nice fix! I just got around to looking at this PR, it was weird thinking about the PyDictKeysObject since they were not PyObjects, so I just let them be. That being said, this LGTM specially if we are trying to share as much state as possible.

ericsnowcurrently added 2 commits May 6, 2023 17:55

Factor out _dict_state_INIT.

5bc2469

Immortalize Py_EMPTY_KEYS.

e472d94

ericsnowcurrently added the skip news label May 6, 2023

ericsnowcurrently requested a review from markshannon May 6, 2023 23:59

ericsnowcurrently requested a review from methane as a code owner May 6, 2023 23:59

bedevere-bot mentioned this pull request May 6, 2023

Immortalize Py_EMPTY_KEYS #104252

Closed

bedevere-bot added the awaiting core review label May 6, 2023

ericsnowcurrently requested review from methane and removed request for methane May 6, 2023 23:59

sunmy2019 requested changes May 7, 2023

View reviewed changes

Objects/dictobject.c Outdated Show resolved Hide resolved

ericsnowcurrently added 2 commits May 8, 2023 07:57

Handle other uses of dk_refcnt.

87351e0

Add a note.

f5da088

methane reviewed May 8, 2023

View reviewed changes

Objects/dictobject.c Outdated Show resolved Hide resolved

sunmy2019 reviewed May 8, 2023

View reviewed changes

Include/internal/pycore_dict_state.h Outdated Show resolved Hide resolved

ericsnowcurrently added 2 commits May 8, 2023 13:25

Drop an old line.

db8612f

Drop an unnecessary condition.

58585a9

methane reviewed May 9, 2023

View reviewed changes

Objects/dictobject.c Outdated Show resolved Hide resolved

methane reviewed May 9, 2023

View reviewed changes

Objects/dictobject.c Show resolved Hide resolved

ericsnowcurrently added 4 commits May 9, 2023 10:11

Drop an unnecessary assert.

6b5c952

Do not bother inc/dec-refing Py_EMPTY_KEYS.

bdbce5f

Do not update Py_Reftotal for Py_EMPTY_KEYS.

aab493a

Clarify the comment about dictkeys_incref() and dictkeys_decref().

5aabfd7

ericsnowcurrently requested a review from sunmy2019 May 9, 2023 17:45

sunmy2019 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label May 9, 2023

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label May 9, 2023

methane approved these changes May 10, 2023

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels May 10, 2023

This comment was marked as duplicate.

Sign in to view

sunmy2019 approved these changes May 10, 2023

View reviewed changes

ericsnowcurrently merged commit b8f7ab5 into python:main May 10, 2023

bedevere-bot removed the awaiting merge label May 10, 2023

ericsnowcurrently deleted the immortalize-empty-keys branch May 10, 2023 13:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-104252: Immortalize Py_EMPTY_KEYS #104253

gh-104252: Immortalize Py_EMPTY_KEYS #104253

ericsnowcurrently commented May 6, 2023 •

edited by bedevere-bot

Loading

sunmy2019 left a comment •

edited

Loading

ericsnowcurrently commented May 8, 2023 •

edited

Loading

methane commented May 9, 2023

ericsnowcurrently commented May 9, 2023

bedevere-bot commented May 9, 2023

This comment was marked as duplicate.

sunmy2019 left a comment

ericsnowcurrently commented May 10, 2023

eduardo-elizondo commented Jun 19, 2023

	/* Since we copied the keys table we now have an extra reference
	in the system. Manually call increment _Py_RefTotal to signal that
	we have it now; calling dictkeys_incref would be an error as
	keys->dk_refcnt is already set to 1 (after memcpy). */
	#ifdef Py_REF_DEBUG
	_Py_IncRefTotal(_PyInterpreterState_GET());
	#endif

	// We can not use free_keys_object here because key's reference
	// are moved already.
	#ifdef Py_REF_DEBUG
	_Py_DecRefTotal(_PyInterpreterState_GET());
	#endif
	if (oldkeys == Py_EMPTY_KEYS) {
	oldkeys->dk_refcnt--;
	assert(oldkeys->dk_refcnt > 0);
	}

gh-104252: Immortalize Py_EMPTY_KEYS #104253

gh-104252: Immortalize Py_EMPTY_KEYS #104253

Conversation

ericsnowcurrently commented May 6, 2023 • edited by bedevere-bot Loading

sunmy2019 left a comment • edited Loading

Choose a reason for hiding this comment

ericsnowcurrently commented May 8, 2023 • edited Loading

methane commented May 9, 2023

ericsnowcurrently commented May 9, 2023

bedevere-bot commented May 9, 2023

This comment was marked as duplicate.

sunmy2019 left a comment

Choose a reason for hiding this comment

ericsnowcurrently commented May 10, 2023

eduardo-elizondo commented Jun 19, 2023

ericsnowcurrently commented May 6, 2023 •

edited by bedevere-bot

Loading

sunmy2019 left a comment •

edited

Loading

ericsnowcurrently commented May 8, 2023 •

edited

Loading