Numerous refleaks on main

[JelleZijlstra]

https://buildbot.python.org/all/#/release_status shows that the refleak buildbots are failing on current main. Example output on https://buildbot.python.org/all/#/builders/75/builds/745/steps/5/logs/warnings__111_.

Failing tests:

test_functools leaked [1, 1, 2] references, sum=4
test_warnings leaked [4, 4, 4] references, sum=12
test_ipaddress leaked [6, 6, 6] references, sum=18
test_zoneinfo leaked [7, 6, 11] references, sum=24
FAIL: test_flock (__main__.FNTLEINTRTest.test_flock)
test_random leaked [1, 2, 4] references, sum=7
test_statistics leaked [175, 174, 175] references, sum=524
test_dataclasses leaked [3, 3, 2] references, sum=8
test_argparse leaked [69, 70, 76] references, sum=215
/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
test_tarfile leaked [17, 17, 16] references, sum=50
test_typing leaked [95, 121, 95] references, sum=311
test_typing leaked [1, 14, 1] memory blocks, sum=16
test_lib2to3 leaked [5, 8, 6] references, sum=19
test_shutil leaked [2, 2, 2] references, sum=6
test_unittest leaked [8, 12, 15] references, sum=35
test_logging leaked [3, 2, 1] references, sum=6
test__xxsubinterpreters leaked [46, 45, 45] references, sum=136
test_mailbox leaked [2, 3, 2] references, sum=7
test_socket leaked [24, 23, 23] references, sum=70
test_htmlparser leaked [2, 2, 2] references, sum=6
test_regrtest leaked [9, 12, 8] references, sum=29
test_datetime leaked [11, 11, 11] references, sum=33
test_nntplib leaked [1, 3, 2] references, sum=6
test_ftplib leaked [10, 13, 9] references, sum=32
test_interpreters leaked [54, 54, 54] references, sum=162
test_zipfile leaked [3, 4, 4] references, sum=11
test_sys_settrace leaked [6, 6, 6] references, sum=18
test_descr leaked [3, 65, 67] references, sum=135
test_compile leaked [7, 7, 7] references, sum=21
test_bytes leaked [2, 2, 2] references, sum=6
test_ast leaked [9, 9, 9] references, sum=27
test_poplib leaked [3, 3, 2] references, sum=8
test_types leaked [8, 50, 43] references, sum=101
test_concurrent_futures leaked [11, 12, 11] references, sum=34
test_genericalias leaked [2, 2, 3] references, sum=7
test_math leaked [32, 31, 33] references, sum=96
test_numeric_tower leaked [2, 1, 2] references, sum=5
test_capi leaked [99, 99, 99] references, sum=297
test_import leaked [170, 171, 168] references, sum=509
test_import leaked [71, 71, 70] memory blocks, sum=212
.test test_monitoring failed -- multiple errors occurred; run in verbose mode for details
test_selectors leaked [5, 6, 4] references, sum=15
test_email leaked [27, 26, 25] references, sum=78
test_subprocess leaked [1, 1, 1] references, sum=3
test_struct leaked [1, 1, 1] references, sum=3
test_weakref leaked [1, 1, 1] references, sum=3
test_imaplib leaked [2, 2, 1] references, sum=5
0:45:54 load avg: 0.13 Re-running failed tests in verbose mode
0:45:54 load avg: 0.13 Re-running test_functools in verbose mode
test_functools leaked [1, 1, 2] references, sum=4
0:45:55 load avg: 0.13 Re-running test_warnings in verbose mode
test_warnings leaked [4, 4, 4] references, sum=12
0:45:59 load avg: 0.20 Re-running test_ipaddress in verbose mode
test_ipaddress leaked [6, 6, 6] references, sum=18
0:46:00 load avg: 0.20 Re-running test_zoneinfo in verbose mode
test_zoneinfo leaked [7, 6, 11] references, sum=24

I saw similar leaks triggering the refleak buildbots on #103866 and #103764, before I realized the issue was probably on main.

Based on @sunmy2019's work in #103764 (comment), we're likely leaking references to the object.__setattr__ wrapper descriptor.

Linked PRs

• gh-103879: Fix refleak in super specialization #103882

[JelleZijlstra]

The failure on test.test_htmlparser bisects to

ef25febcf2ede92a03c5ea00a13e167e0b5cb274 is the first bad commit
commit ef25febcf2ede92a03c5ea00a13e167e0b5cb274
Author: Carl Meyer <carl@oddbird.net>
Date:   Tue Apr 25 11:45:51 2023 -0600

    gh-87729: specialize LOAD_SUPER_ATTR_METHOD (#103809)

@carljm

[carljm]

Given that refleak buildbots passed on #103882, it seems this is fixed now.

[carljm]

Thanks @JelleZijlstra for fixing!

[JelleZijlstra]

No problem!

https://buildbot.python.org/all/#/release_status is still red but I assume the buildbots are running now. Given that the refleak buildbots were green on my PR, we probably did fix the issue.

[JelleZijlstra]

We might have another refleak on test_import, visible on main and on #103866 link.

[JelleZijlstra]

I wasn't able to reproduce the new leak locally (tried both MacOS and Linux). It seems to happen only on one buildbot.

[sunmy2019]

The leak of test_import is still visible on that buildbot, and the issue persists for a long time

Visible at
https://buildbot.python.org/all/#/builders/205/builds/696
... (note: logs will be cleared for old tests, soon the log will not be visible)

Usually, rerunning the tests will let it pass. (Warning, no errors)

I suspect something changed in the test environment.

[sunmy2019]

It seems to happen only on one buildbot.

This is not true. test_import leaked on the first run and passed in the second run. Visible on various arch*os combinations.

https://buildbot.python.org/all/#/builders/766/builds/770
https://buildbot.python.org/all/#/builders/474/builds/1161
https://buildbot.python.org/all/#/builders/411/builds/1216
https://buildbot.python.org/all/#/builders/802/builds/814
https://buildbot.python.org/all/#/builders/210/builds/1196
https://buildbot.python.org/all/#/builders/18/builds/1233
https://buildbot.python.org/all/#/builders/409/builds/1169

...

[sunmy2019]

@JelleZijlstra I can reproduce it with

./python -m test -j1 -R 3:3 test_import

Single processing will not cause leakage. -j is necessary.

[JelleZijlstra]

Good catch, that reproduces it for me too.

[sunmy2019]

Current status:

Found most leakage is from test_import.SinglephaseInitTests.

ref change of type dict: +7
ref change of type float: +21
ref change of type getset_descriptor: +5
ref change of type int: +16
ref change of type str: ... (immortal str not counted)
ref change of type tuple: +7
ref change of type type: +12
ref change of type weakref.ReferenceType: +5

I took a quick look, ref change of type type: +12 contains many instances created by

    state->error = PyErr_NewException("_testsinglephase.error", NULL, NULL);

Modules/_testsinglephase.c:init_module is called several times during -j tests, but only once with the single-threaded one.

cpython/Modules/_testsinglephase.c

Lines 125 to 148 in 738c226

	static int
	init_module(PyObject *module, module_state *state)
	{
	if (PyModule_AddObjectRef(module, "error", state->error) != 0) {
	return -1;
	}
	if (PyModule_AddObjectRef(module, "int_const", state->int_const) != 0) {
	return -1;
	}
	if (PyModule_AddObjectRef(module, "str_const", state->str_const) != 0) {
	return -1;
	}
	double d = _PyTime_AsSecondsDouble(state->initialized);
	PyObject *initialized = PyFloat_FromDouble(d);
	if (initialized == NULL) {
	return -1;
	}
	if (PyModule_AddObjectRef(module, "_module_initialized", initialized) != 0) {
	return -1;
	}
	return 0;
	}

At least, I found initialized is leaked here. PyModule_AddObjectRef creates a new reference. The previous leak statistics indicate state->error may also be leaked elsewhere.

@ericsnowcurrently Can you please take a look since the last change is yours?

[sunmy2019]

@ericvsmith I tracked down the problem. Hope you can fix it and add me as a co-author.

1

initialized is leaked as mentioned above.

2

_testsinglephase_with_state module. Its md_state is not released by _testinternalcapi.clear_extension or _clear_globals.

When I manually add and call the clean-up function. Ref leaks are gone.

3

cpython/Lib/test/test_import/__init__.py

Lines 2458 to 2462 in 738c226

	# Start with an interpreter that gets destroyed right away.
	base = self.import_in_subinterp(postscript='''
	# Attrs set after loading are not in m_copy.
	mod.spam = 'spam, spam, mash, spam, eggs, and spam'
	''')

No cleanup was made against base, which is leaking ref. This line looks intentional from the comment, but it does leak things.

---

Most of the ref leaks in test_import are here. Hopefully, it should be all.

[ericsnowcurrently]

@sunmy2019, there are known refleaks in test_import. See gh-102251. Are you talking about more than that?

Also, your analysis is helpful. Thanks!

[sunmy2019]

Are you talking about more than that?

Yes, I am talking about test_import. The link you give seems to be test_imp.

The ref leak in test_import is introduced at least two months ago:
https://buildbot.python.org/all/#/builders/205/builds/696

and still visible and reproducible on main:

./python -m test -j1 -R 3:3 test_import

---

The above guide should help you clear major ref leaks in test_import.

[ericsnowcurrently]

The tests in question were moved from test_imp to test_import.

[JelleZijlstra]

I think the main refleaks are fixed now, closing this as the test_import refleaks are tracked in #102251.