NetBSD: Process.connections() leaks memory or segfaults

[giampaolo]

======================================================================
FAIL: psutil.tests.test_memory_leaks.TestProcessObjectLeaks.test_connections
----------------------------------------------------------------------
Traceback (most recent call last):
  File "psutil/tests/test_memory_leaks.py", line 317, in test_connections
    self.execute('connections', kind=kind)
  File "psutil/tests/test_memory_leaks.py", line 93, in execute
    % (rss2, rss3, difference))
AssertionError: rss2=392040448, rss3=1955799040, difference=1563758592

UPDATE

Actually it also segfaults:

test_memory_leaks.TestProcessObjectLeaks.test_connections ... [1]   Killed                  PYTHONWARNINGS=a...
*** Error code 137

[giampaolo]

Hi @ryoon / @ryoqun. Long time no see. Sorry to bother you, but I remember you wrote this code, and I don't know how to fix it (test has been failing for years literally). We have a memory leak that is easily reproducible on NetBSD with:

$ make test ARGS="psutil.tests.test_memleaks.TestModuleFunctionsLeaks.test_net_connections"
make build
PYTHONWARNINGS=all python3 setup.py build_ext -i `python3 -c  "import sys, os;  py36 = sys.version_info[:2] >= (3, 6);  cpus = os.cpu_count() or 1 if py36 else 1;  print('--parallel %s' % cpus if cpus > 1 else '')"`
running build_ext
copying build/lib.netbsd-8.0-amd64-cpython-311/psutil/_psutil_bsd.so -> psutil
copying build/lib.netbsd-8.0-amd64-cpython-311/psutil/_psutil_posix.so -> psutil
python3 -c "import psutil"  # make sure it actually worked
PSUTIL_SCRIPTS_DIR=`pwd`/scripts PYTHONWARNINGS=always PSUTIL_DEBUG=1 python3 psutil/tests/runner.py psutil.tests.test_memleaks.TestModuleFunctionsLeaks.test_net_connections
psutil.tests.test_memleaks.TestModuleFunctionsLeaks.test_net_connections ... 
Run #1: extra-mem=40.9M, per-call=209.6K, calls=200
Run #2: extra-mem=81.9M, per-call=209.7K, calls=400
Run #3: extra-mem=122.8M, per-call=209.7K, calls=600
Run #4: extra-mem=163.8M, per-call=209.6K, calls=800
Run #5: extra-mem=204.8M, per-call=209.7K, calls=1000
FAIL

======================================================================
FAIL: psutil.tests.test_memleaks.TestModuleFunctionsLeaks.test_net_connections
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/psutil/psutil/tests/test_memleaks.py", line 76, in wrapper
    return fun(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/psutil/psutil/tests/test_memleaks.py", line 427, in test_net_connections
    self.execute(lambda: psutil.net_connections(kind='all'))
  File "/home/vagrant/psutil/psutil/tests/__init__.py", line 1198, in execute
    self._check_mem(fun, times=times, retries=retries, tolerance=tolerance)
  File "/home/vagrant/psutil/psutil/tests/__init__.py", line 1173, in _check_mem
    raise self.fail(". ".join(messages))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Run #1: extra-mem=40.9M, per-call=209.6K, calls=200. Run #2: extra-mem=81.9M, per-call=209.7K, calls=400. Run #3: extra-mem=122.8M, per-call=209.7K, calls=600. Run #4: extra-mem=163.8M, per-call=209.6K, calls=800. Run #5: extra-mem=204.8M, per-call=209.7K, calls=1000

If you run make test-memleaks it will core dump after a while. I digged a little, and the problem seems to be here (psutil_get_files()):

psutil/psutil/arch/netbsd/socks.c

Lines 321 to 322 in 69bca49

	if (psutil_get_files() != 0)
	goto error;

Memory is being allocated at:

psutil/psutil/arch/netbsd/socks.c

Lines 148 to 149 in 69bca49

	struct kif *kif = malloc(sizeof(struct kif));
	kif->kif = &ki[j];

...and it's supposed to be freed in:

psutil/psutil/arch/netbsd/socks.c

Lines 79 to 87 in 69bca49

	// Clear kinfo_file results list.
	static void
	psutil_kiflist_clear(void) {
	while (!SLIST_EMPTY(&kihead)) {
	SLIST_REMOVE_HEAD(&kihead, kifs);
	}
	return;
	}

...but it doesn't. Do you think you have some time to take a look at this? The code is too arcane to me.

Thank you. CC @0-wiz-0

[0-wiz-0]

Hi @giampaolo
I think the problem is the psutil_kiflist_clear function - it clears up the list, but doesn't free the members.
It should probably be (inside the while loop):

     struct kif *kif = SLIST_FIRST(&kihead);
     SLIST_REMOVE_HEAD(&kihead, kifs);
     free(kif);

(completely untested)

[giampaolo]

Thank you @0-wiz-0 , that worked as intended, meaning that some memory gets freed. I committed a change here:
https://github.com/giampaolo/psutil/compare/netbsd-connections-leak?expand=1

I can solve the memory leak completely if I decomment these 2 free() calls:

• psutil/psutil/arch/netbsd/socks.c

  Line 147 in f3d401d

  // free(buf);

• psutil/psutil/arch/netbsd/socks.c

  Line 206 in f3d401d

  // free(pcb);

  
  ...but then the tests break (meaning the function no longer works as intended). I have to free() all the malloc()ed stuff right before exiting the function somehow.

[0-wiz-0]

Ok... looking closer I see that pointers into these two blocks of memory are kept active

psutil/psutil/arch/netbsd/socks.c

Line 127 in f3d401d

if ((buf = malloc(len + offset)) == NULL) {

and

psutil/psutil/arch/netbsd/socks.c

Line 139 in f3d401d

struct kinfo_file *ki = (struct kinfo_file *)(buf + offset);

So I think one solution would be to replace the pointer:

psutil/psutil/arch/netbsd/socks.c

Line 143 in f3d401d

kif->kif = &ki[j];

with a copy. I.e. malloc another struct kinfo_file ki, copy the contents of &ki[j] into it and use it for kif->kif, and free it in the loops you just modified (and similar for struct kinfo_pcb kp in the pcb loop).

Then you can uncomment the free() calls.

[giampaolo]

I have fixed this in #2341 by using a different approach. Thanks a lot Thomas @0-wiz-0. Without your suggestion I would have been stuck. I'm happy to finally close this 7 years old critical issue.

[0-wiz-0]

I'm glad you could fix this - thank you!

[ryoon]

I am sorry. I am too late. Thank you very much for your fix.