[Bug] RuntimeError in mem.py

[i-am-sijia]

Describe the bug

I have run into RuntimeError: NtQueryVirtualMemory bufsize is too large in ActivitySim test runs with large sample population. It crashes when tracing memory info, even though the run hasn't exploited the RAM on the machine. Below is the full traceback:

25/10/2022 13:25:28 - ERROR - activitysim.cli.run - activitysim run encountered an unrecoverable error
Traceback (most recent call last):
  File "c:\users\wangs1\workspace\consortium-activitysim\activitysim\activitysim\cli\run.py", line 352, in run
    pipeline.run(
  File "c:\users\wangs1\workspace\consortium-activitysim\activitysim\activitysim\core\pipeline.py", line 691, in run
    run_model(model)
  File "c:\users\wangs1\workspace\consortium-activitysim\activitysim\activitysim\core\pipeline.py", line 534, in run_model
    mem.trace_memory_info(f"pipeline.run_model {model_name} finished")
  File "c:\users\wangs1\workspace\consortium-activitysim\activitysim\activitysim\core\mem.py", line 200, in trace_memory_info
    info = current_process.memory_full_info()
  File "c:\Users\wangs1\workspace\ASIM-DEV\lib\site-packages\psutil\__init__.py", line 1078, in memory_full_info
    return self._proc.memory_full_info()
  File "c:\Users\wangs1\workspace\ASIM-DEV\lib\site-packages\psutil\_pswindows.py", line 688, in wrapper
    return fun(self, *args, **kwargs)
  File "c:\Users\wangs1\workspace\ASIM-DEV\lib\site-packages\psutil\_pswindows.py", line 851, in memory_full_info
    uss = cext.proc_memory_uss(self.pid)
RuntimeError: NtQueryVirtualMemory bufsize is too large

Triggering line of code

activitysim/activitysim/core/mem.py

Line 192 in 7ead422

info = current_process.memory_full_info()

and

activitysim/activitysim/core/mem.py

Line 257 in 7ead422

info = psutil.Process().memory_full_info()

Thoughts on resolution

After some googling, it seems to be a reporting issue with psutil.Process().memory_full_info(), when the memory value (in Bytes) exceeds a certain large number, this function crashes. I added a try and except condition to work around it in my runs.

[jpn--]

Duplicate of #570.

I guess my fix (the same solution as @i-am-sijia) got applied only to the sharrow branch I was working on, and was not ported to the PR with the memory monitor. Please accept my apologies.