Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: LSAN Detected Memory Leaks #54865

Open
3 tasks done
WillAyd opened this issue Aug 29, 2023 · 7 comments
Open
3 tasks done

BUG: LSAN Detected Memory Leaks #54865

WillAyd opened this issue Aug 29, 2023 · 7 comments
Labels
Bug IO JSON read_json, to_json, json_normalize

Comments

@WillAyd
Copy link
Member

WillAyd commented Aug 29, 2023 •
edited
Loading

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Invoke meson via pip install -ve . --no-build-isolation --config-settings=builddir="asan" --config-settings=setup-args="-Db_sanitize=address"

Also create an LSAN "suppression file" with these contents:

leak:unicode_decode_utf8
leak:PyObject_Malloc

Name that file and provide it via an absolute path to the below command:

LSAN_OPTIONS=suppressions=<absolute_path_to_suppression_file> LD_PRELOAD=$(gcc -print-file-name=libasan.so) python -m pytest pandas/tests/io/json/

Issue Description

You have to sift through a lot, but ASAN detects direct leaks in our codebase from at least the following locations (just running JSON test suite above):

Direct leak of 1440 byte(s) in 36 object(s) allocated from:
    #0 0x7f953ea95749 in __interceptor_calloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f94d4f543df in __pyx_pw_6pandas_5_libs_9hashtable_17PyObjectHashTable_1__init__ (/home/willayd/clones/pandas/asan/pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so+0x3543df)
    #2 0x55b4f55a1dba in type_call /usr/local/src/conda/python-3.10.12/Objects/typeobject.c:1135

Direct leak of 480 byte(s) in 12 object(s) allocated from:
    #0 0x7f953ea95749 in __interceptor_calloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f94d4f301ca in __pyx_tp_new_6pandas_5_libs_9hashtable_Int64HashTable (/home/willayd/clones/pandas/asan/pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so+0x3301ca)
    #2 0x55b4f55a1d32 in type_call /usr/local/src/conda/python-3.10.12/Objects/typeobject.c:1123
    #3 0x7f94cc8c3da2 in __pyx_f_6pandas_5_libs_5index_11Int64Engine__make_hash_table (/home/willayd/clones/pandas/asan/pandas/_libs/index.cpython-310-x86_64-linux-gnu.so+0xaada2)
    #4 0x7f94cc9262e3 in __pyx_f_6pandas_5_libs_5index_11IndexEngine__ensure_mapping_populated (/home/willayd/clones/pandas/asan/pandas/_libs/index.cpython-310-x86_64-linux-gnu.so+0x10d2e3)
    #5 0x7f94cc8cbfae in __pyx_f_6pandas_5_libs_5index_11IndexEngine__do_unique_check (/home/willayd/clones/pandas/asan/pandas/_libs/index.cpython-310-x86_64-linux-gnu.so+0xb2fae)
    #6 0x7f94cc8c5973 in __pyx_getprop_6pandas_5_libs_5index_11IndexEngine_is_unique (/home/willayd/clones/pandas/asan/pandas/_libs/index.cpython-310-x86_64-linux-gnu.so+0xac973)
    #7 0x55b4f55a6da9 in _PyObject_GenericGetAttrWithDict /usr/local/src/conda/python-3.10.12/Objects/object.c:1254

Direct leak of 240 byte(s) in 6 object(s) allocated from:
    #0 0x7f953ea95749 in __interceptor_calloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f94d4f7b65a in __pyx_tp_new_6pandas_5_libs_9hashtable_UInt64HashTable (/home/willayd/clones/pandas/asan/pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so+0x37b65a)
    #2 0x55b4f55a1d32 in type_call /usr/local/src/conda/python-3.10.12/Objects/typeobject.c:1123
    #3 0x7f94cc8a5442 in __pyx_f_6pandas_5_libs_5index_12UInt64Engine__make_hash_table (/home/willayd/clones/pandas/asan/pandas/_libs/index.cpython-310-x86_64-linux-gnu.so+0x8c442)
    #4 0x7f94cc9262e3 in __pyx_f_6pandas_5_libs_5index_11IndexEngine__ensure_mapping_populated (/home/willayd/clones/pandas/asan/pandas/_libs/index.cpython-310-x86_64-linux-gnu.so+0x10d2e3)
    #5 0x7f94cc8cbfae in __pyx_f_6pandas_5_libs_5index_11IndexEngine__do_unique_check (/home/willayd/clones/pandas/asan/pandas/_libs/index.cpython-310-x86_64-linux-gnu.so+0xb2fae)
    #6 0x7f94cc8c5973 in __pyx_getprop_6pandas_5_libs_5index_11IndexEngine_is_unique (/home/willayd/clones/pandas/asan/pandas/_libs/index.cpython-310-x86_64-linux-gnu.so+0xac973)
    #7 0x55b4f55a6da9 in _PyObject_GenericGetAttrWithDict /usr/local/src/conda/python-3.10.12/Objects/object.c:1254

Expected Behavior

No leaks

Installed Versions

INSTALLED VERSIONS

commit : 29873e4
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 6.2.0-26-generic
Version : #26-Ubuntu SMP PREEMPT_DYNAMIC Mon Jul 10 23:39:54 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.1.0rc0+254.g29873e442c
numpy : 1.22.4
pytz : 2023.3
dateutil : 2.8.2
setuptools : 68.1.2
pip : 23.2.1
Cython : 0.29.33
pytest : 7.4.0
hypothesis : 6.82.7
sphinx : 6.2.1
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : 4.9.3
html5lib : 1.1
pymysql : 1.4.6
psycopg2 : 2.9.6
jinja2 : 3.1.2
IPython : 8.14.0
pandas_datareader : None
bs4 : 4.12.2
bottleneck : 1.3.7
dataframe-api-compat: None
fastparquet : 2023.7.0
fsspec : 2023.6.0
gcsfs : 2023.6.0
matplotlib : 3.7.2
numba : 0.57.1
numexpr : 2.8.4
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : 1.2.3
pyxlsb : 1.0.10
s3fs : 2023.6.0
scipy : 1.11.2
sqlalchemy : 2.0.20
tables : 3.8.0
tabulate : 0.9.0
xarray : 2023.8.0
xlrd : 2.0.1
zstandard : 0.19.0
tzdata : 2023.3
qtpy : None
pyqt5 : None
None
@WillAyd WillAyd added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Aug 29, 2023
@WillAyd
Copy link
Member Author

WillAyd commented Aug 29, 2023

progress towards #52990

@lithomas1
Copy link
Member

Does this work with clang?

Your bitmask PR reminded me to try out asan again, but no leaks seem to be reported (on macos, I don't think asan is loaded in correctly).

@lithomas1 lithomas1 added IO JSON read_json, to_json, json_normalize and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Aug 29, 2023
@WillAyd
Copy link
Member Author

WillAyd commented Aug 29, 2023

It should. At the very least you should be getting a lot of (false?) positives from Python. Do you get any output at all?

@WillAyd
Copy link
Member Author

WillAyd commented Aug 29, 2023

Here are more detailed tracebacks from the leaks above with optimization level 0

Direct leak of 1440 byte(s) in 36 object(s) allocated from:
    #0 0x7f561de95749 in __interceptor_calloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f55b406e882 in traced_calloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:41
    #2 0x7f55b4084d7d in kh_init_pymap ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:393
    #3 0x7f55b41fccc0 in __pyx_pf_6pandas_5_libs_9hashtable_17PyObjectHashTable___init__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:97063
    #4 0x7f55b41fcc0d in __pyx_pw_6pandas_5_libs_9hashtable_17PyObjectHashTable_1__init__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:97034
    #5 0x55b22d97fdba in type_call /usr/local/src/conda/python-3.10.12/Objects/typeobject.c:1135

Direct leak of 480 byte(s) in 12 object(s) allocated from:
    #0 0x7f561de95749 in __interceptor_calloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f55b406e882 in traced_calloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:41
    #2 0x7f55b4073374 in kh_init_int64 ../pandas/_libs/include/pandas/vendored/klib/khash.h:688
    #3 0x7f55b40fd9c0 in __pyx_pf_6pandas_5_libs_9hashtable_14Int64HashTable___cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:40064
    #4 0x7f55b40fd90a in __pyx_pw_6pandas_5_libs_9hashtable_14Int64HashTable_1__cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:40035
    #5 0x7f55b435d993 in __pyx_tp_new_6pandas_5_libs_9hashtable_Int64HashTable pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:161069
    #6 0x55b22d97fd32 in type_call /usr/local/src/conda/python-3.10.12/Objects/typeobject.c:1123
    #7 0x7f55abd5eeae in __Pyx__PyObject_CallOneArg pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:85628
    #8 0x7f55abd5ef97 in __Pyx_PyObject_CallOneArg pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:85647
    #9 0x7f55abc81ca7 in __pyx_f_6pandas_5_libs_5index_11Int64Engine__make_hash_table pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:22390
    #10 0x7f55abc47115 in __pyx_f_6pandas_5_libs_5index_11IndexEngine__ensure_mapping_populated pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:9667
    #11 0x7f55abc445eb in __pyx_f_6pandas_5_libs_5index_11IndexEngine__do_unique_check pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:8788
    #12 0x7f55abc4443b in __pyx_pf_6pandas_5_libs_5index_11IndexEngine_9is_unique___get__ pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:8718
    #13 0x7f55abc44345 in __pyx_pw_6pandas_5_libs_5index_11IndexEngine_9is_unique_1__get__ pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:8684
    #14 0x7f55abd4533d in __pyx_getprop_6pandas_5_libs_5index_11IndexEngine_is_unique pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:77428
    #15 0x55b22d984da9 in _PyObject_GenericGetAttrWithDict /usr/local/src/conda/python-3.10.12/Objects/object.c:1254

Direct leak of 240 byte(s) in 6 object(s) allocated from:
    #0 0x7f561de95749 in __interceptor_calloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f55b406e882 in traced_calloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:41
    #2 0x7f55b4074af5 in kh_init_uint64 ../pandas/_libs/include/pandas/vendored/klib/khash.h:689
    #3 0x7f55b40e433b in __pyx_pf_6pandas_5_libs_9hashtable_15UInt64HashTable___cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:34277
    #4 0x7f55b40e4285 in __pyx_pw_6pandas_5_libs_9hashtable_15UInt64HashTable_1__cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:34248
    #5 0x7f55b435d577 in __pyx_tp_new_6pandas_5_libs_9hashtable_UInt64HashTable pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:160929
    #6 0x55b22d97fd32 in type_call /usr/local/src/conda/python-3.10.12/Objects/typeobject.c:1123
    #7 0x7f55abd5eeae in __Pyx__PyObject_CallOneArg pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:85628
    #8 0x7f55abd5ef97 in __Pyx_PyObject_CallOneArg pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:85647
    #9 0x7f55abc92b77 in __pyx_f_6pandas_5_libs_5index_12UInt64Engine__make_hash_table pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:27630
    #10 0x7f55abc47115 in __pyx_f_6pandas_5_libs_5index_11IndexEngine__ensure_mapping_populated pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:9667
    #11 0x7f55abc445eb in __pyx_f_6pandas_5_libs_5index_11IndexEngine__do_unique_check pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:8788
    #12 0x7f55abc4443b in __pyx_pf_6pandas_5_libs_5index_11IndexEngine_9is_unique___get__ pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:8718
    #13 0x7f55abc44345 in __pyx_pw_6pandas_5_libs_5index_11IndexEngine_9is_unique_1__get__ pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:8684
    #14 0x7f55abd4533d in __pyx_getprop_6pandas_5_libs_5index_11IndexEngine_is_unique pandas/_libs/index.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/index.pyx.c:77428
    #15 0x55b22d984da9 in _PyObject_GenericGetAttrWithDict /usr/local/src/conda/python-3.10.12/Objects/object.c:1254

@WillAyd
Copy link
Member Author

WillAyd commented Aug 31, 2023

Interestingly enough I couldn't get valgrind to detect these. I have a hunch that LSAN may be misreporting bytes that Cython releases in any __dealloc__ calls when they are constructed via pytest parameters. Haven't verified that is the case for these in particular, but something worth further investigation

@WillAyd
Copy link
Member Author

WillAyd commented Dec 5, 2023

As far as a suppression file goes I am having luck with:

leak:obmalloc.c
leak:stringio.c
leak:pyo3
leak:crypto
leak:unicodeobject.c
leak:bytesobject.c
leak:listobject.c
leak:gcmodule.c
leak:PyThread
leak:arrow
leak:numpy

I was somewhat expecting this to be so wide of a net cast of suppressions that it wouldn't yield any results, but looks like there are still plenty:

$ LSAN_OPTIONS=suppressions=lsan_suppr.txt LD_PRELOAD=$(gcc -print-file-name=libasan.so) python -m pytest pandas/tests/io/json/test_pandas.py 
==122102==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 600 byte(s) in 15 object(s) allocated from:
    #0 0x7f9723e95749 in __interceptor_calloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f96b7e55254 in traced_calloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:37
    #2 0x7f96b7e8789c in kh_init_pymap ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:356
    #3 0x7f96b81246a9 in __pyx_pf_6pandas_5_libs_9hashtable_17PyObjectHashTable___init__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:130346
    #4 0x7f96b8124510 in __pyx_pw_6pandas_5_libs_9hashtable_17PyObjectHashTable_1__init__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:130311
    #5 0x5586c60a39ea in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1135
    #6 0x5586c60a39ea in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

Direct leak of 520 byte(s) in 1 object(s) allocated from:
    #0 0x7f9723e94dd2 in __interceptor_realloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f96ba57a129 in parser_trim_buffers ../pandas/_libs/src/parser/tokenizer.c:1211
    #2 0x7f96a4a56606 in __pyx_pf_6pandas_5_libs_7parsers_10TextReader_12read_low_memory pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:29031
    #3 0x7f96a4a5364c in __pyx_pw_6pandas_5_libs_7parsers_10TextReader_13read_low_memory pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:28559
    #4 0x7f96a4b6bca9 in __Pyx_CyFunction_Vectorcall_FASTCALL_KEYWORDS pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:57975
    #5 0x5586c609aa3b in _PyObject_VectorcallTstate /usr/local/src/conda/python-3.10.13/Include/cpython/abstract.h:114
    #6 0x5586c609aa3b in PyObject_Vectorcall /usr/local/src/conda/python-3.10.13/Include/cpython/abstract.h:123
    #7 0x5586c609aa3b in call_function /usr/local/src/conda/python-3.10.13/Python/ceval.c:5893
    #8 0x5586c609aa3b in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.10.13/Python/ceval.c:4198

Direct leak of 520 byte(s) in 1 object(s) allocated from:
    #0 0x7f9723e94dd2 in __interceptor_realloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f96ba579faa in parser_trim_buffers ../pandas/_libs/src/parser/tokenizer.c:1207
    #2 0x7f96a4a56606 in __pyx_pf_6pandas_5_libs_7parsers_10TextReader_12read_low_memory pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:29031
    #3 0x7f96a4a5364c in __pyx_pw_6pandas_5_libs_7parsers_10TextReader_13read_low_memory pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:28559
    #4 0x7f96a4b6bca9 in __Pyx_CyFunction_Vectorcall_FASTCALL_KEYWORDS pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:57975
    #5 0x5586c609aa3b in _PyObject_VectorcallTstate /usr/local/src/conda/python-3.10.13/Include/cpython/abstract.h:114
    #6 0x5586c609aa3b in PyObject_Vectorcall /usr/local/src/conda/python-3.10.13/Include/cpython/abstract.h:123
    #7 0x5586c609aa3b in call_function /usr/local/src/conda/python-3.10.13/Python/ceval.c:5893
    #8 0x5586c609aa3b in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.10.13/Python/ceval.c:4198

Direct leak of 160 byte(s) in 4 object(s) allocated from:
    #0 0x7f9723e95749 in __interceptor_calloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f96b7e55254 in traced_calloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:37
    #2 0x7f96b7e6356d in kh_init_uint64 ../pandas/_libs/include/pandas/vendored/klib/khash.h:733
    #3 0x7f96b7f5fb14 in __pyx_pf_6pandas_5_libs_9hashtable_15UInt64HashTable___cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:59122
    #4 0x7f96b7f5f98c in __pyx_pw_6pandas_5_libs_9hashtable_15UInt64HashTable_1__cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:59087
    #5 0x7f96b8333020 in __pyx_tp_new_6pandas_5_libs_9hashtable_UInt64HashTable pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:187181
    #6 0x5586c60a3908 in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1123
    #7 0x5586c60a3908 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

Direct leak of 80 byte(s) in 2 object(s) allocated from:
    #0 0x7f9723e95749 in __interceptor_calloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f96b7e55254 in traced_calloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:37
    #2 0x7f96b7e5fe2c in kh_init_int64 ../pandas/_libs/include/pandas/vendored/klib/khash.h:732
    #3 0x7f96b7f88e92 in __pyx_pf_6pandas_5_libs_9hashtable_14Int64HashTable___cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:65682
    #4 0x7f96b7f88d0a in __pyx_pw_6pandas_5_libs_9hashtable_14Int64HashTable_1__cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:65647
    #5 0x7f96b8333683 in __pyx_tp_new_6pandas_5_libs_9hashtable_Int64HashTable pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:187347
    #6 0x5586c60a3908 in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1123
    #7 0x5586c60a3908 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f9723e94dd2 in __interceptor_realloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f96ba57abe0 in parser_trim_buffers ../pandas/_libs/src/parser/tokenizer.c:1252
    #2 0x7f96a4a56606 in __pyx_pf_6pandas_5_libs_7parsers_10TextReader_12read_low_memory pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:29031
    #3 0x7f96a4a5364c in __pyx_pw_6pandas_5_libs_7parsers_10TextReader_13read_low_memory pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:28559
    #4 0x7f96a4b6bca9 in __Pyx_CyFunction_Vectorcall_FASTCALL_KEYWORDS pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:57975
    #5 0x5586c609aa3b in _PyObject_VectorcallTstate /usr/local/src/conda/python-3.10.13/Include/cpython/abstract.h:114
    #6 0x5586c609aa3b in PyObject_Vectorcall /usr/local/src/conda/python-3.10.13/Include/cpython/abstract.h:123
    #7 0x5586c609aa3b in call_function /usr/local/src/conda/python-3.10.13/Python/ceval.c:5893
    #8 0x5586c609aa3b in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.10.13/Python/ceval.c:4198

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f9723e94dd2 in __interceptor_realloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f96ba57ad21 in parser_trim_buffers ../pandas/_libs/src/parser/tokenizer.c:1258
    #2 0x7f96a4a56606 in __pyx_pf_6pandas_5_libs_7parsers_10TextReader_12read_low_memory pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:29031
    #3 0x7f96a4a5364c in __pyx_pw_6pandas_5_libs_7parsers_10TextReader_13read_low_memory pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:28559
    #4 0x7f96a4b6bca9 in __Pyx_CyFunction_Vectorcall_FASTCALL_KEYWORDS pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:57975
    #5 0x5586c609aa3b in _PyObject_VectorcallTstate /usr/local/src/conda/python-3.10.13/Include/cpython/abstract.h:114
    #6 0x5586c609aa3b in PyObject_Vectorcall /usr/local/src/conda/python-3.10.13/Include/cpython/abstract.h:123
    #7 0x5586c609aa3b in call_function /usr/local/src/conda/python-3.10.13/Python/ceval.c:5893
    #8 0x5586c609aa3b in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.10.13/Python/ceval.c:4198

Indirect leak of 480 byte(s) in 15 object(s) allocated from:
    #0 0x7f9723e94dd2 in __interceptor_realloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f96b7e552a9 in traced_realloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:45
    #2 0x7f96b7e883ea in kh_resize_pymap ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:356
    #3 0x7f96b8124c05 in __pyx_pf_6pandas_5_libs_9hashtable_17PyObjectHashTable___init__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:130385
    #4 0x7f96b8124510 in __pyx_pw_6pandas_5_libs_9hashtable_17PyObjectHashTable_1__init__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:130311
    #5 0x5586c60a39ea in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1135
    #6 0x5586c60a39ea in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

Indirect leak of 480 byte(s) in 15 object(s) allocated from:
    #0 0x7f9723e94dd2 in __interceptor_realloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f96b7e552a9 in traced_realloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:45
    #2 0x7f96b7e882d2 in kh_resize_pymap ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:356
    #3 0x7f96b8124c05 in __pyx_pf_6pandas_5_libs_9hashtable_17PyObjectHashTable___init__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:130385
    #4 0x7f96b8124510 in __pyx_pw_6pandas_5_libs_9hashtable_17PyObjectHashTable_1__init__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:130311
    #5 0x5586c60a39ea in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1135
    #6 0x5586c60a39ea in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

Indirect leak of 257 byte(s) in 1 object(s) allocated from:
    #0 0x7f9723e94dd2 in __interceptor_realloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f96ba57a437 in parser_trim_buffers ../pandas/_libs/src/parser/tokenizer.c:1226
    #2 0x7f96a4a56606 in __pyx_pf_6pandas_5_libs_7parsers_10TextReader_12read_low_memory pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:29031
    #3 0x7f96a4a5364c in __pyx_pw_6pandas_5_libs_7parsers_10TextReader_13read_low_memory pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:28559
    #4 0x7f96a4b6bca9 in __Pyx_CyFunction_Vectorcall_FASTCALL_KEYWORDS pandas/_libs/parsers.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/parsers.pyx.c:57975
    #5 0x5586c609aa3b in _PyObject_VectorcallTstate /usr/local/src/conda/python-3.10.13/Include/cpython/abstract.h:114
    #6 0x5586c609aa3b in PyObject_Vectorcall /usr/local/src/conda/python-3.10.13/Include/cpython/abstract.h:123
    #7 0x5586c609aa3b in call_function /usr/local/src/conda/python-3.10.13/Python/ceval.c:5893
    #8 0x5586c609aa3b in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.10.13/Python/ceval.c:4198

Indirect leak of 128 byte(s) in 4 object(s) allocated from:
    #0 0x7f9723e94dd2 in __interceptor_realloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f96b7e552a9 in traced_realloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:45
    #2 0x7f96b7e640ae in kh_resize_uint64 ../pandas/_libs/include/pandas/vendored/klib/khash.h:733
    #3 0x7f96b7f60070 in __pyx_pf_6pandas_5_libs_9hashtable_15UInt64HashTable___cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:59161
    #4 0x7f96b7f5f98c in __pyx_pw_6pandas_5_libs_9hashtable_15UInt64HashTable_1__cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:59087
    #5 0x7f96b8333020 in __pyx_tp_new_6pandas_5_libs_9hashtable_UInt64HashTable pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:187181
    #6 0x5586c60a3908 in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1123
    #7 0x5586c60a3908 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

Indirect leak of 128 byte(s) in 4 object(s) allocated from:
    #0 0x7f9723e94dd2 in __interceptor_realloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f96b7e552a9 in traced_realloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:45
    #2 0x7f96b7e63f96 in kh_resize_uint64 ../pandas/_libs/include/pandas/vendored/klib/khash.h:733
    #3 0x7f96b7f60070 in __pyx_pf_6pandas_5_libs_9hashtable_15UInt64HashTable___cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:59161
    #4 0x7f96b7f5f98c in __pyx_pw_6pandas_5_libs_9hashtable_15UInt64HashTable_1__cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:59087
    #5 0x7f96b8333020 in __pyx_tp_new_6pandas_5_libs_9hashtable_UInt64HashTable pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:187181
    #6 0x5586c60a3908 in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1123
    #7 0x5586c60a3908 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

Indirect leak of 64 byte(s) in 2 object(s) allocated from:
    #0 0x7f9723e94dd2 in __interceptor_realloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f96b7e552a9 in traced_realloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:45
    #2 0x7f96b7e6096d in kh_resize_int64 ../pandas/_libs/include/pandas/vendored/klib/khash.h:732
    #3 0x7f96b7f893ee in __pyx_pf_6pandas_5_libs_9hashtable_14Int64HashTable___cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:65721
    #4 0x7f96b7f88d0a in __pyx_pw_6pandas_5_libs_9hashtable_14Int64HashTable_1__cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:65647
    #5 0x7f96b8333683 in __pyx_tp_new_6pandas_5_libs_9hashtable_Int64HashTable pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:187347
    #6 0x5586c60a3908 in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1123
    #7 0x5586c60a3908 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

Indirect leak of 64 byte(s) in 2 object(s) allocated from:
    #0 0x7f9723e94dd2 in __interceptor_realloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f96b7e552a9 in traced_realloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:45
    #2 0x7f96b7e60855 in kh_resize_int64 ../pandas/_libs/include/pandas/vendored/klib/khash.h:732
    #3 0x7f96b7f893ee in __pyx_pf_6pandas_5_libs_9hashtable_14Int64HashTable___cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:65721
    #4 0x7f96b7f88d0a in __pyx_pw_6pandas_5_libs_9hashtable_14Int64HashTable_1__cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:65647
    #5 0x7f96b8333683 in __pyx_tp_new_6pandas_5_libs_9hashtable_Int64HashTable pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:187347
    #6 0x5586c60a3908 in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1123
    #7 0x5586c60a3908 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

Indirect leak of 60 byte(s) in 15 object(s) allocated from:
    #0 0x7f9723e955f8 in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7f96b7e55205 in traced_malloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:29
    #2 0x7f96b7e88187 in kh_resize_pymap ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:356
    #3 0x7f96b8124c05 in __pyx_pf_6pandas_5_libs_9hashtable_17PyObjectHashTable___init__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:130385
    #4 0x7f96b8124510 in __pyx_pw_6pandas_5_libs_9hashtable_17PyObjectHashTable_1__init__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:130311
    #5 0x5586c60a39ea in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1135
    #6 0x5586c60a39ea in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

Indirect leak of 16 byte(s) in 4 object(s) allocated from:
    #0 0x7f9723e955f8 in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7f96b7e55205 in traced_malloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:29
    #2 0x7f96b7e63e4b in kh_resize_uint64 ../pandas/_libs/include/pandas/vendored/klib/khash.h:733
    #3 0x7f96b7f60070 in __pyx_pf_6pandas_5_libs_9hashtable_15UInt64HashTable___cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:59161
    #4 0x7f96b7f5f98c in __pyx_pw_6pandas_5_libs_9hashtable_15UInt64HashTable_1__cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:59087
    #5 0x7f96b8333020 in __pyx_tp_new_6pandas_5_libs_9hashtable_UInt64HashTable pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:187181
    #6 0x5586c60a3908 in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1123
    #7 0x5586c60a3908 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

Indirect leak of 8 byte(s) in 2 object(s) allocated from:
    #0 0x7f9723e955f8 in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7f96b7e55205 in traced_malloc ../pandas/_libs/include/pandas/vendored/klib/khash_python.h:29
    #2 0x7f96b7e6070a in kh_resize_int64 ../pandas/_libs/include/pandas/vendored/klib/khash.h:732
    #3 0x7f96b7f893ee in __pyx_pf_6pandas_5_libs_9hashtable_14Int64HashTable___cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:65721
    #4 0x7f96b7f88d0a in __pyx_pw_6pandas_5_libs_9hashtable_14Int64HashTable_1__cinit__ pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:65647
    #5 0x7f96b8333683 in __pyx_tp_new_6pandas_5_libs_9hashtable_Int64HashTable pandas/_libs/hashtable.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/hashtable.pyx.c:187347
    #6 0x5586c60a3908 in type_call /usr/local/src/conda/python-3.10.13/Objects/typeobject.c:1123
    #7 0x5586c60a3908 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.10.13/Objects/call.c:215

-----------------------------------------------------
Suppressions used:
  count      bytes template
   7213   13128616 obmalloc.c
      2         16 stringio.c
      3         96 pyo3
    182      10066 crypto
    247     676487 unicodeobject.c
    386     179166 bytesobject.c
      9      28224 listobject.c
     37      26920 gcmodule.c
     25        800 PyThread
    957     117662 arrow
    286       5103 numpy
-----------------------------------------------------

SUMMARY: AddressSanitizer: 3597 byte(s) leaked in 89 allocation(s).

@WillAyd WillAyd mentioned this issue Dec 19, 2023
Merged
@WillAyd
Copy link
Member Author

WillAyd commented Dec 19, 2023

FWIW some of these leaks I could only get when running under pytest. Not sure if it is a pytest/pytest-xdist thing

@WillAyd WillAyd mentioned this issue Jan 26, 2024
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug IO JSON read_json, to_json, json_normalize
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@WillAyd @lithomas1