Segmentation fault 139 (SIGSEGV)

[lucadix]

What happened?

While opening a set of netCDF files in a for loop, using xr.open_dataset().load(), I get a segmentation error (nr. 139). Please see code example below:

for region in region_list:
   [some code to read data associated to each region...]

   region_pred = xr.open_dataset(io.BytesIO(data)).load()

   [other code working on region_pred...]

The error is shown in Linux/Mac after running my Python code, whereas Windows seems to be masking it. I was able to catch that on Windows by launching my code as:

python3 my_code.py && echo ok || echo KO

In this way, KO gets printed and the segmentation fault is now noticeable.
I managed to fix the issue by using a second variable (called reg_pred) in addition to region_pred:

for region in region_list:
   [some code to read data associated to each region...]

   region_pred = xr.open_dataset(io.BytesIO(data))
   reg_pred = region_pred.load()

   [other code working on reg_pred...]

What did you expect to happen?

I don't know if the issue I described is something that the developers made on purpose. Personally, I think it is an issue and that's why I am reporting it. If it is not an issue, I would like to get a clarification in order to understand what am I missing.
Thank you in advance.

Minimal Complete Verifiable Example

for region in region_list:
   with storage_client.open(region, "rb") as f:
      data = f.read()
   region_pred = xr.open_dataset(io.BytesIO(data)).load()

   # some code working on region_pred to compute weather indices...

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS ------------------ commit: None python: 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 141 Stepping 1, GenuineIntel byteorder: little LC_ALL: None LANG: it_IT.UTF-8 LOCALE: ('Italian_Italy', '1252') libhdf5: 1.14.0 libnetcdf: 4.9.2

xarray: 2023.8.0
pandas: 2.1.0
numpy: 1.26.0
scipy: 1.11.2
netCDF4: 1.6.4
pydap: None
h5netcdf: 1.2.0
h5py: 3.9.0
Nio: None
zarr: None
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
iris: None
bottleneck: None
dask: 2023.10.0
distributed: 2023.10.0
matplotlib: 3.8.0
cartopy: None
seaborn: None
numbagg: None
fsspec: 2023.9.1
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 68.2.2
pip: 23.2.1
conda: None
pytest: None
mypy: None
IPython: 8.15.0
sphinx: None

[welcome]

Thanks for opening your first issue here at xarray! Be sure to follow the issue template!
If you have an idea for a solution, we would really welcome a Pull Request with proposed changes.
See the Contributing Guide for more.
It may take us a while to respond here, but we really value your contribution. Contributors like you help make xarray better.
Thank you!

[keewis]

This is definitely not intentional, segmentation faults always mean there's a bug somewhere.

See #7079, #7549, #7879, and #8289 for related issues, and in particular the comments in #8289 might point us in the right direction to finally fix this. As a workaround, you might be able to pin either netcdf4 to <=1.6.1 (or libnetcdf to 4.8.1, not sure if this is necessary).

[max-sixty]

I appreciate these are quite difficult to debug.

It is quite unlikely to be a problem with xarray itself, since xarray only python code — even though xarray is wrapping other libraries.

It's worth looking through the other issues and seeing whether those are similar.

Unless there's an MCVE of something that's not in those issues, we don't generally leave issues open, so in that case we can close this in time.

[max-sixty]

Closing but feel free to reopen with an MCVE