Flask - PyWSGI and concurrent access to dataset (netcdf)

[kgiannakelos]

Hello,

i have a NetCDF file which i would like to retrieve data from using a REST API. I have a python with flask restful with gevent pyWSGI server and applied monkey patching to allow concurrent requests on the same REST endpoint.

Each time a single request is executed, the NetCDF is opened as xarray dataset and data get extracted. But unfortunately when multiple concurrent requests are executed, xarray seems to handle them in sequential manner which i suspect occurs because it uses all available threads and flask-pywsgi cannot use another thread to serve the next request until the previous is finished.

Is there any workaround or a recommended practice to have multiple concurrent requests access the same netcdf dataset without having blocking behavior?

Thank you!

[shoyer]

Do you know if your data is stored in netCDF3 or netCDF4 files?

If you're using netCDF3, the SciPy netCDF reader engine='scipy is threadsafe and can be used in parallel.

If you're using netCDF4, unfortunately the underlying HDF5 library is not threadsafe. In this case, there is no avoiding the use of a global locks for access within a single process. If you need to paralelize access to netCDF4 files, you'll need to make use of multiple processes.

[kgiannakelos]

Thanks for your answer. I create the netcdf using xarray so i can change to NetCDF3_64BIT (it's 35 gb file) and see how it goes.

Maybe i could create some processes on the controller and have user do some rest polling to get the result.

[benbovy]

Although it might not answer your specific question, have you had a chance to look at https://github.com/xarray-contrib/xpublish ? It seems a good fit for what you would like to do.

[kgiannakelos]

@benbovy I already came across this library but i cannot say it fits my needs since i may need some preprocessing based on the user's parameters and cannot except users to have this library to access the dataset. Besides, isn't there gonna be the same issue due to non-threadsafe access?

Anyways thank your reply i will try these two methods and decide what is the best solution.

I will close this after i reach to any conclusions.

[benbovy]

Note that xpublish is designed to be fully customizable so you can restrict access to the data and instead only provide custom API endpoints with the parameters and preprocessing that you need. See https://xpublish.readthedocs.io/en/latest/tutorial.html#custom-api-routes.

Regarding thread-safety, I don't know what is your application but since you've created the data using Xarray maybe there are alternative options to netcdf files that could better suit your needs? (e.g., based on zarr, xpublish's application cache, etc.)

[kgiannakelos]

Oh i only thought the endpoints were hardcoded! That's nice to know.

I am not very familiar with zarr but i will definitely check it out.

Thanks a lot!