-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Zarr support in NetCDF-Fortran for "cloud-native" model simulations? #209
Comments
We are hoping to have a version out in the next month or two, so the Feb-March timeframe is perfectly reasonable! |
Just to check -- is it possible to get a testing version this month? |
In fortran no. In C maybe. But we still need an S3 driver. We are currently |
I take that back. Once the C version is working, it should also work with any |
@DennisHeimbigner and @WardF, do you think it would be possible to write Zarr from FORTRAN using the new 4.8.0 NetCDF C library with this approach @ocefpaf pointed me toward: |
It should be possible assuming that the nf_open path can take |
Cool! Which intern was it? It would be nice to find out what they discovered. |
@DennisHeimbigner pingity ping ping |
I just built netcdf-c-4.8.0 with netcdf-fortran-4.5.3, also using MPI for parallelIO. All tests passed. I had to use: The fortran library just hands the path over to the C library, so Zarr stuff should work transparently to Fortran, just as DAP does. |
@edhartnett , you had to use "-g"? So not ready for prime time (e.g. "-O3" yet)? What I'd like to do is write Zarr from our ocean modeling simulations that would look exactly like what xarray produces... |
No, the -g and -Wall are not what I meant. I had to
use -fallow-argument-mismatch.
Absolutely this is ready for prime-time. ;-)
…On Wed, May 5, 2021 at 5:29 AM Rich Signell ***@***.***> wrote:
@edhartnett <https://github.com/edhartnett> , you had to use "-g"? So not
ready for prime time (e.g. "-O3" yet)?
What I'd like to do is write Zarr from our ocean modeling simulations that
would look exactly like what xarray produces...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#209 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABCSXXG5RE72VBYZGU3GEK3TMETX3ANCNFSM4KFS26AQ>
.
|
@edhartnett, do you have a sample fortran program that creates a zarr dataset you could share? |
No, sorry. I haven't tried Zarr. |
@edhartnett, Ah bummer. But it should now be possible for me to do that, right? |
Take any simple Fortran program that creates a simple netcdf4 dataset. |
@DennisHeimbigner, okay, I'll try that! And |
Depends. If you use the github master, then yes, mode=xarray,file should produce |
First thanks for all the great work on NetCDF!
I have a research project that will significantly benefit from NetCDF-Zarr. I recently saw a tweet from @jhamman that "pre-alpha will be available early in 2020". I also notice some Zarr-related updates like Unidata/netcdf-c#1259. I am excited to test the new Zarr capability with real models and give feedbacks. Is it possible to get a preliminary version to play with around Feb-March? Or is it still too early to say?
More details about the use case: My workflow involves running Fortran-based models in a cloud-native container environment, for example AWS Batch or Kubernetes cluster. The main benefit is to scale out ensemble runs quickly via AWS Batch Array Jobs or Kubernetes Parallel Jobs . Similar to what Pangeo does, but here for Fortran models instead of Dask workers. However I/O is a major pain in a container environment (need to deal with Persistent Volumes for example). It is actually possible to mount a Lustre to Kubernetes, but the workflow will be much, much simpler if the model can directly read/write with S3.
The text was updated successfully, but these errors were encountered: