Best way to write zone ordered data in a collective way - Fortran

[artu72]

Hi everyone,
I hope that I can ask for some hint here since I am new in this topic. I would like to implement the most effective way (collective) to write down a set of ordered data in a MPI Fortran CFD code. In each MPI process, I have NZ zones, and in each of them there are there are some 3d variables in a structure
VAR(Z)%R(NI(Z),NJ(Z),NK(Z))
where R is a variable (with some other with the same dimension) and NI(Z), NJ(Z),NK(Z) are the number of cells in dimensions 1,2,3.
Thank you in advance.

[wkliao]

There are a few F90 example programs under folder examples/F90.
The closest one to your case is put_var.f90.

[artu72]

Thank for quick reply. The example you suggested me concerns a distributed array, and processes write down different slices to disk. In my cas I have independent arrays on each process.

[wkliao]

There are two ways to do this.

1. Each process creates it own file and then defines and writes its variables
   to its own file. This will give you P files, if you ran on P processes.
2. If one single file is preferred, then you must collectively define all variables first.
   This requires all processes to define the variables using the same parameters,
   i.e. names, dimensions, data types, and attributes. Once all definitions are done, call
   nonblocking APIs (e.g. nf90mpi_iput_var()) to write the data and at the end
   call nf90mpi_wait_all() to flush all write data to the file.

For nonblocking APIs, see example nonblocking_write.f90

[artu72]

Since I prefer writing a single file, I will consider second option. I assume that each process write in a separate way own data in a totally independent fashion sicne in the definition pahse relative file offsets will be set. I will keep you updated on my progrss and ask you for some help if necessary. Thank again.

[artu72]

Hi, I have written a simple code to test non blocking write suited to my case. I have defined two double precision 3d arrays r and p, and in each rank they have a independent size and values therein. So, in the resulting netcdf file i should have nprocs*2 total variables. The code is not working, surely there is some bug in the variable definition and dimensioning, and I am puzzled in writing the variables (nf90mpi_iput_var) since each rank should write only the owned variables. Some hint here will then be welcomed!

nbw_test.txt

[wkliao]

A few suggestions.

• Since there are (NUM_VARS*nprocs) variables, the size of varids should be of the same size.
• When each process writes its own variables, you will want to add an if statement inside of loop NUM_VARS
  that checks if the varids(i) is owned by the writing process.
• Write buffers r and p should not be reused for different calls to nf90mpi_iput_var.

[artu72]

I have implemented your suggestions. I attach here the updated code, that is still giving a "In nf90mpi_wait_allInvalid nonblocking request ID." issue. Please note that each rank will write only two buffers (by setting i=rank*2+1 and i=rank*2+2)I do not understand third one: are you meaning that in each rank I should use once r and p buffers, right?
nbw_test.txt

[wkliao]

The number of requests should be 2 per process (i.e. NUM_VARS).

% diff nbw_test.txt nbw_test.txt.1
45,46c45,46
<           allocate(req(NUM_VARS*nprocs))
<           allocate(st(NUM_VARS*nprocs))
---
>           allocate(req(NUM_VARS))
>           allocate(st(NUM_VARS))
59c59
<           counts = ni*nj*nk
---
>           counts = (/ ni,nj,nk /)
102c102
<           err = nf90mpi_iput_var(ncid, varids(i), r, req(i) ) !, starts, counts)
---
>           err = nf90mpi_iput_var(ncid, varids(i), r, req(1) ) !, starts, counts)
106c106
<           err = nf90mpi_iput_var(ncid, varids(i), p, req(i) ) !, starts, counts)
---
>           err = nf90mpi_iput_var(ncid, varids(i), p, req(2) ) !, starts, counts)
111c111
<           err = nf90mpi_wait_all(ncid, NUM_VARS*nprocs, req, st)
---
>           err = nf90mpi_wait_all(ncid, NUM_VARS, req, st)
115c115
<           do i=1, NUM_VARS*nprocs
---
>           do i=1, NUM_VARS

[artu72]

Now the code is about ok. It is running but it does not give correct results. For instance, running it on two processes produces four variables in the nbwt.nc file. Var 1 and 2 are ok, but 3 and 4 ones are instead 0. I attach here the updated code. Thank you in advance.
nbw_test.txt

[wkliao]

I made some changes. It assumes the variable's dimension sizes are different among processes.
nbw_test_v2.txt

[artu72]

I made some changes. It assumes the variable's dimension sizes are different among processes. nbw_test_v2.txt

Thank you very much, I think that is definitive answer! I will update you after implementing in CFD code.

[artu72]

Hi, the code now works. but I have a size problem. I explain you in some detail. In the code, I have implemented the nf90mpi_iput_var in the following way:
err = nf90mpi_iput_var(ncid, varids(i), r(1:ni,1:nj,1:nk), req(i) )
since the r array contains ghost cell that do not need to be written out. (please note that starts and count optional variables do not work as expected to select out a subarray to be written).
Now, in the case ni=nj=nk=8 (total dimension of 512), the run is ok. Instead, in the case ni=nj=nk=16 (total size 4096), the file is always written properly, but the buffer r is corrupted after the write operation, and it is filled with random values.
Some hint?

[wkliao]

Arguments start and count are describing the subarray of a variable in the file, not for the write buffers.
For using buffers contains ghost cells, see examples/F90/flexible_api.f90

When using nonblocking APIs, one should not touch the buffers before wait_all API is returned.

[artu72]

Ok. With this, I have concluded pnetcdf implementation in the code. You support has been determinant, thank you again.