HDF5 error

[milankl]

With v0.10.3 on Julia 1.5.2 I'm currently producing this error

using NetCDF

x = Array{Float32}(1:10)
y = Array{Float32}(1:10)

xdim = NcDim("x",length(x),values=x)
ydim = NcDim("y",length(y),values=y)
tdim = NcDim("t",0,unlimited=true)

var = NcVar("u",[xdim,ydim,tdim],t=Float32)
tvar = NcVar("t",tdim,t=Int32)

nc = NetCDF.create("u.nc",[var,tvar],mode=NC_NETCDF4)
NetCDF.putatt(nc,"u",Dict("units"=>"m/s","long_name"=>"velocity"))

for i in 1:10
    NetCDF.putvar(nc,"u",rand(Float32,length(x),length(y)),start=[1,1,i],count=[-1,-1,1])
end

The file exists, and also the variable nc in the same repl looks fine

julia> nc

##### NetCDF File #####

u.nc

##### Dimensions #####

Name                                                Length                    
--------------------------------------------------------------------------------
t                                                   UNLIMITED (10 currently)  
x                                                   10                        
y                                                   10                        

##### Variables #####

Name                            Type            Dimensions                      
--------------------------------------------------------------------------------
t                               INT             t                               
x                               FLOAT           x                               
u                               FLOAT           x y t                           
y                               FLOAT           y                               

##### Attributes #####

Variable            Name                Value                                   
--------------------------------------------------------------------------------
u                   units               m/s                                     
u                   long_name           velocity   

However, if I try to NetCDF.open the file again I get

julia> u = NetCDF.open("u.nc")
ERROR: NetCDF error code -101:
	NetCDF: HDF error

And similarly with ncdump, which returns

milan@milanair ~ % ncdump -h u.nc
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 4371963328:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed
    major: Virtual File Layer
    minor: Can't update object
  #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 35, error message = 'Resource temporarily unavailable'
    major: File accessibilty
    minor: Bad file ID accessed
ncdump: u.nc: u.nc: NetCDF: HDF error

Can anyone reproduce this error?

[milankl]

Okay, I can't reproduce this error on a server I'm using. Any idea, how this error could be caused?

[milankl]

Okay that's weird, I did a port upgrade outdated as I use macports for ncdump and ncview, this updated hdf5, and returned the following note

--->  Some of the ports you installed have notes:
  hdf5 has the following notes:
    Mac users may need to set the environment variable "HDF5_USE_FILE_LOCKING"
    to the five-character string "FALSE" when accessing network mounted files. 
    This is an application run-time setting, not a configure or build setting. 
    Otherwise errors such as "unable to open file" or "HDF5 error" may be 
    encountered.

but then without doing anything, it now works again. How can the hdf5 installation from macports interfere with NetCDF.jl?

[visr]

I can't reproduce on Windows either.

How can the hdf5 installation from macports interfere with NetCDF.jl?

This library still uses the old-style BinDeps solution, see build.jl, which will happily pick up libraries from your system if they satisfy the criteria. So I think that is what's going on. You should be able check the contents of deps.jl, to see what it picked up. Or you can do Pkg.build() to force it to re-run build.jl.

With BinaryBuilder.jl the general trend is to isolate the binaries used by Julia packages, to make it less dependent on what happens to be installed on the system, and provide a better cross platform experience. #123 makes that switch, so if you have issues with the build you could try out that branch. Unfortunately there are still some issues with those binaries, since HDF5 cannot be cross compiled so easily, leading to some mismatches.

[milankl]

Thanks Martijn for the answer. Simply doing a re-build as you describe doesn't work for me, however adding export HDF5_USE_FILE_LOCKING="FALSE" to my .zshrc as it says in the warning, does the job.

Good to hear that NetCDF is moving towards a separately compiled binaries, that really seems to be the way to go, as shown by this issue.

[visr]

Ok, glad that you found a way to make it work, and noting it here, such that others may be able to find it as well.

Since I don't think there is anything we can do here to make this better, I'll go ahead and close the issue.