(precompilation) cachefiles always considered stale in newly instantiated Docker container

[josefsachsconning]

I create a Docker image with some julia packages precompiled, but when I run a container from that image, the packages are always re-precompiled. I believe this is because the mtimes stored within the .ji files (and returned by Base.cache_dependencies()) contain fractional seconds, but the mtimes of the files in the container (essentially coming from a tar file) no not contain fractional seconds. Perhaps the comparison in stale_cachefile() should be slightly relaxed to eliminate this problem? Any other suggestions?

[malmaud]

Can confirm I've also been hit by this issue. I'll take a look today at resolving it if someone doesn't beat me too it.

[StefanKarpinski]

I wonder if this is the dreaded timestamp issues that we've all been warned of...

[vtjnash]

for completely untracked loading, it's also an option to call ccall(:jl_restore_incremental, Any, (Ptr{UInt8},), "filename.ji") directly

[ScottPJones]

It's not the problems I'd brought up, that I've run into before, of clock skew between machines, and/or having only second resolution timestamps, and having machine generated code being produced and compiled under a second and the change being missed.
The proposed fix would actually make that second problem much more likely.

[malmaud]

@josefsachsconning I can confirm your hypothesis is correct.

Example:

julia> t1 = mtime("/root/.julia/v0.4/Requests/src/Requests.jl")  # on freshly-created docker container from an image with pre-compiled Requests
1.444849347e9
julia> t2 = Base.cache_dependencies("/root/.julia/lib/v0.4/Requests.ji")[2][1][2]
1.4448493470268579e9
julia> abs(t1-t2)
0.026857852935791016

So now the question is what to do about it. The easiest solution, and the one I'm in favor of, is to simly replace https://github.com/JuliaLang/julia/blob/master/base/loading.jl#L511 with an approximate equality with resolution of ~0.03 seconds.

The other possibilities are not using mtimes for invalidating cache dependencies at all, which I think is basically out of the question in the near future, or having special logic for the Docker situation, which I fear will end up hacky and fragile.

[tkelman]

If docker's rounding of mtimes is predictably always to the nearest second, could it also work to manually overwrite the saved mtimes in the .ji files after initially writing them?

[malmaud]

Yes, but that would require action on the package of the image creator, right? Seems undesirable if it can be avoided. If the rounding is always to the floor, then maybe mtime(f)=ftime could just be replaced with mtime(f)>ftime.

[josefsachsconning]

Yes, @tkelman, I was considering that.

Alternatively, how about

fmtime = mtime(f)
if fmtime != ftime && fmtime != floor(ftime)

replacing
https://github.com/JuliaLang/julia/blob/master/base/loading.jl#L511
?

[tkelman]

Equality is more robust against clock skew than inequality (which I was hitting, ref #12559). I like @josefsachsconning's idea of also checking for equality against the floor, since that's not as likely to collide by accident.

[malmaud]

Yes, that seems good - it's a minimal change to the current system that will solve this issue. I'll fire up a PR.

[dnk8n]

I seem to still hit this problem in Julia 0.6.4 (The Dockerfile if interested)

[stevengj]

@dnk8n, Julia < 1.0 is unsupported these days, so there is not much point in reporting bugs for such old versions.

[dnk8n]

Oops, Sorry, I forgot about that. I am used to using Julia 1.x when I can. Unfortunately these libraries are not compatible.