Functions inside pmap call can't see SharedArrays unless preceded by @async

[JohnHBrock]

I don't know if this is a feature or a bug. Functions used inside a pmap call can only see SharedArrays when the pmap call is preceded by @async. To replicate, be sure to start Julia with more than one process (e.g., julia -p 2):

julia> foo = convert(SharedArray, [1,2,3,4]);

julia> @async pmap(i->println(foo), 1:2) 
Task (waiting) @0x000000010cd8f730 

julia> From worker 2: [1,2,3,4] 
       From worker 3: [1,2,3,4] 

julia> pmap(i->println(foo), 1:2) 
2-element Array{Any,1}: 
 RemoteException(2,CapturedException(UndefVarError(:foo),Any[(:anonymous,:none,1,symbol(""),-1,1),(:anonymous,symbol("multi.jl"),907,symbol(""),-1,1),(:run_work_thunk,symbol("multi.jl"),645,symbol(""),-1,1),(:anonymous,symbol("multi.jl"),907,symbol("task.jl"),63,1)])) 
 RemoteException(3,CapturedException(UndefVarError(:foo),Any[(:anonymous,:none,1,symbol(""),-1,1),(:anonymous,symbol("multi.jl"),907,symbol(""),-1,1),(:run_work_thunk,symbol("multi.jl"),645,symbol(""),-1,1),(:anonymous,symbol("multi.jl"),907,symbol("task.jl"),63,1)]))

julia> versioninfo()
Julia Version 0.4.2
Commit bb73f34 (2015-12-06 21:47 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin15.0.0)
  CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

I can also replicate this on Julia v0.3.9.

[amitmurthy]

@async localizes the variables which results in the shared array objects being local to the closure and hence serialized to the remote worker. This is another manifestation of the general scoping issue JuliaLang/Distributed.jl#30

[amitmurthy]

Same as #13756

[amitmurthy]

Closed by #19594

[tkelman]

is there a test for this?

[amitmurthy]

This and the others you reported -

julia/test/parallel_exec.jl

Lines 1392 to 1422 in 1303dfb

	# reported github issues - Mostly tests with globals and various parallel macros
	#2669, #5390
	v2669=10
	@test fetch(@spawn (1+v2669)) == 11
	#12367
	refs = []
	if true
	n = 10
	for p in procs()
	push!(refs, @spawnat p begin
	@sync for i in 1:n
	nothing
	end
	end)
	end
	end
	foreach(wait, refs)
	#14399
	s = convert(SharedArray, [1,2,3,4])
	@test pmap(i->length(s), 1:2) == [4,4]
	#6760
	if true
	a = 2
	x = @parallel (vcat) for k=1:2
	sin(a)
	end
	end
	@test x == map(_->sin(2), 1:2)

[amitmurthy]

Generally they were all manifestations of the same issue.