Does workers hold previous intermediate task results after a compute?

[collinwo]

In below two examples, plan_valuation will generate a graph in a range of dates. If the workers in grid keep the intermediate tasks results, I guess Cell-2 will run very fast, which may take no time. However, It looks like Cell-2 run doesn't leverage the Cell-1 run and its intermediate tasks result in workers. Did I understand dask cache wrongly?

Cell-1
%%time
dates = [datetime.datetime(2018,5,17-n) for n in [0,1,2,3,4,5]]
ts2 = plan_valuation('1234', dates)
t2 = client.persist(ts2)
t2 = client.compute(t2)
res2 = t2.result()

Cell-2
%%time
ts2 = plan_valuation('1234', dates[1:3])
t2 = client.compute(ts2)
res2 = t2.result()

[collinwo]

BTW, I checked worker.data dict. It looks like that it will clean all intermediate results after a run. So, later run may see no tasks results. Correct me if I am wrong.

Is there a way to enable some cache sharing between runs?

[mrocklin]

Calling persist will keep the final results in distributed memory. Intermediate results will be cleared.

Overwriting the value t2 will cause the workers to remove those final results.

There is an open issue for opportunistic caching of intermediates.

[collinwo]

Yes, opportunistic caching is what I want. Had followed that issue.