[Feature request]: Save models to disk progressively instead of once at the end

[mxgrey]

Before proceeding, is there an existing issue or discussion for this?

• I have done a search for similar issues and discussions.

Description

I haven't looked deeply into the implementation, but based on observed behavior, it seems that models only get saved to disk after all data has been retrieved. This means that if the cache update gets interrupted, no models will be preserved at all. Since the download can take a significant amount of time, this could lead to a lot of wasted effort.

It would be nice if models could get saved to disk as they arrive so even an incomplete cache update has some benefit.