Model keeps reloading shards in every iteration

[arunpatro]

When I run this program, the HF model loads the shards again and again for each loop iteration. Could this be a bug, where the model is not cached in memory and needs to reload again and again?

import outlines.models as models
from tqdm import tqdm

inputs = ["apple", "banana", "oranges", "grapes"]

model  = models.text_completion.hf("mosaicml/mpt-7b-instruct", max_tokens=50)

for prompt in tqdm(inputs):
    answer = model(prompt)
    print(answer)

[rlouf]

This is due to the fact that we load the model when it is called. We could load it when the model is initialized, but this leads to other problems when several models are involved. I will soon open an issue regarding the implementation of a supervisor process that handles the loading/unloading of models.

[rlouf]

This won't happen anymore with the new API since #139, closing. We will have different problems when it comes to using different models, but this is tracked by #162