Set model attributes from runner?

[RobbieFernandez]

I'm using BentoML to wrap a YOLO model that's created with this library: https://github.com/fcakyon/yolov5-pip.

From the library docs, this is how it can be used outside of BentoML (Note that are extra parameters that can be set on the model instance that will affect the inference) :

model = yolov5.load('train/best.pt')
  
# set model parameters
model.conf = 0.25  # NMS confidence threshold
model.iou = 0.45  # NMS IoU threshold
model.agnostic = False  # NMS class-agnostic
model.multi_label = False  # NMS multiple labels per box
model.max_det = 1000  # maximum number of detections per image

# set image
img = 'https://github.com/ultralytics/yolov5/raw/master/data/images/zidane.jpg'

# perform inference
results = model(img)

I'm saving the model with this code:

model = yolov5.load('train/best.pt')

bentoml.pytorch.save_model(
        "Yolo model",
        model,
        signatures={"__call__": {"batchable": False}},
        **kwargs
)

My question is, when writing a service for this model, is it possible to expose those additional attributes (model.conf etc)? I would like for clients to be able to specify the confidence threshold when making inference requests, but I'm not sure how I can set those attributes with a Runner instance.

[RobbieFernandez]

I wasn't able to do it with the bentoml.pytorch.get("..").to_runner() runner. But I've written my own basic runner that adds the parameter:

class YoloRunnable(bentoml.Runnable):
    SUPPORTS_CPU_MULTI_THREADING = True

    def __init__(self, model_file):
        self.model = torch.hub.load(
            'ultralytics/yolov5',
            'custom',
            path=model_file,
            trust_repo=True
        )

    @bentoml.Runnable.method()
    def predict(self, image, confidence_threshold):
        # Mutating this object seems like a bad idea :(
        self.model.conf = confidence_threshold
        return self.model(image)


runner = bentoml.Runner(
    YoloRunnable,
    name="fish_finder",
    runnable_init_params={"model_file": "my_model_file"}
)

My question now is, are there any issues around mutating the model like that? If there are multiple requests being handled asynchronously will it cause them to interfere with each other?