latest docker image with mlflow v1.20 gives protobuf error

[salil-at]

Description

I wanted to use the mlflow docker image and ran into the below protobuf error.

Steps to reproduce:

$ docker run --rm -it -p 5000:5000 atcommons/mlflow-server:latest
Traceback (most recent call last):
  File "/usr/local/bin/mlflow", line 5, in <module>
    from mlflow.cli import cli
  File "/usr/local/lib/python3.10/site-packages/mlflow/__init__.py", line 32, in <module>
    import mlflow.tracking._model_registry.fluent
  File "/usr/local/lib/python3.10/site-packages/mlflow/tracking/__init__.py", line 8, in <module>
    from mlflow.tracking.client import MlflowClient
  File "/usr/local/lib/python3.10/site-packages/mlflow/tracking/client.py", line 16, in <module>
    from mlflow.entities import Experiment, Run, RunInfo, Param, Metric, RunTag, FileInfo, ViewType
  File "/usr/local/lib/python3.10/site-packages/mlflow/entities/__init__.py", line 6, in <module>
    from mlflow.entities.experiment import Experiment
  File "/usr/local/lib/python3.10/site-packages/mlflow/entities/experiment.py", line 2, in <module>
    from mlflow.entities.experiment_tag import ExperimentTag
  File "/usr/local/lib/python3.10/site-packages/mlflow/entities/experiment_tag.py", line 2, in <module>
    from mlflow.protos.service_pb2 import ExperimentTag as ProtoExperimentTag
  File "/usr/local/lib/python3.10/site-packages/mlflow/protos/service_pb2.py", line 18, in <module>
    from .scalapb import scalapb_pb2 as scalapb_dot_scalapb__pb2
  File "/usr/local/lib/python3.10/site-packages/mlflow/protos/scalapb/scalapb_pb2.py", line 29, in <module>
    options = _descriptor.FieldDescriptor(
  File "/usr/local/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

Additional Information

I found the bug being discussed in mlflow issues - mlflow/mlflow#5949

Possible Fix

I didn't try much, upgrading the mlflow version from 1.20 to 1.27 fixed the problem for me.

[mathematicalmichael]

I had a similar problem with https://github.com/ml-starter-packs/mlflow-experiment and ==1.27.0 seemed to fix it for me. I was on ~=1.24.0 before.

[amirhmk]

How did you upgrade mlflow? The latest tag for atcommons/mlflow-server seems to be 1.20 still.

[jasonmhite]

@amirhmk Clone this repo, edit requirements.txt to the newer version, edit docker-compose.yml to say image: mlflow-server, docker compose build, docker compose up -d.

Though I note I had problems with 1.27.0, I just changed requirements.txt to mlflow (no equals) to install the latest version as of today and that worked.