BentoML Roadmap Update - Q4 2020

[parano]

Hi everyone, this is a snapshot of our current roadmap for BentoML. We are going back to the schedule of releasing a new version every 2 weeks in October, release 0.9.0 will come out this week, and here's what to expect in the next a few releases. Comment below if you are interested in some of the features listed here, that will help us prioritize. And feel free to suggest other features you'd like to see in future versions of BentoML.

Edit: We have moved all roadmap information to the GitHub Projects page, under the Roadmap project: https://github.com/bentoml/BentoML/projects/1

[parano]

Improve Model Management functionality

• 0.10.0 Add the push/pull API for moving saved BentoService to and from a remote YataiService
• 0.10.0 Add documentation on best practices for running a shared YataiService for team collaboration
• 0.11.0 Add YataiService's own monitoring endpoint YataiService monitoring and prometheus endpoint #1060
• Allow configuring YataiService web UI's prefix URL Support URL location prefix in yatai-service gRPC/Web server #1063
• Allow attaching additional metadata with packaged model Add API for storing trained model metadata #881

[parano]

Improve Documentation

• 0.10.0 Best practices on integrating with model development workflow and training pipelines
• 0.10.0 Example projects showing sample project structure and best practices
• 0.11.0 Deep dive on how BentoML's adaptive micro-batching mechanism works and how it compares to Clipper and TF-serving's implementation
• 0.12.0 Finish the rest of the advanced guides section, on monitoring, logging best practices, how to customize artifact and adapters

[parano]

More built-in Input/Output adapters

• 0.10.0 Add FileOutput adapter
• 0.10.0 Add ImageOutput adapter
• 0.11.0 Add numpy ndarray input/output adapter

[parano]

Better Spark Support

• 0.11.0 Add Spark MLLib support Spark MLlib support #666
• 0.11.0 Document best practices on running distributed inference job with BentoML packaged model on Spark feat: Support distributed batch inferencing job on Apache Spark cluster #890
• 0.12.0 Add type convertion in defaul t input/output adapter for working with Spark dataset

[Talador12]

This is probably the # 2 value addition in my eyes

[parano]

"One-click deployment" for more platforms

BentoML currently provides simple model deployment management for AWS Lambda, AWS SageMaker, and Azure Functions. And we want to add support for these additional platforms:

• DONE 0.12.0 Add support for deploying directly to AWS EC2 with auto-scaling and load balancer
• Add support for deploying directly to GCP Cloud Run
• Support micro-batching when deploying to AWS SageMaker #673

[mayurnewase]

I can take on this for aws with cloudformation templates for one click deployment.

[parano]

@mayurnewase that'd be great!! Let me know if you want to chat about the deployment operator design in BentoML or have any questions, sorry that this part of the project is still lacking proper documentation. Ping me in slack if you need any help!

[parano]

API Server lifecycle callback functions

• 0.10.0 Add server startup and shutdown callback Add startup / shutdown functions #1065 Allow access to artifacts in the service __init__(self) during pack process #1037

GRPC support in API model server

• 0.12.0 Add support for gRPC/protobuf data for all the built-in input/output adapters and allow exposing a gRPC serving endpoint
• Support KFServing API V2 predict protocol Support KFServing API V2 predict protocol #899

Better KFServing integration

• 0.13.0 Support KFServing V2 API Support KFServing API V2 predict protocol #899

Model Orchestration

For users that are managing model serving workloads on a Kubernetes cluster, we are developing advanced model orchestration tools for you, more details/design will be released soon

• 0.12.0 Design doc & documentation for community review

[Talador12]

This is probably the # 1 value addition in my eyes

[parano]

Framework specific integration & support

• Better Tensorflow Integration Better TensorFlow integration #1188
• Add Pytorch Lightning support Add Pytorch Lightning support #1137
• PytorchModelArtifact support JIT traced models PytorchModelArtifact support JIT traced models #873
• Pycaret support Pycaret support #988

[parano]

API server improvements

• Better GPU support GPU Support #886
• Option to disable swagger UI Option to disable swagger UI #1167
• Allow adding extra custom URL routes Provide a general-purpose API decorator #1161
• Allow configuring API server logging behavior Configuration option to disable logs in production #1009
• Allow setting micro-batching parameters from CLI argument Allow setting micro-batching parameters via environment variable and CLI arg #930

[parano]

Advanced Model Packaging features

• Allow specifying allowed Python version range Provide option to enforce Python Runtime version via env_decorator / setup.py #1004

[parano]

Testing

• Test dockerized API Server in artifact integration test
  Test dockerized API Server in artifact integration test #891

[parano]

For those interested in following our progress on this roadmap, please refer to our roadmap project kanban here: https://github.com/bentoml/BentoML/projects/1