Skip to content

Latest commit

 

History

History
127 lines (95 loc) · 9.66 KB

HISTORY.CM.md

File metadata and controls

127 lines (95 loc) · 9.66 KB

This document narrates the history of the creation and design of CM and CM4MLOps (also known as CK2) by Grigori Fursin. It also highlights the donation of this open-source technology to MLCommons, aimed at benefiting the broader community and fostering its ongoing development as a collaborative, community-driven initiative:

  • Jan 28, 2021: After delivering an invited ACM TechTalk'21 about the Collective Knowledge framework (CK1) and reproducibility initiatives for conferences, as well as CK-MLOps and MLPerf automations, Grigori received lots of positive feedback and suggestions for improvements to workflow automations: https://learning.acm.org/techtalks/reproducibility.

    Following this, Grigori began prototyping CK2 (later CM) to streamline CK1, CK-MLOps and MLPerf benchmarking. The goal was to dramatically simplify CK1 workflows by introducing just a few core and portable automations, which eventually evolved into CM script and CM cache.

    At that time, the cTuning foundation hosted CK1 and all the prototypes for the CM framework at https://github.com/ctuning/ck: ref1, ref2, ref3.

  • Sep 23, 2021: donated CK1, CK-MLOps, MLPerf automations and early prototypes of CM from the cTuning repository to MLCommons: ref1, ref2, ref3, ref4, ref5.

    Prepared MLCommons proposal for the creation of the MLCommons Task Force on Automation and Reproducibility, aimed at fostering community-driven support for CK and CM developments to benefit everyone.

  • Jan, 2022: hired Arjun Suresh at OctoML to support and maintain CK1 framework and help prepare OctoML's MLPerf submissions using CK1. Meanwhile, transitioned to focusing on CM and CM-MLOps development, building upon the prototypes created in 2021.

  • Mar 1, 2022: started developing cm-mlops: ref.

  • Mar 24, 2022: after successfully stabilizing the initial prototype of CM, donated it to MLCommons to benefit the entire community: ref1, ref2, ref3, ref4.

  • Apr 6, 2022: started transitioning previous MLOps and MLPerf automations from the mlcommons/ck-mlops format to the new CM format using the cm-mlops repository (will be later renamed to cm4mlops): ref1, ref2.

  • Apr 22, 2022: began architecting "Intelligent Components" in the CM-MLOps repository, which will be renamed to CM Script at a later stage: ref1, ref2, ref3.

    At the same time, prototyped other core CM automations, including IC, Docker, and Experiment: ref1, ref2.

  • Apr 28, 2022: donated CM-MLOps to MLCommons, which was later renamed to CM4MLOps: ref.

  • May 9, 2022: developed the initial set of core IC automations for MLOps (aka CM scripts): ref1, ref2, ref3.

  • May 11, 2022: After successfully prototyping CM and CM-MLOps, deprecated the CK1 framework in favor of CM. Transferred Arjun Suresh to the CM project as a maintainer and tester for CM and CM-MLOps: ref.

    Created a file to document and track our public developments at MLCommons.

  • Jun 8, 2022: renamed the 'IC' automation to the more intuitive 'CM script' automation. ref1, ref2, ref3.

  • Jun 16, 2022: prototyped the CM cache automation to facilitate caching and reuse of the outputs from CM scripts: ref1, ref2, ref3.

  • Sep 6, 2022: delivered CM demo to run MLPerf while deprecating CK1 automations for MLPerf: ref1, ref2.

    Welcomed Arjun Suresh as a contributor to CM automations for MLPerf: ref.

  • From September 2022: coordinated community development of CM and CM4MLOps to modularize and automate MLPerf benchmarks and support reproducibility initiatives at ML and Systems conferences through the MLCommons Task Force on Automation and Reproducibility.

    • Directed and financed the creation of (CM) automations to streamline the MLPerf power measurement processes.

    • Proposed to use MLPerf benchmarks for the Student Cluster Competition, led the developments and prepared a tutorial to run MLPerf inference at SCC'22 via CM: ref

  • April 2023: departed OctoML to focus on the development of the CK playground and CM automations to make Mlperf accessible to everyone. Hired Arjun Suresh to help with developments.

    • Initiated and funded development of the MLPerf explorer to improve visualization of results

  • August 2023: organized the 1st mass-scale MLPerf community submission of 12217 inference benchmark v3.1 results out of total 13351 results (including 90% of all power results) across diverse models, software and hardware from different vendors via open challenges funded by cTuning.org : LinkedIn article with results visualized by the MLPerf explorer, CM4MLOps challenges at GitHub.

  • February, 2024: proposed to use CM to automate MLPerf automotive benchmark (ABTF).

    • moved my prototypes of the CM automation for ABTF to cm4abtf repo: ref
    • led further CM4ABTF developments funded by cTuning.org.

  • Starting in April 2024, began the gradual transfer of ongoing maintenance and enhancement responsibilities for CM and CM4MLOps, including MLPerf automations, to MLCommons. Welcomed Anandhu Sooraj as a maintainer and contributor to CM4MLOps with MLPerf automations.

  • Took a break from all development activities.

  • July 2024: started prototyping the next generation of CM (CMX and CMX4MLOps) with simpler interfaces based on user feedback while maintaining backward compatibility.

  • 2025: continue developing CMX and CMX4MLOPs to make it easier to run and customize MLPerf inference, training and other benchmarks across diverse models, datasets, software and hardware.

For more details, please refer to the white paper and the ACM REP'23 keynote.