How does it proceed with this project?

[ynqa]

I'd like to contribute/help to this project, are there any milestones or actual codes for pipelines?

It looks like that there are two procedures:

1. Move argoproj/argo into here. Probably it's necessary to permit from argoproj members.
2. Implement the pipeline module from scratch using kubernetes APIs (e.g. CRD) like argo.

please reply if you don't mind.

[jlewi]

@vicaire is leading this project.

[vicaire]

Hi ynqa@, we plan to provide some tools built on top of Argo to schedule and trigger pipelines. Argo has an ambitious project to tackle this (https://github.com/argoproj/argo-events) and hopefully we can switch to what Argo provides once it has taken shape.

[ynqa]

@vicaire Thanks for your reply. that means this repo will include the tools which are wrapped Argo trigger/scheduler and specialized in kubeflow, not Argo itself?

[vicaire]

Yes. We are still figuring out the details.

The goal is to provide pipeline functionality that either doesn't not make sense to add to Argo, or that won't be available in Argo for a while. We are still figuring out the details and will discuss further with the Argo team.

[ynqa]

@vicaire I got it, thanks. If you decided what to make on this projects or there is anything that I can help with, please tell here. I'm going to cooperate argo projects progresses also.

[vicaire]

That sounds great. Thanks!

[gaocegege]

Hi, I am also interested in the project, could we add the scope of the project in README? Personally, I am not sure if I understand it. Do you want to implement the pipeline based on argo to support ML workload?

[vicaire]

Hi gaocegege, we are still figuring out the details and scope. Stay tuned. Thanks!

[ddysher]

From what I explored so far, the project now supports scheduled workflow on top of argo, and the main use case i've learned is to support training model on a regular basis, as mentioned here.

While @vicaire is working on the details, I'm curious about the regular training use case, what's the concrete scenario we are targeting? I don't quite understand this :)

For us, we have a use case to train a model, save the model somewhere (version controlled) then serve the model; preferably, the model can be validated and evaluated automatically, providing feedback for another round of training (not necessarily auto-start by a scheduled workflow). Is this a use case within our scope?

[jlewi]

Model validation and evaluation are within scope of Kubeflow and TFX but I think the initial work in this repo is more narrowly scoped; i.e ScheduledWorkflow.

@ddysher I think we as a community should figure out how to address use cases like the ones you mentioned. There are lots of issues about continuously deploying models e.g. using GitOps kubeflow/kubeflow#1118 and there are lot of pieces e.g (Argo, Weave, Seldon, etc...)

[gaocegege]

Could you please explain what ScheduledWorkflow is, I can not find the proposal or something else in the community, thanks. 😄

[IronPan]

@gaocegege ScheduleWorkflow is the CRD in this repo. Definition

[IronPan]

@ddysher You are correct that one of the main goal of this project is to run an argo workflow on a regular basis by with the help of ScheduledWorkflow CRD.
One aspect of the use case you are describing is event based triggering of an Argo workflow. Just FYI Argo community is working on a general solution. Although we might need to evaluate whether it would the best fit for a ML specific workflow.

[ddysher]

@jlewi @IronPan thanks for the kind reply.

I understand argo workflow and how we leverage it for kubeflow. My question above is why do we want a ScheduledWorkflow for kubeflow, what is the concrete use case? From what i understand, regular training won't be useful unless we have feedback from somewhere.

[IronPan]

@ddysher Thanks for clarifying the question. Argo was initially created for K8s native CI/CD. The pipeline project is a nature extension of the idea that applies to the case of ML model development. One of the main use case for ScheduledWorkflow CRD is continuous learning. It's not rare, especially in production environment, to see model continuous adapting the new data coming in, and continuously evolving on a regular basis(daily/weekly/monthly). Initiating the process manually would be a repetitive and tedious work that naturally make sense to be automated. Ideally, the system would fully automate the workflow that does data ingesting, training, model analysis, and deployment.

[ddysher]

Thanks @IronPan, what you described aligns with my understanding. We have similar effort internally so would like to merge the effort, do you have an estimated time of the proposal? @vicaire @IronPan

/cc @Muzry

[vicaire]

Hi @ddysher,

We do not have an estimated time yet for a proposal but will update the thread as soon as we do. Thanks!

[nicholas-fwang]

It's interested project to me as well.
@IronPan I understand that continuous learning means model's CD in production env. then, as you know, this process has to be rollback when bad evaluation. Is it considered in this project? for this, I think, needs model or data versioning like pachyderm. and, I think, in model continuous learning, it's difficult point. so, how do you think about managing data/model version?
I hope to participate in this project as possible 👍

[vicaire]

Thanks fisache@. We haven't yet established the exact scope of this project. For now, it only aims to provide a set of useful tools for running workflows. We will provide more information as we make progress. Thanks.