Persist meta data of experiments in DB

[pod3275]

/kind feature

Describe the solution you'd like

I have one question about issue #1128.

We can now control whether the Suggestion deploy is deleted after the experiment is finished by setting ResumePolicy=never or ResumePolicy=FromVolume.

However, Trial, Suggestion, and Experiment CRs still remain.
If we delete Experiment CR, the results of the experiment will not be displayed on the Katib UI.
Managing CRs could be challenging when a large number of experiments are performed.

As discussed at the beginning of this issue, how about adding an option to persist all meta data of the experiment in DB and delete Katib CRs?

Anything else you would like to add:

---

Love this feature? Give it a 👍 We prioritize the features with the most 👍

[johnugeorge]

Is this feature for solving the need to persist metadata experiment in DB or for need to delete the Katib CRs?

[pod3275]

@johnugeorge Thank you for your reply!

I hope that the results of the experiment will be displayed in the Katib UI even if the Katib CRs for the experiment that has already been completed are deleted.

In fact, the metadata of the experiment does not necessarily have to be stored in DB. I just thought the most convenient way to permanently persist metadata was to store it in DB.

[johnugeorge]

@pod3275 metrics are fetched from DB, but other metadata like Experiment. Trials, etc are using k8s storage itself. If we delete CRs, we wil lose these info.

In the previous version, we had a active backup in DB. But, we found that it is difficult to keep them in sync. Hence, we moved to this design to have a single source of truth. We can use DB for archival purposes if we need such a feature.(passive)

[pod3275]

@johnugeorge
Thank you for your detailed reply!

From this previous issue, I expected that there was a DB schema for storing experiment meta data, but now I understand why the method of storing has changed.

As you mentioned, in the case of an active experiment, I agree to bring the meta data of the experiment by referring to k8s storage.

But in the case of a completed experiment, how about storing the meta data of the experiment in DB when it is completed, and displaying the results in Katib UI by referring to DB? In this way, it is not necessary to keep them in sync, and it will be easy to manage Katib CRs when the number of experiments is large.

[skgreenstar]

When the experiment is completed, it is necessary to store the experiment metadata in the DB and refer to the DB to deliver the results to the Katib UI!

The reason for this is that we are eating up resources in k8s, so we are interfering with resource organization.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.