Persist meta data of experiments

[sperlingxx]

/kind feature

Describe the solution you'd like
Meta data generated during HPO process is worthy to restore. It can be used to guide subsequent tuning works. Maybe it can also be precious data of further academic research.

To me, since katib has DB abstraction layer (KatibDBInterface) , we can extend it to persist meta data such as experiment settings and corresponding trial information (param assignments and metrics).

Anything else you would like to add:
In current katib UI system, visualization will be unavailable once experiment instance deleted. If we can persist all information in DB, it is possible for users to review all history experiments. I think it will be helpful.
/cc @andreyvelich @gaocegege @richardsliu @hougangliu @johnugeorge

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.93

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[andreyvelich]

@sperlingxx Thank you for the issue. SGTM.
Previously, we stored Experiments search space in the Katib DB as well as metrics.
Currently, ones user deletes Experiment, all Trials metrics are cleaned in the DB.

Maybe we can control it with resuming Experiment flag.

[sperlingxx]

@andreyvelich I fully agree with the approach "control it with resuming Experiment flag". Shall we add this issue into 2020 roadmap?

[andreyvelich]

@gaocegege @johnugeorge Any thoughts?

[johnugeorge]

related:#1150 (comment)

[andreyvelich]

Resuming Experiment was implemented, feed free to re-open the issue if you have any other questions.

[pod3275]

I have one question about this issue.

We can now control whether the Suggestion deploy is deleted after the experiment is finished by setting ResumePolicy=never or ResumePolicy=FromVolume.

However, Trial, Suggestion, and Experiment CRs still remain.
If we delete Experiment CR, the results of the experiment will not be displayed on the Katib UI.
Managing the number of CRs could be challenging when a large number of experiments are performed.

As discussed at the beginning of this issue, how about adding an option to persist all meta data of the experiment in DB and delete Katib CRs?