MetricsCollector does not parse time of the metric

[andreyvelich]

/kind bug

I can't see time in observation_logs table after experiment finish.

What steps did you take and what happened:
I ran random-example on my GCP cluster. In the Trial's containers I got warning like this:

W1202 17:26:56.160885      15 file-metricscollector.go:50] Metrics will not have timestamp since error parsing time INFO:root:Epoch[0]: parsing time "INFO:root:Epoch[0]" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "INFO:root:Epoch[0]" as "2006"

After that, I checked information in observation_logs table and I saw this time: 001-01-01T00:00:00 in each recorded metric.
From my understanding, if we use default metrics collector, it parses /var/log/katib/metrics.log file. This file doesn't contain timestamp of the metric and we can't get information about metrics changing through time.
Maybe we can save also timestamp for all lines in the log file?
Or we can find better solution to indicate timestamp of the metrics somehow.
/cc @hougangliu @gaocegege @johnugeorge

[hougangliu]

default metricsCollector always try to parse each line of user-application log, and timestamp is expected at start of the line, if there is no timestamp, ignore timestamp (use NULL in time field).
So maybe we can fix the example image and make its log follow this format

[andreyvelich]

@hougangliu Yes, I agree we need to create example with printing timestamp in training container, like we did with file-metrics-collector.
Will take it.
/assign

[andreyvelich]

@hougangliu Should I use docker.io/kubeflowkatib repo to push my image or I should use my own docker hub?

[hougangliu]

we should use docker.io/kubeflowkatib, please leave your docker hub account so that I can add you to the org

[andreyvelich]

@hougangliu Sure, my id is: andreyvelichkevich