Web UI should allow access to model training/validation metrics in real time

[mameshini]

Currently, TensorBoard button is only available in Pipelines UI at the end of each experiment run.

As a data scientist performing model training, I want to access to TensorBoard immediately after the start of the pipeline step, so I can monitor taining/validation metrics and often I may want to abort an experiment based on model metrics. TensorBoard is logging training and validation metrics such as error, loss, and accuracy. These metrics have to be used in real time to diagnose everything from vanishing gradients to exactly when models start overfitting. There is no need to wait extra hours to complete the run, if it does not help to improve the model.

This issue is related to:
#1497

TensorBoard data should be stored in a volume, and Pipelines UI can provide a button to TensorBoard once the data is available without waiting to complete the pipeline step.

[mameshini]

Duplicate of 2416