A few question about validation, train losses and evaluation

[nihil39]

Hi,

I have already asked this in a old thread yesterday but maybe I won't get any answer so I decided to open a new one.

At the end of one of my training I have this results in the log file

+-------------+---------------------+------------------+-------------------+---------------------------------------+
| config_type | RMSE E / meV / atom | RMSE F / meV / A | relative F RMSE % | RMSE Stress (Virials) / meV / A (A^3) |
+-------------+---------------------+------------------+-------------------+---------------------------------------+
|    train    |         0.2         |       17.4       |        2.30       |                  0.2                  |
|    valid    |         0.2         |       17.6       |        2.32       |                  0.2                  |
|   Default   |         0.2         |       17.6       |        2.34       |                  0.2                  |
+-------------+---------------------+------------------+-------------------+---------------------------------------+

They appear to be normal because errors on training is slightly lower than the errors on validation and test sets.

{"loss": 0.040587769153017754, "time": 0.7549424171447754, "mode": "opt", "epoch": 2}
{"loss": 0.0038969941308411114, "mae_e": 0.62485660798769, "mae_e_per_atom": 0.0016662842879671736, "rmse_e": 0.7314864436925352, "rmse_e_per_atom": 0.0019506305165134274, "q95_e": 1.30251505414011, ... "time": 2.437013864517212, "mode": "eval", "epoch": 2}
{"loss": 0.037334632390278746, "time": 0.7609601020812988, "mode": "opt", "epoch": 3}

I can't understand why in the Results file I have the error on the training set (here I have copied only the last batch but it's the same for other) more or less ten times higher than the error on the validation set (which does not make any sense to me). It is the same for every epoch more or less.
I set the evaluation interval (eval_interval) of my model on every epoch, but what does it mean? Does it mean to compute the loss and other quantities like rmse on energies, forces etc on the validation set? I don't think I have understood this clearly.

What is the relation between errors and losses? I am using the sress loss function with Energy weight 1, Force weight = Stress weight = 1

[ilyes319]

Hello,
This behavior can happen even though it is not common. Usually the cause is the presence of noisy data. The reason why the evaluated model can be better than the trained one is because they are not the same. The evaluated model is an exponential moving average (ema) of the trained model over all its epoch. If your training is quite noisy, it is possible that you see a big different between the performance of non ema model compared to the ema model.