Custom NER training - understanding loss and learn rate schedule

[deepakdalakoti]

Hi,

I am training a custom NER model. I am trying to understand the output of the logger. I can see that the model outputs a few metrics including LOSS NER, ENTS_F, ENTS_P, ENTS_R, however I am not sure what dataset these metrics are calculated on. From my understanding so far, LOSS NER is calculated on the training set while other metrics (ENTS_F, ENTS_P, ENTS_R) are calculated on the dev set. Is that true.

A second question around implementing learning rate scheduler. I can see thinc provides some registry functions which I can use to change my learning rate. I am however not sure if the learning rate is updated every step or every epoch. Any clarifications would be really appreciated.

[thomashacker]

Hello,
You are correct. The performance metrics (ENTS_F, ENTS_P, ENTS_R) are calculated on the development set and the loss on the training set. The learning rate schedules for the optimizer are updated every step.

[yaelkurz]

what does Loss transformer and Loss NER refer to?

[thomashacker]

The loss indicates how well the model generalizes the training data and is used to perform backpropagation. The performance metrics are evaluated on the development data.