LOSS is skyrocketing when trying to train a NER classifier with transformers #9090

dchaplinsky · 2021-08-30T15:56:45Z

dchaplinsky
Aug 30, 2021

Hello.

I'm trying to train a SpaCy NER classifier for Ukrainian language (https://github.com/lang-uk/ner-uk) using https://huggingface.co/youscan/ukr-roberta-base roberta embeddings.

I've used config, generated on the SpaCy website
After a couple of epochs my learning rate is skyrocketing like that:

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'ner']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  --------  ------  ------  ------  ------
  0       0         196.73   1242.89    2.85    1.47   45.59    0.03
  1     200      284768.46  172895.53    0.00    0.00    0.00    0.00
  3     400       55577.38  24946.34    0.00    0.00    0.00    0.00
  5     600     1232626.81  24823.55    0.00    0.00    0.00    0.00
  6     800      439013.75  23708.81    0.00    0.00    0.00    0.00
  8    1000      278862.71  23112.37    0.00    0.00    0.00    0.00
 10    1200      812892.02  23164.47    0.00    0.00    0.00    0.00
 11    1400       44229.09  22807.88    0.00    0.00    0.00    0.00
 13    1600     6189781.89  22582.66    0.00    0.00    0.00    0.00

I've tried to increase initial learning rate a bit (from initial_rate = 5e-5 to initial_rate = 0.01) but that didn't help.

On my first run it seems that model didn't train at all. On the second run it looks anything but promising too.
Please, help :) I really want to create and publish that SpaCy model on the data we've tagged. We already have models for MITIE, Flair and Stanza.

I'm attaching my config below:

[paths]
train = "/tmp"
dev = "/tmp"
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "uk"
pipeline = ["transformer","ner"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "youscan/ukr-roberta-base"

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.tokenizer_config]
use_fast = true

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.01

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

Answered by dchaplinsky

Feb 17, 2023

I'm glad to let you know that somehow the problem disappeared.
I ran it today with my old configs using new version of spacy and spacy-transformers and voila:

ℹ Initial learn rate: 1e-05
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  -------------  --------  ------  ------  ------  ------
  0       0         898.86    701.86    1.00    0.52   11.43    0.01

  1     200       12545.41  13049.79   75.57   79.44   72.05    0.76
  2     400        2371.97   2280.48   82.97   83.34   82.60    0.83
  4     600        1049.16   1168.24   84.29   85.43   83.18    0.84
  5     800         507.07    606.52   85.73   86.99   84.50    0.86

View full answer

dchaplinsky · 2021-08-30T15:59:06Z

dchaplinsky
Aug 30, 2021
Author

train_test_data.zip
I'm attaching my train/test dataset in both, json and binary format just in case someone would like to look into it or try it.

0 replies

polm · 2021-08-31T04:29:16Z

polm
Aug 31, 2021

P/R/F1 scores of 0 generally suggest an issue with the data rather than a hyperparameter problem. I would suggest trying a CPU model but it looks like we don't have those for Ukranian.

What's the output of spacy debug data with your data?

1 reply

dchaplinsky Aug 31, 2021
Author

Hello and thanks for the quick response.

Here is the full output of the debug data:

============================ Data file validation ============================
Some weights of the model checkpoint at youscan/ukr-roberta-base were not used when initializing RobertaModel: ['lm_head.decoder.bias', 'lm_head.layer_norm.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.decoder.weight', 'lm_head.layer_norm.weight', 'lm_head.bias']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: uk
Training pipeline: transformer, ner
1060 training docs
1060 evaluation docs
⚠ 1060 training examples also in evaluation data
⚠ Low number of examples to train a new pipeline (1060)

============================== Vocab & Vectors ==============================
ℹ 237327 total word(s) in the data (50204 unique)
ℹ No word vectors present in the package

========================== Named Entity Recognition ==========================
ℹ 4 label(s)
0 missing value(s) (tokens with '-' label)
⚠ 7 entity span(s) with punctuation
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels
✔ No entities consisting of or starting/ending with whitespace
Entity spans consisting of or starting/ending with punctuation can not be
trained with a noise level > 0.

================================== Summary ==================================
✔ 5 checks passed
⚠ 3 warnings

I'm quite sure in my original data, since we've used it to successfully train other models. Maybe I converted them wrong or the Roberta embeddings are wrong?

dchaplinsky · 2021-08-31T06:39:09Z

dchaplinsky
Aug 31, 2021
Author

Aha, one obvious thing is that I need to explicitly set path to the file with train and dev data, not a path to the folder that contains both (some models I've used to train worked with directory, autodiscovering the data files)

3 replies

polm Sep 1, 2021

You can specify a folder with data and all spaCy files in the directory will be used, though it won't automatically differentiate between train/dev data.

svlandeg Sep 2, 2021
Maintainer

Does that change the output of the debug data command?

dchaplinsky Sep 2, 2021
Author

Yes, a tiny bit (it stopped complaining about the overlap between train and dev, for example) :)

But anyway, as I told below, it doesn't seems like an issue with the data. Once I've sorted out data paths, the CPU model converged quite fast and reasonably well (up-to 0.8 of F1, when using my own fasttext vectors), while on the same data, transformer model behavior remains the same (i.e bad).

dchaplinsky · 2021-08-31T06:58:41Z

dchaplinsky
Aug 31, 2021
Author

I've regenerated data files like that now, dropping -n 10 param:

!python -m spacy convert /gdrive/MyDrive/NER_UK/train.txt.iob /tmp
!python -m spacy convert /gdrive/MyDrive/NER_UK/test.txt.iob /tmp

and specified correct paths to train/dev.

No improvement:

ℹ Initial learn rate: 1e-06
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  --------  ------  ------  ------  ------
  0       0         216.65   1792.17    3.18    1.64   49.30    0.03
  1     200       78034.04  218759.47    3.81    1.98   48.49    0.04
  3     400         346.10  212207.07    3.81    1.98   48.49    0.04
  4     600         279.08  215531.78    3.81    1.98   48.49    0.04
  6     800         322.94  210982.32    3.81    1.98   48.57    0.04

Doesn't look like it converges at all.

What do I need to train CPU model? Some kind of vectors? How can I train them myself?

2 replies

polm Sep 1, 2021

Hm, that is odd. Normally this would only happen with serious data problems, like maybe an incompatible tokenizer or something, but debug data didn't show anything like that...

What do I need to train CPU model? Some kind of vectors? How can I train them myself?

You can train a model on CPU by using a tok2vec instead of Transformers, you don't need any data in particular. You could certainly try it here.

For pretrained pipelines we provide we always have vectors, NER data, Part of Speech, and Dependency Parse annotations. Even if you don't use all those components, the model trained on them often has good representations for many tasks.

polm Sep 1, 2021

Using a CPU model with the default Quickstart settings (Ukranian/CPU/NER/efficiency) I was rapidly able to train a model with 50F1 (I didn't let it run to completion), so that suggests your data is OK. I don't have a GPU that can use the default transformer model so I can't test that directly.

A couple of things that might be problems:

spaCy trains the transformer rather than using it for feature extraction. Since you have little data that might be an issue. Maybe try freezing the transformer and listing it under "annotating components".
Your documents are kind of long (looks like average 256 tokens?). Maybe something is going wrong with the way spans are passed to the transformer model. It will increase memory requirements a lot but maybe try increasing the values in get_spans

dchaplinsky · 2021-09-01T06:19:10Z

dchaplinsky
Sep 1, 2021
Author

Hello and thanks for the detailed answers!

I've also trained a model using CPU config (and fasttext vectors I had for Ukrainian) and got F1 around 0.8 (which is comparable to the other models like MITIE/Flair/Stanza, but a bit lower). So I assume, my data is okay.

Speaking of number of tokens, is it related to the -n 10 setting I've used when converted my IOB data? (in any case, I've tried to train transformers model using both, grouped and singular sentences with no result)

Maybe try freezing the transformer and listing it under "annotating components".

I'll give it a try as soon as I realize how to change my config for that (any guidance is more than welcome)!

1 reply

polm Sep 1, 2021

Speaking of number of tokens, is it related to the -n 10 setting I've used when converted my IOB data? (in any case, I've tried to train transformers model using both, grouped and singular sentences with no result)

Ah, that could definitely be making your docs longer. Can you run debug data without using -n 10? And was the data you provided for download made with -n 10 or not?

I'll give it a try as soon as I realize how to change my config for that (any guidance is more than welcome)!

frozen_components = []
annotating_components = ["transformer"]

I thought you had to put it in frozen components too, but that appears to not be the case - this is a usage pattern we've only recently introduced so I'm still getting used to it.

dchaplinsky · 2021-09-01T07:16:19Z

dchaplinsky
Sep 1, 2021
Author

Those files I've uploaded initially was made with -n 10.

Check files below.
updated_data.zip

Thanks for the hint on frozen components, will give it a try

0 replies

dchaplinsky · 2021-09-01T07:18:52Z

dchaplinsky
Sep 1, 2021
Author

Close but no cigar:

⚠ Aborting and saving the final best model. Encountered exception:
TypeError("'NoneType' object is not callable")
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/train.py", line 63, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 209, in train_while_improving
    annotates=annotating_components,
  File "/usr/local/lib/python3.7/dist-packages/spacy/language.py", line 1123, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])
  File "spacy/pipeline/transition_parser.pyx", line 416, in spacy.pipeline.transition_parser.Parser.update
  File "spacy/ml/parser_model.pyx", line 300, in spacy.ml.parser_model.ParserStepModel.finish_steps
  File "/usr/local/lib/python3.7/dist-packages/thinc/layers/chain.py", line 60, in backprop
    dX = callback(dY)
  File "/usr/local/lib/python3.7/dist-packages/thinc/layers/chain.py", line 60, in backprop
    dX = callback(dY)
  File "/usr/local/lib/python3.7/dist-packages/spacy_transformers/layers/listener.py", line 39, in backprop_and_clear
    result = self._backprop(*args, **kwargs)
TypeError: 'NoneType' object is not callable

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = ["transformer"]
before_to_disk = null

1 reply

polm Sep 1, 2021

Can you try putting "transformer" in the frozen_components list? I think that should work. (My using just annotating_components on CPU may have been related to not having a pretrained tok2vec, or I might just be confused.)

dchaplinsky · 2021-09-01T10:19:02Z

dchaplinsky
Sep 1, 2021
Author

I actually tried all 4 possible combinations.

Putting transformers in frozen_components always ending up like this:

=========================== Initializing pipeline ===========================
[2021-09-01 10:17:18,369] [INFO] Set up nlp object from config
[2021-09-01 10:17:18,378] [INFO] Pipeline: ['transformer', 'ner']
[2021-09-01 10:17:18,382] [INFO] Created vocabulary
[2021-09-01 10:17:18,383] [INFO] Finished initializing nlp object
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/train.py", line 60, in train_cli
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/initialize.py", line 84, in init_nlp
    nlp.initialize(lambda: train_corpus(nlp), sgd=optimizer)
  File "/usr/local/lib/python3.7/dist-packages/spacy/language.py", line 1273, in initialize
    proc.initialize(get_examples, nlp=self, **p_settings)
  File "spacy/pipeline/transition_parser.pyx", line 565, in spacy.pipeline.transition_parser.Parser.initialize
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 299, in initialize
    self.init(self, X=X, Y=Y)
  File "/usr/local/lib/python3.7/dist-packages/spacy/ml/tb_framework.py", line 45, in init
    model.get_ref("tok2vec").initialize(X=X)
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 299, in initialize
    self.init(self, X=X, Y=Y)
  File "/usr/local/lib/python3.7/dist-packages/thinc/layers/chain.py", line 86, in init
    layer.initialize(X=curr_input, Y=Y)
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 299, in initialize
    self.init(self, X=X, Y=Y)
  File "/usr/local/lib/python3.7/dist-packages/thinc/layers/chain.py", line 90, in init
    curr_input = layer.predict(curr_input)
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 315, in predict
    return self._func(self, X, is_train=False)[0]
  File "/usr/local/lib/python3.7/dist-packages/spacy_transformers/layers/listener.py", line 61, in forward
    width = model.get_dim("nO")
  File "/usr/local/lib/python3.7/dist-packages/thinc/model.py", line 175, in get_dim
    raise ValueError(err)
ValueError: Cannot get dimension 'nO' for model 'transformer-listener': value unset

8 replies

dchaplinsky Sep 4, 2021
Author

So, guess what. I've tried replacing ukrainian roberta with xlm-roberta-baseand it immediately gave me some nice results:

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'ner']
ℹ Initial learn rate: 0.0001
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  --------  ------  ------  ------  ------
  0       0        8711.37   1560.51    3.14    1.62   49.46    0.03
  1     200       44886.04  10887.49   73.69   77.97   69.84    0.74
  3     400        7073.40   3331.04   80.07   80.65   79.50    0.80
  4     600        2463.16   2813.93   79.76   80.63   78.91    0.80
  6     800        1685.65   1793.83   80.83   82.29   79.42    0.81
  7    1000        1310.40   1694.30   77.79   78.78   76.82    0.78
  9    1200        1491.74   1407.26   77.52   79.18   75.93    0.78
 10    1400        1587.34   1385.44   82.80   83.89   81.74    0.83
 12    1600         586.28    704.68   81.82   82.09   81.55    0.82
 13    1800         324.61    502.20   76.81   79.31   74.46    0.77
 15    2000         349.14    500.89   82.01   82.84   81.20    0.82
 16    2200         380.96    398.59   80.12   80.43   79.81    0.80
 18    2400         363.75    486.01   78.73   79.86   77.64    0.79
 20    2600         292.58    373.20   81.39   81.93   80.85    0.81
 21    2800         202.81    288.49   76.82   78.15   75.54    0.77
 23    3000         309.55    334.86   81.40   82.41   80.43    0.81

0.83 is close to what we've got on that data from other models like Flair, Mitie and Stanza.

No idea, what's wrong with ukrainian-roberta-base tho :(. I tested it for fill mask and other tasks (but not in downstream models like NER) and results was sane.

polm Sep 5, 2021

Thanks for checking that! That definitely suggests something is weird with the Ukranian model, we'll have a look at it. Sorry for the trouble.

dchaplinsky Sep 5, 2021
Author

My pleasure! Thanks for all the help!

dchaplinsky Sep 5, 2021
Author

Oh. I've tried to run it with another ROBERTA model for Russian.

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'ner']
ℹ Initial learn rate: 0.0001
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  --------  ------  ------  ------  ------
  0       0        2740.32   1707.12    3.81    1.98   48.49    0.04
  1     200        1373.57  210322.72    3.81    1.98   48.49    0.04
  3     400         314.47  208670.00    3.81    1.98   48.49    0.04
  4     600         196.74  208809.85    3.81    1.98   48.49    0.04
  6     800         259.57  200371.28    3.81    1.98   48.49    0.04
  7    1000         230.81  199319.66    3.81    1.98   48.49    0.04
  9    1200         279.23  179387.57    3.81    1.98   48.49    0.04
 10    1400         292.79  171253.21    3.81    1.98   48.49    0.04
 12    1600         406.76  144448.71    3.81    1.98   48.49    0.04
 13    1800        1336.08  111682.80    0.00    0.00    0.00    0.00

And it makes me think, that problem might be not in the embeddings per se, but in embeddings model (Roberta VS xlm-roberta)

adrianeboyd Sep 6, 2021

I'll add that we made some guesses about models to recommend for languages where we weren't as familiar with the available models based what looked most popular in the huggingface model repo.

We tested that our recommendations could be loaded and that training ran without crashing, but we didn't evaluate the results beyond that. If you know of a better recommendation, we're happy to hear about it.

dchaplinsky · 2021-09-01T10:37:35Z

dchaplinsky
Sep 1, 2021
Author

Just in case, my notebook:
https://colab.research.google.com/drive/11kDjB3pCfpCOYEUDIIlfSH43VrXrb1P7?usp=sharing

And my config:
https://drive.google.com/file/d/15jvgxWaJ-2jsTD50QLmCV6B7R7L3nqzy/view?usp=sharing

0 replies

dchaplinsky · 2023-02-17T19:08:20Z

dchaplinsky
Feb 17, 2023
Author

I'm glad to let you know that somehow the problem disappeared.
I ran it today with my old configs using new version of spacy and spacy-transformers and voila:

ℹ Initial learn rate: 1e-05
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  -------------  --------  ------  ------  ------  ------
  0       0         898.86    701.86    1.00    0.52   11.43    0.01

  1     200       12545.41  13049.79   75.57   79.44   72.05    0.76
  2     400        2371.97   2280.48   82.97   83.34   82.60    0.83
  4     600        1049.16   1168.24   84.29   85.43   83.18    0.84
  5     800         507.07    606.52   85.73   86.99   84.50    0.86

1 reply

svlandeg Feb 20, 2023
Maintainer

All's well that ends well! ;-)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LOSS is skyrocketing when trying to train a NER classifier with transformers #9090

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 10 comments 17 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

LOSS is skyrocketing when trying to train a NER classifier with transformers #9090

Replies: 10 comments · 17 replies

dchaplinsky Aug 30, 2021 Author

dchaplinsky Aug 31, 2021 Author

dchaplinsky Aug 31, 2021 Author

svlandeg Sep 2, 2021 Maintainer

dchaplinsky Sep 2, 2021 Author

dchaplinsky Aug 31, 2021 Author

dchaplinsky Sep 1, 2021 Author

dchaplinsky Sep 1, 2021 Author

dchaplinsky Sep 1, 2021 Author

dchaplinsky Sep 1, 2021 Author

dchaplinsky Sep 4, 2021 Author

dchaplinsky Sep 5, 2021 Author

dchaplinsky Sep 5, 2021 Author

dchaplinsky Sep 1, 2021 Author

dchaplinsky Feb 17, 2023 Author

svlandeg Feb 20, 2023 Maintainer

Replies: 10 comments 17 replies

dchaplinsky
Aug 30, 2021
Author

dchaplinsky Aug 31, 2021
Author

dchaplinsky
Aug 31, 2021
Author

svlandeg Sep 2, 2021
Maintainer

dchaplinsky Sep 2, 2021
Author

dchaplinsky
Aug 31, 2021
Author

dchaplinsky
Sep 1, 2021
Author

dchaplinsky
Sep 1, 2021
Author

dchaplinsky
Sep 1, 2021
Author

dchaplinsky
Sep 1, 2021
Author

dchaplinsky Sep 4, 2021
Author

dchaplinsky Sep 5, 2021
Author

dchaplinsky Sep 5, 2021
Author

dchaplinsky
Sep 1, 2021
Author

dchaplinsky
Feb 17, 2023
Author

svlandeg Feb 20, 2023
Maintainer