Classification_report is going really slow for mode='strict'

[eloukas]

I have a dummy dataset in my local machine.
While my sklearn token-level evaluation (strict mode on/off) and my seqeval entity-level evaluation (strict mode off) run all together in 5 seconds, for some reason the seqeval entity-level evaluation with arg mode='strict' takes around 70 seconds, which is too much.

Is there any way to speed it up somehow? Maybe the code needs to get more optimized?

I can't run experiments with more data on my AWS machine using mode='strict'.
The evaluation on mode='strict' takes more time than the training of the neural models.

Many thanks!

• Operating System: Ubuntu 18 (LTS)
• Python Version: 3.8
• Package Version: 1.1.0

[Hironsan]

Profile:

1. unique_labels
2. extended_tokens
3. is_valid
4. Enum is slow

[eloukas]

What do you mean? Do you want me to report you anything from my current program?
Thank you again.

[Hironsan]

I performed the profiling and found it takes a long time to execute the above.
I'm thinking about how to solve the problem now.

By the way, what the number of samples did you try to evaluate?

[eloukas]

Oh, ok, cool!
In my dummy dataset, the numbers of gold/predicted tokens are 8721.
(I have them all in a list of a single list.)

In my big dataset in the cloud, I probably have one gazzilion millions of these :)