Using SpaCy's Name Entity Recognition for large scale

[kostaDimitrijevic]

Hello my project wants to use SpaCy's NER model for processing millions of text documents. I'm using Databricks, PySpark and I have a computational power needed, but never the less with the current code I'm unable to run this on large scale. I am using all of the Sparks benefits but It always ends with Out Of Memory errors even when I'm using Nodes with huge amount of Memory. So that leads me to believe that my code isn't well written for large scale processing.

I can see that in your documentation your mentioning these potential solutions:

1. This will disable other models and use only NER, which is good in my case because I only need that:

import spacy

texts = [
    "Net income was $9.4 million compared to the prior year of $2.7 million.",
    "Revenue exceeded twelve billion dollars, with a loss of $1b.",
]

nlp = spacy.load("en_core_web_sm")
for doc in nlp.pipe(texts, disable=["tok2vec", "tagger", "parser", "attribute_ruler", "lemmatizer"]):
    # Do something with the doc here
    print([(ent.text, ent.label_) for ent in doc.ents])

2. You have multiprocess ability (I'm not using GPU nodes):

# Multiprocessing with 4 processes
docs = nlp.pipe(texts, n_process=4)

# With as many processes as CPUs (use with caution!)
docs = nlp.pipe(texts, n_process=-1)

3. You mention also batch size

Are these solutions something I should try to implement? Can you give me some advices and tips for my use case. What is the best approach. Can this be used with PySpark, etc.