Feature Request for TextCategorizer: add numeric features for text classification

[mhigginslp]

I have extra, numeric data that comes along with the text that I am trying to classify. I suspect that these features will help the model learn.
I also have sentence/paragraph level embeddings that would help the classifier out.
I imagine this situation comes up fairly frequently for text classification - there is data that comes along with the text that may be able to help classification.

Your Environment

• Operating System:
• Python Version Used:
• spaCy Version Used:
• Environment Information:

[vtsimbaluk]

Looking for a similar feature. I have a text and some additional numerical fields that should help in classification.

[NixBiks]

Until it'll be possible for spaCy to incorporate additional features for the classifier I have a quick question for the explosion team, which I assume can be useful for followers of this thread.

Imagine I have some text that I want to classify - I have some really useful metadata as well that should be used as additional features for the classifier.

Would it make sense to take my features and combine with nlp(text).vector as my feature space and then run a standard classifier (xgboost, log-linear, ...)? Is that the right use of Doc.vector?

The custom similarity section might be useful as well.

[ClaudMor]

Regarding @mr-bjerre quick question (in case he hasn't found an answer yet) I just dealt with a task which maybe fits his imaginary scenario.
In a confidential dataset, I had to classify some texts in two exclusive classes, and each text had some metadata which consisted of more strings, regarding the topics the text was about.
TLDR: if you plan to classify text only using spaCy vectors, then I had a better score (roc_auc) on validation using vectors coming from text with metadata. But if you wish to do something that (hopefully) performs better, you may try the following.

This is the way I actually dealt with it:
I had a train and a test set.
I converted and saved train texts in vectors with nlp(text).vector in two ways:

1. after doing " ".join(text, metadata) for each text ( which produced the vectorized file that we will call vectorization1).
2. without doing it (vectorization2).

Then the same has been done with the test set.

In both cases, I used en_core_web_lg (is there any better choice?).
Then I trained a scikit MLP (that was later saved as MLP1) on 80% of vectorization1, and another MLP on the corresponding 80% vectorization2 (MLP2). Then I let both independently predict the remaining 20% of train and all of test.
I also trained two ensemble textcats, one on the corresponding texts of the 80% of vectorization1 (t_textcat1) and the other on the corresponding 80% of vectorization2 (t_textcat2). Then the two texcats were used to independently predict the remaining 20% of train and all of test.
Moreover, I trained another ensemble textcat on 80% of metadata only (we will call this textcat m_textcat), and let it predict the remaining 20% of train and all pf test.

Finally, those predictions on the 20% of train were used to train a scikit logistic regression (also tried a scikit MLP and a scikit SVM with poorer validation performance).
So in this context i had the chance to train the logistic regression using various mixes of the predictions I had before. It turned out that the best one was obtained using the predictions of MLP1, t_textcat2 and m_textcat. All other combinations/ intermediate models performed worse on test.

(You may find a more detailed explanation of the procedure in this Stack Overflow issue I wrote)

Hope it helps.

Anyway, I feel this is a very artificial and "third party" way to deal with metadata.
Is there any update about this thread?

[svlandeg]

While I realise this is quite an old thread, I just wanted to point out that with the upcoming v3 version of spaCy, it should become much easier to customize these kind of things. You can find more information about the current Release Candidate here: https://nightly.spacy.io/

Another interesting recent discussion about using custom features (in v3) is here: #6527, specifically see this comment by @bratao!

For more details on customizing model architectures, see here: https://nightly.spacy.io/usage/layers-architectures