How to use a BiLSTM with a Transformer model?

[ALEKS0805]

Is it possible to implement BiLSTM encoder in HuggingFace transformers model ?
I know that transformer model like BERT comes with both the "embed" and "encode" steps but is it possible to include BILSTM encoder?

[polm]

Can you clarify what you want the BiLSTM to do? Do you want to put one on top of the Transformer embeddings as a final layer or something? You'd need to write a custom component to do that, which is possible but involved.

Can you clarify why you want to do this?

[ALEKS0805]

yes, that was the idea. To put one on top of the Transformer embeddings as a final layer. I am dealing with legal documents, and used legal Bert for this with good results, but wanted to experiment with BILSTM and see how that goes.

[polm]

I see. spaCy is not really designed to allow that kind of experimentation at the architecture level as a first priority - it's more focused on presenting known good options and letting you choose between them. If you want to add a layer here, a convolution there, and so on, you're probably better off trying something with Thinc or PyTorch or something.

If you want to do something like this the Thinc tutorial about using Transformers might be a good starting point.

Another thing you could do is design your custom network in HuggingFace directly and then just load it with spacy-transformers, like any other Transformer model.