Computing Embeddings from Transformer?

[nickbhat]

Hello,

Is there any ability to create an API for producing embeddings from input sequences? Users could implement this themselves if #19 turns out to be true. If not, perhaps an embedding API could be exposed without having to make weights publicly available?

Thanks!

[prihoda]

Hi @nickbhat there's a function intended for this:

BioPhi/biophi/humanization/methods/sapiens/predict.py

Line 36 in f764b61

def sapiens_predict_seq(seq, chain_type, model_version='latest', return_all_hiddens=False):

However, looks like the return_all_hiddens arg is actually ignored :) Do you want to create a pull request by any chance? I'm always happy to get more contributors.

It would be a simple fix here:

BioPhi/biophi/humanization/methods/sapiens/roberta.py

Line 76 in f764b61

return pred

Btw, this way we can only get the embeddings, if you also wanted the attention weights, that would require some changes to fairseq code.

[nickbhat]

I'll take a look at this and submit a PR! Always happy to contribute :) I'll get around to it in a couple of weeks, if that's okay.

[prihoda]

Fixed by #21

Example usage:

from biophi.humanization.methods.sapiens.predict import sapiens_predict_seq

pred, extra = sapiens_predict_seq(
    seq=seq, # seq should be the variable region sequence only
    chain_type='H', # chain type is H or L
    return_all_hiddens=True
)

embeddings_per_layer = extra['inner_states']