Support multiple epitope prediction scores per tool

[lkuchenb]

Most tools report multiple scores for each prediction. Currently the framework interfaces are affinity-centric, which is actually discouraged by most tool developers.

The epitope prediction method interface should support multiple scores and all scores produced by each method should be extracted from their outputs.

[b-schubert]

agreed! As disucussed, we should probably rethink how to store results etc and perhaps use something similar to anndata (https://anndata.readthedocs.io/en/latest/) to store results?

[christopher-mohr]

agreed! As disucussed, we should probably rethink how to store results etc and perhaps use something similar to anndata (https://anndata.readthedocs.io/en/latest/) to store results?

Hi @b-schubert, did you use the anndata package before? Do you think it would be a good fit for this feature? I think especially the rank-based prediction would be a really important thing to add (also based on feature requests I get).

[lkuchenb]

If we want to stick to the current pandas dataframe layout as a return value for predictions we could also realize multiple scores with a multi-layer index on the columns, for example

Allele→      A*01:01-----------------
Method→      NetMHC4------- SYFPEITHI
ScoreType→   PercRank- Aff- SyfScore-
↓Peptide

SYFPEITHI    0.001     123     0.123
EITHYSYFP    0.9       12312   0.904
ITYPEISYF    ...       ...     ...

[b-schubert]

@christopher-mohr

Hi @b-schubert, did you use the anndata package before? Do you think it would be a good fit for this feature? I think especially the rank-based prediction would be a really important thing to add (also based on feature requests I get).

anndata is essential a Matrix format with additional pandas data frames on the rows columns. I guess it would be quite similar to an additional multi-index dimension?! But would give us perhaps a better way to store the peptide and HLA objects (instead of the index and column keys)....

Opinions?

[christopher-mohr]

Hard to say without any detailed knowledge about anndata. Currently, I would also vote for sticking to pandas as @lkuchenb suggested.

[jonasscheid]

Hey! I would like to help implementing this functionality :)

If the EpitopePredictionResult class in Result.py should be changed from

Peptide Obj	Method Name	Allele1 Obj	Allele2 Obj
Peptide1	Method 1	0.324	0.56
	Method 2	20	15
Peptide2	Method 1	0.50	0.36.
	Method 2	26	10

to something like

Allele	A*01:01
Method	NetMHC4		SYFPEITHI
ScoreType	PercRank	Aff	SyfScore
Peptide
SYFPEITHI	0.001	123	0.123
EITHYSYFP	0.9	12312	0.904
ITYPEISYF	...	...	...

using multi-index on columns I need to change the functions filter_result and merge_results in the EpitopePredictionResult class right?

Additionally parse_external_result in the predict function of External.py needs to be changed such that it returns a dictionary with separated metrics. I would suggest to change the output of that function (for all classes) from

defaultdict(dict, {'HLA-Allele1': {'Pep1': -999, 'Pep2': 500}})

to

defaultdict(dict, {'HLA-Allele1': {'Rank':{'Pep1': 15.0, 'Pep2': 16.0}, 'Affinity'{'Pep1': -999, 'Pep2': 500}}, HLA-Allele2{...})

Also the current multi-indexing in the predict function needs to be changed then.

What are your opinions on that? Would that be sufficient? Did I miss anything? Please let me know!

Best,
Jonas

[christopher-mohr]

Hi @jonasscheid, thanks for starting to work on this! In my opinion, this is already a good starting point to carry forward this discussion. I think it's also important to check what is the impact of changes to EpitopePredictionResult.

What do you think @lkuchenb and @b-schubert? Should we proceed with this?

[b-schubert]

It will also affect all downstream methods (e.g., vaccine design frameworks)

[b-schubert]

• Multiindexing in all results classes
• Introduce an Enum for Rank and Score index
• Extend OptiTope, EpitopeAssemlby specifying wich score type to use and in which direction to optimize; Implement internal routine to transformed scores accordingly if necessary (to not changes the optimization problems)

[jonasscheid]

• Multiindexing in all results classes

Hey!
I went through the CleavageAndTAPPrediction tutorial and the data structures in Results.py and recognized, that there might actually not be a need to change the remaining result classes towards more layers of multi-indices. I can e.g. do Chapter 3: Consensus prediction for natural ligand prediction in the tutorial only with the changes in EpitopePredictionResult and obtain meaningful results.

I also have no meaningful idea on adding another layer of multi-indices to the remaining results classes. Maybe you have one. Would be nice to have some feedback!

Thanks in advance!

[christopher-mohr]

@b-schubert, @lkuchenb what's your opinion on this? I guess one reason for changing it would be to have a consistent structure of result objects.

@jonasscheid I think in the case of the `CleavageSitePrediction, the "problem " is that you don't have any allele-dependency right?

[jonasscheid]

Exactly @christopher-mohr. This is also the case for CleavageFragmentPredictionResult, TAPPredictionResult. Would not make sense biologically as well..

Additionally for CleavageSitePredictionResult, CleavageFragmentPredictionResult, TAPPredictionResult (all remaining classes in Results.py) there is no predictor, as far as I investigated, which has multiple scores as output. So no need for multi-indices in that regard as well.

[b-schubert]

Those prediction tools indeed do not offer multiple scores, but I was thinking of future-proofing them given that we could also calculate rank-based scores for them at some point.

[christopher-mohr]

Solved by #42.