Benchmarking T cell receptor - Epitope Predictors with ePytope

This repository contains the code to recreate the study "Benchmarking of T-Cell Receptor-Epitope Predictors with ePytope-TCR" by Drost et al.

Install

Stand-alone Benchmark-Suite:

pip install -e .

Benchmark-Suite with ePytope:

Note: you will need to install the individual predictors, separately. You may consult the dockerfile in ./docker/Dockerfile

pip install -e .[epytope]

Reproducibility: To reproduce the environment of the paper. To reproduce the specific environment of the predictors, we advise to use the dockerfile in ./docker/Dockerfile

pip install -e .[epytope]

Reproduce Benchmark

To setup the a docker container with all tools:

git clone https://github.com/SchubertLab/benchmark_TCRprediction.git
cd benchmark_TCRprediction
sudo docker build -t img_benchmark -f ./docker/Dockerfile ..
sudo docker run --gpus all -d --name ctr_benchmark -p 8001 img_benchmark
sudo docker exec -it ctr_benchmark /bin/bash

Within the docker container, you can reproduce the benchmark via

conda activate epytope_base
./scripts/run_viral.sh
./scripts/run_mutations.sh

The results can be found in ./results/* and ./tcr_benchmark/results/*. Please note, that despite best efforts, not all tools are fully reproducible.

Test own Method

To test your own method, you can use the programmatic interface:

from tcr_benchmark.pp.datasets import download_datasets
from tcr_benchmark.study.benchmark import evaluate_predictor

download_datasets("all")
results = evaluate_predictor(prediction_func, predictor_name, datasets, config)

• prediction_func: python function that obeys the following interface
  • input: a pandas data frame of the columns ['', '', ...] #todo
  • output: the input pandas dataframe with the additional columns <predictor_name> containing binding scores with higher scores representing higher binding probabilities
• predictor_name: str, name of your predictor
• datasets:
  • Viral dataset: 'viral'
  • Mutational dataset: 'mutation'
  • Both datasets: 'all'
• configs: kwargs that will forwarded to your prediction function

Cite

When you reference the benchmark or use the provided utilities to evaluate your method please cite the benchmarking paper and the corresponding datasets:

@article{drost2024benchmarking,
  title={Benchmarking of T-Cell Receptor-Epitope Predictors with ePytope-TCR},
  author={Drost, Felix and Chernysheva, Anna and Albahah, Mahmoud and Kocher, Katharina and Schober, Kilian and Schubert, Benjamin},
  journal={bioRxiv},
  pages={2024--11},
  year={2024},
  publisher={Cold Spring Harbor Laboratory}
}

Viral Dataset

@article{kocher2024quality,
  title={Quality of vaccination-induced T cell responses is conveyed by polyclonality and high, but not maximum, antigen receptor avidity},
  author={Kocher, Katharina and Drost, Felix and Tesfaye, Abel Mekonnen and Moosmann, Carolin and Schuelein, Christine and Grotz, Myriam and D'Ippolito, Elvira and Graw, Frederik and Spriewald, Bernd and Busch, Dirk H and others},
  journal={bioRxiv},
  pages={2024--10},
  year={2024},
  publisher={Cold Spring Harbor Laboratory}
}

@article{adams2023integrated,
  title={An integrated reagent and multimodal analysis workflow to enrich and characterize peptide-specific CD8+ T cells},
  author={Adams, Bruce A and Shahi, Payam and Reyes, Daniel and Maheshwari, Shamoni and Mousavi, Nima and Krishnan, Sreenath and Ramen, Nandhini and Tsai, FuNien and Kumar, Poornasree and Finnegan, Peter and others},
  journal={The Journal of Immunology},
  volume={210},
  number={1\_Supplement},
  pages={249--17},
  year={2023},
  publisher={American Association of Immunologists}
}

Mutation-Dataset

@article{drost2024predicting,
  title={Predicting T cell receptor functionality against mutant epitopes},
  author={Drost, Felix and Dorigatti, Emilio and Straub, Adrian and Hilgendorf, Philipp and Wagner, Karolin I and Heyer, Kersten and Montes, Marta L{\'o}pez and Bischl, Bernd and Busch, Dirk H and Schober, Kilian and Schubert, Benjamin},
  journal={Cell Genomics},
  volume={4},
  number={9},
  year={2024},
  publisher={Elsevier}
}