Merge pull request #87 from BodenmillerGroup/development

v3

.github/workflows/docs.yml

@@ -0,0 +1,18 @@
+on: 
+  push: 
+    branches: [main] 
+  pull_request:
+    branches: [main]
+
+name: docs
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.x
+      - run: pip install mkdocs-material
+      - run: mkdocs gh-deploy --force

.gitignore

@@ -0,0 +1,159 @@
+*.ilp
+*.DS_Store
+/.vscode/
+/analysis/
+/raw/
+sample_metadata.xlsx
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "ImcPluginsCP"]
+	path = resources/ImcPluginsCP
+	url = https://github.com/BodenmillerGroup/ImcPluginsCP.git
+	branch = master

CHANGELOG.md

@@ -0,0 +1,32 @@
+# Changelog
+
+## [3.0]
+
+ - replaced `imctools` by internal `imcsegpipe` package calling [readimc](https://github.com/BodenmillerGroup/readimc)
+ - adjusted the pre-processing script in terms of function calls and documentation
+ - added script to download raw IMC data from [zenodo.org/record/5949116](https://zenodo.org/record/5949116) and the ilastik pixel classifier from [zenodo.org/record/6043544](https://zenodo.org/record/6043544)
+ - moved the hot pixel filtering step from the CellProfiler pipelines into the `create_analysis_stacks` function call
+ - removed the default `tiffs` folder. All files related to ilastik pixel classification are stored in the `ilastik` folder and image crops are stored in `crops`
+ - hot pixel filtered images are directly written out to `cpout/images`
+ - segmentation masks are directly written out to `cpout/masks` in the second pipeline and read in as objects in the last pipeline
+ - pixel probabilities are downscaled in the second pipeline and directly written into `cpout/probabilites`
+ - cell segmentation is performed on downscaled pixel probabilities
+
+## [2.3]
+
+ - Bugfixes: `1_prepare_ilastik`: Removed special characters from pipeline comments as this caused encoding issues.
+
+## [2.1]
+
+ - Bugfixes: `1_prepare_ilastik`: Fix range to 0-1 for mean image, preventing out of range errors
+
+## [2.0]
+
+ - Change to imctools v2: Changes the structure of the folder to the new format, changing the naming of the .ome.tiff files
+ - Change to Cellprofiler v4: Requires the use of the ImcPluginsCP master branch or a release > v.4.1
+ - Updated documentation
+ - Adds var_Cells.csv containing metadata for the measurements
+ - Adds panel to cpout folder
+
+
+

README.md

@@ -1,68 +1,101 @@
 [![DOI](https://zenodo.org/badge/103582813.svg)](https://zenodo.org/badge/latestdoi/103582813)
-# A flexible  image segmentation pipeline for heterogeneous multiplexed tissue images based on pixel classification
+# A flexible multiplexed image segmentation pipeline based on pixel classification
 
 ## Introduction
-The pipeline is based on CellProfiler (http://cellprofiler.org/, tested v4.0.6) for segmentation and Ilastik
-(http://ilastik.org/, tested v1.3.5)
-for pixel classification. It is streamlined by using the specially developed imctools python
-package (https://github.com/BodenmillerGroup/imctools, >v2.1) as well as custom CellProfiler modules 
-(https://github.com/BodenmillerGroup/ImcPluginsCP, release v4.2.1).
 
-This repository showcases the basis of the workflow with step-by-step instructions. To run this more automatized, we recommend
-our Snakemake implementation: https://github.com/BodenmillerGroup/ImcSegmentationSnakemake
+The pipeline is based on [CellProfiler](http://cellprofiler.org/) (tested v4.2.1) for segmentation and [Ilastik](http://ilastik.org/) (tested v1.3.3post3) for pixel classification. 
+It is streamlined by using the `imcsegpipe` python package available via this repository as well as custom CellProfiler modules ([ImcPluginsCP](https://github.com/BodenmillerGroup/ImcPluginsCP), release v4.2.1).
 
-This pipeline was developed in the Bodenmiller laboratory at the University of Zurich (http://www.bodenmillerlab.org/)
-to segment hundreds of highly multiplexed imaging mass cytometry (IMC) images.
-However we think that this concept also works well for other multiplexed imaging modalities..
+This repository showcases the basis of the workflow with step-by-step instructions. 
+As an alternative and dockerized version of the pipeline, check out [steinbock](https://github.com/BodenmillerGroup/steinbock).
 
-The document to start can be found at 'scripts/imc_preprocessing.ipynb'
-(https://nbviewer.jupyter.org/github/BodenmillerGroup/ImcSegmentationPipeline/blob/development/scripts/imc_preprocessing.ipynb).
-The pdf found in 'Documentation/201709_imctools_guide.pdf' is still conceptually valid, however the installation
-procedures described are outdated. Please follow the instructions in the imc_preprocessing.ipynb document!
+This pipeline was developed in the Bodenmiller laboratory at the University of Zurich ([www.bodenmillerlab.com](https://www.bodenmillerlab.com/)) to segment hundreds of highly multiplexed imaging mass cytometry (IMC) images.
+The concepts applied here to IMC data can also be transfered to data generated by other highly multiplexed imaging modalities.
+
+For a general overview on IMC as technology and data processing tasks, please refer to [bodenmillergroup.github.io/IMCWorkflow](https://bodenmillergroup.github.io/IMCWorkflow/).
+
+## Usage
+
+For the main part of the analysis, you will need to install [Ilastik](https://www.ilastik.org/download.html) and [CellProfiler](https://cellprofiler.org/releases).
+
+Before being able to pre-process the data, you will need to setup the environment:
+
+1. [Install conda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/)
+
+2. Clone the repository: 
+
+```bash
+git clone --recursive https://github.com/BodenmillerGroup/ImcSegmentationPipeline.git
+```
+
+3. Setup the conda environment: 
+
+```bash
+cd ImcSegmentationPipeline
+conda env create -f environment.yml
+```
+
+4. Configure CellProfiler to use the plugins by opening the CellProfiler GUI, selecting `Preferences` and setting the `CellProfiler plugins directory` to `path/to/ImcSegmentationPipeline/resources/ImcPluginsCP/plugins`. Alternatively you can clone the `ImcPluginsCP` repository individually and set the path correctly in CellProfiler.
+
+5. Activate the environment created in 3. and start a jupyter instance
+
+```bash
+conda activate imcsegpipe
+jupyter-lab
+```
+
+This will automatically open a jupyter instance at `http://localhost:8888/lab` in your browser.
+From there, you can open the `scripts/imc_preprocessing.ipynb` notebook and start the data pre-processing.
+
+In brief, the main analysis steps include:
+
+1. Pre-processing of the raw images to create `.ome.tiffs` and `.tiff` stacks for ilastik training and measurement (python).   
+2. Ilastik pixel classification based on random crops of the images (CellProfiler, Ilastik).  
+3. Image segmentation based on the classification probabilities (CellProfiler).  
+4. Measurement and export of cell-specific features, such as marker expression (CellProfiler).  
+
+## Example data
+
+To test these pipelines on example data, please run the `scripts/download_examples.ipynb` script.
+
+## Documentation
+
+For a more detailed overview on the individual analysis steps, please visit [https://bodenmillergroup.github.io/ImcSegmentationPipeline/](https://bodenmillergroup.github.io/ImcSegmentationPipeline/).
 
 This pipeline was presented at the 2019 Imaging Mass Cytometry User Group Meeting.
 [The slides can be downloaded here](https://drive.google.com/file/d/1ajPzlJ2CUj6sFYSOq0HR2dOJehHIlCJt/view).
 The slides briefly explain why we chose this approach to image segmentation and provide help to run the pipeline.
 
-We freely share this pipeline in the hope that it will be useful for others to perform high quality image segmentation
-and serve as a basis to develop more complicated open source IMC image processin17g workflows. In return we would like
-you to be considerate and give us and others feedback if you find a bug or issue and raise a GitHub Issue
-on the affected projects or on this page.
-
 ## Changelog
-- v2.3:
-    - Bugfixes:
-        - 1_prepare_ilastik: Removed special characters from pipeline comments as this caused encoding issues.
-
-- v2.1:
-    - Bugfixes:
-        - 1_prepare_ilastik: Fix range to 0-1 for mean image, preventing out of range errors
-
-- v2.0:
-    - Change to imctools v2:
-        - Changes the structure of the folder to the new format, changing the
-          naming of the .ome.tiff files => If you use this pipeline you need to re-generate the OME tiff
-
-    - Change to Cellprofiler v4:
-        - Requires the use of the ImcPluginsCP master branch or a release > v.4.1
+
+For changes in specific releases, please refer to the [CHANGELOG](CHANGELOG.md).
 
-    - Varia:
-        - Updated documentation
-        - Adds var_Cells.csv containing metadata for the measurements
-        - Adds panel to cpout folder
+## License
+
+We [freely share](LICENSE) this pipeline in the hope that it will be useful for others to perform high quality image segmentation and serve as a basis to develop more complicated open source IMC image processing workflows. 
+In return we would like you to be considerate and give us and others feedback if you find a bug/issue and [raise a GitHub Issue](https://github.com/BodenmillerGroup/ImcSegmentationPipeline/issues) on the affected projects or on this page.
+
+## Contributing
+
+To contribute to this work, please fork the repository, make changes to it and open a pull request.
 
+## Contributors
+
+**Creator:** Vito Zanotelli  
+**Contributor:** Jonas Windhager, Nils Eling  
+**Maintainer:** Nils Eling  
 
 ## Citation
-d
+
 If you use this workflow for your research, please cite us:
+
 ```
 @misc{ImcSegmentationPipeline,
     author       = {Vito RT Zanotelli, Bernd Bodenmiller},
-    title        = {{ImcSegmentationPipeline: A pixelclassification based multiplexed image segmentation pipeline}},
-    month        = Sept,
-    year         = 2017,
+    title        = {{ImcSegmentationPipeline: A pixel-classification based multiplexed image segmentation pipeline}},
+    year         = 2022,
     doi          = {10.5281/zenodo.3841961},
-    version      = {0.9},
+    version      = {3.0},
     publisher    = {Zenodo},
     url          = {https://doi.org/10.5281/zenodo.3841961}
     }