RADAR Pipeline

An open-source python feature generation and visualization package use with RADAR project data.

---

Wiki

Please visit the RADAR Pipeline Wiki to learn more about RADAR Pipeline. Also see the RADAR-base Analytics Catalogue for available pipelines for processing RADAR-base data.

Wiki resources:

• Home
• Installation
• Contributor Guide
• Mock Pipeline
• Configuration
• Data Ingestion
• Setup
• Pipeline Core Topics
• Creating Citable Analytics Pipelines

How to run

Note

If you are using Windows, please install Spark and set environment variables as mentioned here before going through the installation below. You'll need to set the environment variables given here.

• Clone the repository (with all the submodules):

  $ git clone --recurse-submodules https://github.com/RADAR-base/radarpipeline.git

• Change the directory to radarpipeline:

  $ cd radarpipeline

• Checkout the development branch:

  $ git checkout dev

• Create a virtual environment and activate it. The instructions here use virtualenv but feel free to use any python virtual environment manager of your choice.

  • Install the virtualenv package:

    $ python -m pip install --user virtualenv

  • Create a python virtual environment:

    $ python -m virtualenv env

  • Activate the virtual environment:

    On Windows, run:

    $ .\env\Scripts\activate

    On Linux or MacOS, run:

    $ source ./env/bin/activate

• Install the dependencies:

  $ python -m pip install -r requirements.txt

• Install the module as a python package by running the command

  $ python -m pip install -e .

• To verify the installations, run the following command in the project root directory to run the pipeline:

  $ python .

• The pipeline would do a mock run and ingest the data in the mock-data directory. You can see some outputs in the CLI and if the project is installed correctly, the mock pipeline would run without errors and save the data to the output directory.

License

This project is licensed under the Apache License, Version 2.0.

Citation & Acknowledgment

Please use citation or or see CITATION.cff

Pushkar patel has done a great amount of work under Google Summer of Code 2022. His work report can be found here. We would like to thank Pushkar for all his contribution and GSoC for giving us this opportunity.