Apache Amaterasu

                                           /\
                                          /  \ /\
                                         / /\ /  \
    _                 _                 / /  / /\ \   
   /_\   _ __   __ _ | |_  ___  _ _  __(_( _(_(_ )_) 
  / _ \ | '  \ / _` ||  _|/ -_)| '_|/ _` |(_-<| || |
 /_/ \_\|_|_|_|\__,_| \__|\___||_|  \__,_|/__/ \_,_|

Apache Amaterasu is an open-source, deployment tool for data pipelines. Amaterasu allows developers to write and easily deploy data pipelines, and clusters manage their configuration and dependencies.

Download

For this preview version, we have packaged amaterasu nicely for you to just download and extract. Once you do that, you are just a couple of easy steps away from running your first job.

Creating a dev/test Mesos cluster

We have also created a Mesos cluster you can use to test Amaterasu or use for development purposes. For more details, visit the amaterasu-vagrant repo

Configuration

Configuring amaterasu is very simple. Before running amaterasu, open the amaterasu.properties file in the top-level amaterasu directory, and verify the following properties:

property	Description	Default value
zk	The ZooKeeper connection string to be used by amaterasu	192.168.33.11
master	The clusters' Mesos master	192.168.33.11
user	The user that will be used to run amaterasu	root

Running a Job

To run an amaterasu job, run the following command in the top-level amaterasu directory:

ama-start.sh --repo="https://github.com/shintoio/amaterasu-job-sample.git" --branch="master" --env="test" --report="code" 

We recommend you either fork or clone the job sample repo and use that as a starting point for creating your first job.

Apache Amaterasu Developers Information

Building Apache Amaterasu

Instructions to download gradle wrapper when building from source distribution

As part of the ASFs policy, when building Apache Amaterasu from a source distribution and not from git, the gradle-wrapper.jar needs to be downloaded. The following command, will download the gradle wrapper from Amaterasu git repository and puts under incubator-amaterasu/gradle/wrapper

wget --no-check-certificate -P incubator-amaterasu/gradle/wrapper https://github.com/apache/incubator-amaterasu/raw/version-0.2.0-incubating-rc4/gradle/wrapper/gradle-wrapper.jar (or) curl --insecure -L https://github.com/apache/incubator-amaterasu/raw/version-0.2.0-incubating-rc4/gradle/wrapper/gradle-wrapper.jar > incubator-amaterasu/gradle/wrapper/gradle-wrapper.jar

Building Amaterasu with Gradle

to build the amaterasu home dir (for dev purposes) run:

./gradlew buildHomeDir test

to create a distributable jar (clean creates the home dir first) run:

./gradlew buildDistribution test

Architecture

Amaterasu is an Apache Mesos framework with two levels of schedulers:

• The ClusterScheduler manages the execution of all the jobs
• The JobScheduler manages the flow of a job

The main clases in Amateraso are listed bellow:

+-------------------------+   +------------------------+
| ClusterScheduler        |   | Kami                   |
|                         |-->|                        |
| Manage jobs:            |   | Manages the jobs queue |
| Queue new jobs          |   | and Amaterasu cluster  |
| Reload interrupted jobs |   +------------------------+
| Monitor cluster state   |
+-------------------------+
            |
            |     +------------------------+
            |     | JobExecutor            |
            |     |                        |
            +---->| Runs the Job Scheduler |
                  | Communicates with the  |
                  | ClusterScheduler       |
                  +------------------------+
                             |
                             |
                  +------------------------+      +---------------------------+                      
                  | JobScheduler           |      | JobParser                 |
                  |                        |      |                           |
                  | Manages the execution  |----->| Parses the kami.yaml file |
                  | of the job, by getting |      | and create a JobManager   |
                  | the  execution flow    |      +---------------------------+
                  | fron the JobManager    |                    |
                  | and comunicating with  |      +---------------------------+
                  | Mesos                  |      | JobManager                |                      
                  +------------------------+      |                           |
                             |                    | Manages the jobs workflow |
                             |                    | independently of mesos    |
                  +------------------------+      +---------------------------+
                  | ActionExecutor         |
                  |                        |
                  | Executes ActionRunners |
                  | and manages state for  |
                  | the executor           |
                  +------------------------+