- {% block content %}
- {{ page.content }}
- {% endblock %}
-
- 
+
+
+ {%- endblock %}
\ No newline at end of file
diff --git a/docs/docs/apispec.md b/docs/docs/apispec.md
index c00142043d..94df57fcaf 100644
--- a/docs/docs/apispec.md
+++ b/docs/docs/apispec.md
@@ -1,21 +1,23 @@
+# API Specification
+
## Task & Workflow Metadata
| Endpoint | Description | Input |
|------------------------------------------|:---------------------------------|-------------------------------------------------------------|
| `GET /metadata/taskdefs` | Get all the task definitions | n/a |
| `GET /metadata/taskdefs/{taskType}` | Retrieve task definition | Task Name |
-| `POST /metadata/taskdefs` | Register new task definitions | List of [Task Definitions](../configuration/taskdef) |
-| `PUT /metadata/taskdefs` | Update a task definition | A [Task Definition](../configuration/taskdef) |
+| `POST /metadata/taskdefs` | Register new task definitions | List of [Task Definitions](/configuration/taskdef.html) |
+| `PUT /metadata/taskdefs` | Update a task definition | A [Task Definition](/configuration/taskdef.html) |
| `DELETE /metadata/taskdefs/{taskType}` | Delete a task definition | Task Name |
|||
| `GET /metadata/workflow` | Get all the workflow definitions | n/a |
-| `POST /metadata/workflow` | Register new workflow | [Workflow Definition](../configuration/workflowdef) |
-| `PUT /metadata/workflow` | Register/Update new workflows | List of [Workflow Definition](../configuration/workflowdef) |
+| `POST /metadata/workflow` | Register new workflow | [Workflow Definition](/configuration/workflowdef.html) |
+| `PUT /metadata/workflow` | Register/Update new workflows | List of [Workflow Definition](/configuration/workflowdef.html) |
| `GET /metadata/workflow/{name}?version=` | Get the workflow definitions | workflow name, version (optional) |
|||
## Start A Workflow
### With Input only
-See [Start Workflow Request](../gettingstarted/startworkflow/#start-workflow-request).
+See [Start Workflow Request](/gettingstarted/startworkflow.html).
#### Output
Id of the workflow (GUID)
@@ -142,8 +144,9 @@ Optionally updating task's input and output as specified in the payload.
| `GET /tasks/queue/sizes?taskType=&taskType=&taskType` | Return the size of pending tasks for given task types |
|||
-## Polling and Update Task
-These are critical endpoints used to poll for task and updating the task result by worker.
+## Polling, Ack and Update Task
+These are critical endpoints used to poll for task, send ack (after polling) and finally updating the task result by worker.
+
| Endpoint | Description |
|---------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -151,6 +154,7 @@ These are critical endpoints used to poll for task and updating the task result
| `GET /tasks/poll/batch/{taskType}?count=&timeout=&workerid=&domain` | Poll for a task in a batch specified by `count`. This is a long poll and the connection will wait until `timeout` or if there is at-least 1 item available, whichever comes first.`workerid` identifies the worker that polled for the job and `domain` allows the poller to poll for a task in a specific domain |
| `POST /tasks` | Update the result of task execution. See the schema below. |
+
### Schema for updating Task Result
```json
{
@@ -165,3 +169,5 @@ These are critical endpoints used to poll for task and updating the task result
}
```
+!!!Info "Acknowledging tasks after poll"
+ If the worker fails to ack the task after polling, the task is re-queued and put back in queue and is made available during subsequent poll.
diff --git a/docs/docs/architecture.md b/docs/docs/architecture.md
deleted file mode 100644
index 3440557681..0000000000
--- a/docs/docs/architecture.md
+++ /dev/null
@@ -1,136 +0,0 @@
-## High Level Architecture
-
-
-
-The API and storage layers are pluggable and provide ability to work with different backends and queue service providers.
-
-## Installing and Running
-
-!!! hint "Running in production"
- For a detailed configuration guide on installing and running Conductor server in production visit [Conductor Server](../server) documentation.
-
-### Running In-Memory Server
-
-Follow the steps below to quickly bring up a local Conductor instance backed by an in-memory database with a simple kitchen sink workflow that demonstrate all the capabilities of Conductor.
-
-!!!warning:
- In-Memory server is meant for a quick demonstration purposes and does not store any data on disk. All the data is lost once the server dies.
-
-#### Checkout the source from GitHub
-
-```
-git clone git@github.com:Netflix/conductor.git
-```
-
-#### Start Local Server
-
-
-> **NOTE for Mac users**: If you are using a new Mac with an Apple Silicon Chip, you must make a small change to ```conductor/grpc/build.gradle``` - adding "osx-x86_64" to two lines:
-```
-protobuf {
- protoc {
- artifact = "com.google.protobuf:protoc:${revProtoBuf}:osx-x86_64"
- }
- plugins {
- grpc {
- artifact = "io.grpc:protoc-gen-grpc-java:${revGrpc}:osx-x86_64"
- }
- }
-...
-}
-```
-
-
-
-The server is in the directory `conductor/server`. To start it, execute the following command in the root of the project.
-
-```shell
-./gradlew bootRun
-# wait for the server to come online
-```
-Swagger APIs can be accessed at [http://localhost:8080/swagger-ui.html](http://localhost:8080/swagger-ui.html)
-
-#### Start UI Server
-
-The UI Server is in the directory `conductor/ui`.
-
-To run it you need to have [Node](https://nodejs.org) 14 (or greater) and [Yarn](https://yarnpkg.com/) installed.
-
-In a terminal other than the one running the Conductor server:
-
-```shell
-cd ui
-yarn install
-yarn run start
-```
-
-If you get an error message `ReferenceError: primordials is not defined`, you need to use an earlier version of Node (pre-12). See [this issue for more details](https://github.com/Netflix/conductor/issues/1232).
-
-#### Or Start all the services using [docker-compose](https://github.com/Netflix/conductor/blob/master/docker/docker-compose.yaml)
-- Using compose (with Dynomite):
- ```shell
- docker-compose -f docker-compose.yaml -f docker-compose-dynomite.yaml up
- ```
-- Using compose (with Postgres):
- ```shell
- docker-compose -f docker-compose.yaml -f docker-compose-postgres.yaml up
- ```
-
-Assuming that you started Conductor locally (directly, or with Docker), launch the UI at [http://localhost:5000/](http://localhost:5000/).
-
-!!! Note
- The server will load a sample kitchensink workflow definition by default. See [here](../labs/kitchensink) for details.
-
-## Runtime Model
-Conductor follows RPC based communication model where workers are running on a separate machine from the server. Workers communicate with server over HTTP based endpoints and employs polling model for managing work queues.
-
-
-
-**Notes**
-
-* Workers are remote systems that communicate over HTTP with the conductor servers.
-* Task Queues are used to schedule tasks for workers. We use [dyno-queues][1] internally but it can easily be swapped with SQS or similar pub-sub mechanism.
-* conductor-redis-persistence module uses [Dynomite][2] for storing the state and metadata along with [Elasticsearch][3] for indexing backend.
-* See section under extending backend for implementing support for different databases for storage and indexing.
-
-[1]: https://github.com/Netflix/dyno-queues
-[2]: https://github.com/Netflix/dynomite
-[3]: https://www.elastic.co
-
-## High Level Steps
-**Steps required for a new workflow to be registered and get executed:**
-
-1. Define task definitions used by the workflow.
-2. Create the workflow definition
-3. Create task worker(s) that polls for scheduled tasks at regular interval
-
-The [Beginner lab](../labs/beginner) has a good walk-through of steps 1-3.
-
-**Trigger Workflow Execution**
-
-```
-POST /workflow/{name}
-{
- ... //json payload as workflow input
-}
-```
-
-**Polling for a task**
-
-```
-GET /tasks/poll/batch/{taskType}
-```
-
-**Update task status**
-
-```
-POST /tasks
-{
- "outputData": {
- "encodeResult":"success",
- "location": "http://cdn.example.com/file/location.png"
- //any task specific output
- },
- "status": "COMPLETED"
-}
-```
diff --git a/docs/docs/architecture/overview.md b/docs/docs/architecture/overview.md
new file mode 100644
index 0000000000..81bd00e38b
--- /dev/null
+++ b/docs/docs/architecture/overview.md
@@ -0,0 +1,21 @@
+# Overview
+
+
+
+The API and storage layers are pluggable and provide ability to work with different backends and queue service providers.
+
+## Runtime Model
+Conductor follows RPC based communication model where workers are running on a separate machine from the server. Workers communicate with server over HTTP based endpoints and employs polling model for managing work queues.
+
+
+
+**Notes**
+
+* Workers are remote systems that communicate over HTTP with the conductor servers.
+* Task Queues are used to schedule tasks for workers. We use [dyno-queues][1] internally but it can easily be swapped with SQS or similar pub-sub mechanism.
+* conductor-redis-persistence module uses [Dynomite][2] for storing the state and metadata along with [Elasticsearch][3] for indexing backend.
+* See section under extending backend for implementing support for different databases for storage and indexing.
+
+[1]: https://github.com/Netflix/dyno-queues
+[2]: https://github.com/Netflix/dynomite
+[3]: https://www.elastic.co
diff --git a/docs/docs/tasklifecycle.md b/docs/docs/architecture/tasklifecycle.md
similarity index 94%
rename from docs/docs/tasklifecycle.md
rename to docs/docs/architecture/tasklifecycle.md
index 699418052d..57f99512e3 100644
--- a/docs/docs/tasklifecycle.md
+++ b/docs/docs/architecture/tasklifecycle.md
@@ -1,14 +1,14 @@
## Task state transitions
The figure below depicts the state transitions that a task can go through within a workflow execution.
-
+
## Retries and Failure Scenarios
### Task failure and retries
Retries for failed task executions of each task can be configured independently. retryCount, retryDelaySeconds and retryLogic can be used to configure the retry mechanism.
-
+
1. Worker (W1) polls for task T1 from the Conductor server and receives the task.
2. Upon processing this task, the worker determines that the task execution is a failure and reports this to the server with FAILED status after 10 seconds.
@@ -18,7 +18,7 @@ Retries for failed task executions of each task can be configured independently.
### Timeout seconds
Timeout is the maximum amount of time that the task must reach a terminal state in, else the task will be marked as TIMED_OUT.
-
+
**0 seconds** -> Worker polls for task T1 fom the Conductor server and receives the task. T1 is put into IN_PROGRESS status by the server.
Worker starts processing the task but is unable to process the task at this time. Worker updates the server with T1 set to IN_PROGRESS status and a callback of 9 seconds.
@@ -36,7 +36,7 @@ Server puts T1 back in the queue but makes it invisible and the worker continues
### Response timeout seconds
Response timeout is the time within which the worker must respond to the server with an update for the task, else the task will be marked as TIMED_OUT.
-
+
**0 seconds** -> Worker polls for the task T1 from the Conductor server and receives the task. T1 is put into IN_PROGRESS status by the server.
diff --git a/docs/docs/configuration/eventhandlers.md b/docs/docs/configuration/eventhandlers.md
index e86bffcea8..8417d67c09 100644
--- a/docs/docs/configuration/eventhandlers.md
+++ b/docs/docs/configuration/eventhandlers.md
@@ -1,4 +1,4 @@
-## Introduction
+# Event Handlers
Eventing in Conductor provides for loose coupling between workflows and support for producing and consuming events from external systems.
This includes:
@@ -11,7 +11,7 @@ Conductor provides SUB_WORKFLOW task that can be used to embed a workflow inside
## Event Task
Event task provides ability to publish an event (message) to either Conductor or an external eventing system like SQS. Event tasks are useful for creating event based dependencies for workflows and tasks.
-See [Event Task](../../reference-docs/event-task) for documentation.
+See [Event Task](/reference-docs/event-task.html) for documentation.
## Event Handler
Event handlers are listeners registered that executes an action when a matching event occurs. The supported actions are:
@@ -108,7 +108,7 @@ Given the following payload in the message:
"expandInlineJSON": true
}
```
-Input for starting a workflow and output when completing / failing task follows the same [expressions](/configuration/workflowdef/#wiring-inputs-and-outputs) used for wiring workflow inputs.
+Input for starting a workflow and output when completing / failing task follows the same [expressions](/configuration/workflowdef.html#wiring-inputs-and-outputs) used for wiring workflow inputs.
!!!info "Expanding stringified JSON elements in payload"
`expandInlineJSON` property, when set to true will expand the inlined stringified JSON elements in the payload to JSON documents and replace the string value with JSON document.
diff --git a/docs/docs/configuration/isolationgroups.md b/docs/docs/configuration/isolationgroups.md
index 8aecdac225..1d2320261b 100644
--- a/docs/docs/configuration/isolationgroups.md
+++ b/docs/docs/configuration/isolationgroups.md
@@ -1,15 +1,16 @@
-#### Isolation Group Id
+# Isolation Groups
Consider an HTTP task where the latency of an API is high, task queue piles up effecting execution of other HTTP tasks which have low latency.
We can isolate the execution of such tasks to have predictable performance using `isolationgroupId`, a property of task definition.
-When we set isolationGroupId, the executor(SystemTaskWorkerCoordinator) will allocate an isolated queue and an isolated thread pool for execution of those tasks.
+When we set isolationGroupId, the executor `SystemTaskWorkerCoordinator` will allocate an isolated queue and an isolated thread pool for execution of those tasks.
If no `isolationgroupId` is specified in task definition, then fallback is default behaviour where the executor executes the task in shared thread-pool for all tasks.
-Example taskdef
+## Example
+** Task Definition **
```json
{
"name": "encode_task",
@@ -35,7 +36,7 @@ Example taskdef
"isolationgroupId": "myIsolationGroupId"
}
```
-Example Workflow task
+** Workflow Definition **
```json
{
"name": "encode_and_deploy",
@@ -73,7 +74,7 @@ The property `workflow.isolated.system.task.worker.thread.count` sets the threa
isolationGroupId is currently supported only in HTTP and kafka Task.
-#### Execution Name Space
+### Execution Name Space
`executionNameSpace` A property of taskdef can be used to provide JVM isolation to task execution and scale executor deployments horizontally.
@@ -112,10 +113,7 @@ If the property is not set, the executor executes tasks without any executionNam
}
```
-
-
-
-Example Workflow task
+#### Example Workflow task
```json
{
diff --git a/docs/docs/configuration/sysoperator.md b/docs/docs/configuration/sysoperator.md
index d85bfb2111..181d72fd3d 100644
--- a/docs/docs/configuration/sysoperator.md
+++ b/docs/docs/configuration/sysoperator.md
@@ -1,4 +1,4 @@
-# Operators
+# System Operators
Operators are built-in primitives in Conductor that allow you to define the control flow in the workflow.
Operators are similar to programming constructs such as for loops, decisions, etc.
@@ -9,12 +9,12 @@ Conductor supports the following programming language constructs:
| Language Construct | Conductor Operator |
|----------------------------------|-------------------------------------------------------------|
-| Do/While or Loops | [Do While Task](../../reference-docs/do-while-task) |
-| Dynamic Fork | [Dynamic Fork Task](../../reference-docs/dynamic-fork-task) |
-| Fork / Parallel execution | [Fork Task](../../reference-docs/fork-task) |
-| Join | [Join Task](../../reference-docs/join-task) |
-| Sub Process / Sub-Flow | [Sub Workflow Task](../../reference-docs/sub-workflow-task) |
-| Switch//Decision/if..then...else | [Switch Task](../../reference-docs/switch-task) |
-| Terminate | [Terminate Task](../../reference-docs/terminate-task) |
-| Variables | [Variable Task](../../reference-docs/set-variable-task) |
-| Wait | [Wait Task](../../reference-docs/wait-task) |
+| Do/While or Loops | [Do While Task](/reference-docs/do-while-task.html) |
+| Dynamic Fork | [Dynamic Fork Task](/reference-docs/dynamic-fork-task.html) |
+| Fork / Parallel execution | [Fork Task](/reference-docs/fork-task.html) |
+| Join | [Join Task](/reference-docs/join-task.html) |
+| Sub Process / Sub-Flow | [Sub Workflow Task](/reference-docs/sub-workflow-task.html) |
+| Switch//Decision/if..then...else | [Switch Task](/reference-docs/switch-task.html) |
+| Terminate | [Terminate Task](/reference-docs/terminate-task.html) |
+| Variables | [Variable Task](/reference-docs/set-variable-task.html) |
+| Wait | [Wait Task](/reference-docs/wait-task.html) |
diff --git a/docs/docs/configuration/systask.md b/docs/docs/configuration/systask.md
index 5a155be713..4b2f964a63 100644
--- a/docs/docs/configuration/systask.md
+++ b/docs/docs/configuration/systask.md
@@ -1,8 +1,4 @@
----
-sidebar_position: 1
----
-
-# System Task
+# System Tasks
System Tasks (Workers) are built-in tasks that are general purpose and re-usable. They run on the Conductor servers.
Such tasks allow you to get started without having to write custom workers.
@@ -11,14 +7,13 @@ Such tasks allow you to get started without having to write custom workers.
Conductor has the following set of system tasks available.
-
| Task | Description | Use Case |
|-----------------------|--------------------------------------------------------|------------------------------------------------------------------------------------|
-| Event Publishing | [Event Task](../../reference-docs/event-task) | External eventing system integration. e.g. amqp, sqs, nats |
-| HTTP | [HTTP Task](../../reference-docs/http-task) | Invoke any HTTP(S) endpoints |
-| Inline Code Execution | [Inline Task](../../reference-docs/inline-task) | Execute arbitrary lightweight javascript code |
-| JQ Transform | [JQ Task](../../reference-docs/json-jq-transform-task) | Use JQ to transform task input/output |
-| Kafka Publish | [Kafka Task](../../reference-docs/kafka-publish-task) | Publish messages to Kafka |
+| Event Publishing | [Event Task](/reference-docs/event-task.html) | External eventing system integration. e.g. amqp, sqs, nats |
+| HTTP | [HTTP Task](/reference-docs/http-task.html) | Invoke any HTTP(S) endpoints |
+| Inline Code Execution | [Inline Task](/reference-docs/inline-task.html) | Execute arbitrary lightweight javascript code |
+| JQ Transform | [JQ Task](/reference-docs/json-jq-transform-task.html) | Use JQ to transform task input/output |
+| Kafka Publish | [Kafka Task](/reference-docs/kafka-publish-task.html) | Publish messages to Kafka |
| Name | Description |
|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
diff --git a/docs/docs/configuration/taskdef.md b/docs/docs/configuration/taskdef.md
index af066343bf..075912ff47 100644
--- a/docs/docs/configuration/taskdef.md
+++ b/docs/docs/configuration/taskdef.md
@@ -1,4 +1,4 @@
-## Task Definition
+# Task Definition
Tasks are the building blocks of workflow in Conductor. A task can be an operator, system task or custom code written in any programming language.
diff --git a/docs/docs/configuration/taskdomains.md b/docs/docs/configuration/taskdomains.md
index c16647c20c..2d3f3d7da8 100644
--- a/docs/docs/configuration/taskdomains.md
+++ b/docs/docs/configuration/taskdomains.md
@@ -1,4 +1,4 @@
-## Task Domains
+# Task Domains
Task domains helps support task development. The idea is same “task definition” can be implemented in different “domains”. A domain is some arbitrary name that the developer controls. So when the workflow is started, the caller can specify, out of all the tasks in the workflow, which tasks need to run in a specific domain, this domain is then used to poll for task on the client side to execute it.
As an example if a workflow (WF1) has 3 tasks T1, T2, T3. The workflow is deployed and working fine, which means there are T2 workers polling and executing. If you modify T2 and run it locally there is no guarantee that your modified T2 worker will get the task that you are looking for as it coming from the general T2 queue. “Task Domain” feature solves this problem by splitting the T2 queue by domains, so when the app polls for task T2 in a specific domain, it get the correct task.
diff --git a/docs/docs/configuration/workerdef.md b/docs/docs/configuration/workerdef.md
index 8f3af163ae..34347a2ad8 100644
--- a/docs/docs/configuration/workerdef.md
+++ b/docs/docs/configuration/workerdef.md
@@ -1,4 +1,5 @@
-## Worker
+# Worker Definition
+
A worker is responsible for executing a task. Operator and System tasks are handled by the Conductor server, while user
defined tasks needs to have a worker created that awaits the work to be scheduled by the server for it to be executed.
Workers can be implemented in any language, and Conductor provides support for Java, Golang and Python worker framework that provides features such as
diff --git a/docs/docs/configuration/workflowdef.md b/docs/docs/configuration/workflowdef.md
index 9ccca3fd11..2a4a50c098 100644
--- a/docs/docs/configuration/workflowdef.md
+++ b/docs/docs/configuration/workflowdef.md
@@ -1,4 +1,4 @@
-# Workflows
+# Workflow Definition
## What are Workflows?
@@ -26,7 +26,7 @@ execution in a reliable & scalable manner.
Let's start with a basic workflow and understand what are the different aspects of it. In particular, we will talk about
two stages of a workflow, *defining* a workflow and *executing* a workflow
-### *Simple Workflow Example*
+### Simple Workflow Example
Assume your business logic is to simply to get some shipping information and then do the shipping. You start by
logically partitioning them into two tasks:
@@ -70,7 +70,7 @@ First we would build these two task definitions. Let's assume that ```shipping i
"failureWorkflow": "shipping_issues",
"restartable": true,
"workflowStatusListenerEnabled": true,
- "ownerEmail": "devrel@orkes.io",
+ "ownerEmail": "conductor@example.com",
"timeoutPolicy": "ALERT_ONLY",
"timeoutSeconds": 0,
"variables": {},
@@ -111,9 +111,9 @@ The mail_a_box workflow has 2 tasks:
| description | Description of the task | optional |
| optional | true or false. When set to true - workflow continues even if the task fails. The status of the task is reflected as `COMPLETED_WITH_ERRORS` | Defaults to `false` |
| inputParameters | JSON template that defines the input given to the task | See [Wiring Inputs and Outputs](#wiring-inputs-and-outputs) for details |
-| domain | See [Task Domains](/conductor/configuration/taskdomains) for more information. | optional |
+| domain | See [Task Domains](/configuration/taskdomains.html) for more information. | optional |
-In addition to these parameters, System Tasks have their own parameters. Checkout [System Tasks](/conductor/configuration/systask/) for more information.
+In addition to these parameters, System Tasks have their own parameters. Checkout [System Tasks](/configuration/systask.html) for more information.
## Wiring Inputs and Outputs
@@ -226,4 +226,4 @@ And `url` would be `https://some_url:7004` if no `url` was provided as input to
## Workflow notifications
-Conductor can be configured to publish notifications to external systems upon completion/termination of workflows. See [extending conductor](../../extend#workflow-status-listener) for details.
+Conductor can be configured to publish notifications to external systems upon completion/termination of workflows. See [extending conductor](/extend.html) for details.
diff --git a/docs/docs/css/custom.css b/docs/docs/css/custom.css
index e41a2e9b55..3e4728cd5d 100644
--- a/docs/docs/css/custom.css
+++ b/docs/docs/css/custom.css
@@ -1,20 +1,313 @@
-.hljs {
- font-size: 13px;
- background-color: #f3f6f6;
+:root {
+ /*--main-text-color: #212121;*/
+ --brand-blue: #1976d2;
+ --brand-dark-blue: #242A36;
+ --caption-color: #4f4f4f;
+ --brand-lt-blue: #f0f5fb;
+ --brand-gray: rgb(118, 118, 118);
+ --brand-lt-gray: rgb(203,204,207);
+ --brand-red: #e50914;
}
-.hljs-attribute {
- color: #000000
+body {
+ color: var(--brand-dark-blue);
+ font-family: "Roboto", sans-serif;
+ font-weight: 400;
}
-.hljs-string {
- color: green
+
+body::before {
+ background: none;
+ display: none;
}
-code {
- color: green;
- font-size: 13px
+body > .container {
+ padding-top: 30px;
}
-.wy-side-nav-search {
- margin-bottom: 0;
+
+.bg-primary {
+ background: #fff !important;
+}
+
+/* Navbar */
+.navbar {
+ box-shadow: 0 4px 8px 0 rgb(0 0 0 / 10%), 0 0 2px 0 rgb(0 0 0 / 10%);
+ padding-left: 30px;
+ padding-right: 30px;
+ height: 80px;
+}
+.navbar-brand {
+ background-image: url(/img/logo.svg);
+ background-size: cover;
+ color: transparent !important;
+ padding: 0;
+ text-shadow: none;
+ margin-top: -6px;
+ height: 37px;
+ width: 175px;
+}
+.navbar-nav {
+ margin-left: 50px;
+}
+.navbar-nav > .navitem, .navbar-nav > .dropdown {
+ margin-left: 30px;
+}
+.navbar-nav > li .nav-link{
+ font-size: 15px;
+}
+
+.navbar-nav .nav-link {
+ color: #242A36 !important;
+ font-family: "Inter";
+ font-weight: 700;
+}
+
+.navbar-nav.ml-auto > li:first-child {
+ display: none;
+}
+.navbar-nav.ml-auto .nav-link{
+ font-size: 0px;
+}
+.navbar-nav.ml-auto .nav-link .fa{
+ font-size: 30px;
+}
+.navbar-nav .dropdown-item {
+ color: var(--brand-dark-blue);
+ font-family: "Inter";
+ font-weight: 500;
+ font-size: 14px;
+ background-color: transparent;
+}
+.navbar-nav .dropdown-menu > li:hover {
+ background-color: var(--brand-blue);
+}
+.navbar-nav .dropdown-menu > li:hover > .dropdown-item {
+ color: #fff;
+}
+.navbar-nav .dropdown-submenu:hover > .dropdown-item {
+ background-color: var(--brand-blue);
+}
+
+
+.navbar-nav .dropdown-menu li {
+ margin: 0px;
+ padding-top: 5px;
+ padding-bottom: 5px;
+}
+.navbar-nav .dropdown-item.active {
+ background-color: transparent;
+}
+
+.brand-darkblue {
+ background: #242A36 !important;
+}
+
+.brand-gray {
+ background: rgb(245,245,245);
+}
+.brand-blue {
+ background: #1976D2;
+}
+.brand-white {
+ background: #fff;
+}
+.logo {
+ height: 444px;
+}
+
+/* Fonts */
+h1, h2, h3, h4, h5, h6 {
+ color: var(--brand-dark-blue);
+ margin-bottom: 20px;
+}
+h1:first-child {
+ margin-top: 0;
+}
+
+h1 {
+ font-family: "Inter", sans-serif;
+ font-size: 32px;
+ font-weight: 700;
+ margin-top: 50px;
+}
+
+h2 {
+ font-family: "Inter", sans-serif;
+ font-size: 24px;
+ font-weight: 700;
+ margin-top: 40px;
+}
+
+h3 {
+ font-family: "Roboto", sans-serif;
+ font-size: 20px;
+ font-weight: 500;
+ margin-top: 30px;
+}
+
+h4 {
+ font-family: "Roboto", sans-serif;
+ font-size: 18px;
+ font-weight: 400;
+ margin-top: 20px;
+}
+
+.main li {
+ margin-bottom: 15px;
+}
+
+
+.btn {
+ font-family: "Roboto", sans-serif;
+ font-size: 14px;
+}
+.btn-primary {
+ background: #1976D2;
+ border: none;
+}
+
+.hero {
+ padding-top: 100px;
+ padding-bottom: 100px;
+}
+
+.hero .heading {
+ font-size: 56px;
+ font-weight: 900;
+ line-height: 68px;
+}
+
+.hero .btn {
+ font-size: 16px;
+ padding: 10px 20px;
+}
+
+.hero .illustration {
+ margin-left: 35px;
+}
+
+
+.bullets .heading, .module .heading {
+ font-family: "Inter", sans-serif;
+ font-size: 26px;
+ font-weight: 700;
+}
+.bullets .row {
+ margin-bottom: 60px;
+}
+.bullets .caption {
+ padding-top: 10px;
+ padding-right: 30px;
+}
+.icon {
+ height: 25px;
+ margin-right: 5px;
+ vertical-align: -3px;
+}
+
+.caption {
+ font-weight: 400;
+ font-size: 17px;
+ line-height: 24px;
+ color: var(--caption-color);
+}
+
+.module {
+ margin-top: 80px;
+ margin-bottom: 80px;
+ padding-top: 50px;
+ padding-bottom: 50px;
+}
+
+.module .caption {
+ padding-top: 10px;
+ padding-right: 80px;
+}
+.module .screenshot {
+ width: 600px;
+ height: 337px;
+ box-shadow:inset 0 1px 0 rgba(255,255,255,.6), 0 22px 70px 4px rgba(0,0,0,0.56), 0 0 0 1px rgba(0, 0, 0, 0.0);
+ border-radius: 5px;
+ background-size: cover;
+}
+
+/* Footer */
+footer {
+ margin: 0px;
+ padding: 0px !important;
+ text-align: left;
+ font-weight: 400;
+}
+.footer {
+ background-color: var(--brand-dark-blue);
+ padding: 50px 0px;
+ color: #fff;
+ font-size: 14px;
+ margin-top: 50px;
+}
+.footer a {
+ color: var(--brand-lt-gray);
+}
+.footer .subhead {
+ font-weight: 700;
+ color: #fff;
+ font-size: 15px;
+ margin-bottom: 10px;
+}
+.footer .red {
+ color: var(--brand-red);
+}
+.footer .fr {
+ text-align: right;
+}
+
+/* TOC menu */
+.toc ul {
+ list-style: none;
+ padding: 0px;
+}
+.toc > ul > li li {
+ padding-left: 15px;
+ font-weight: 400;
+ font-size: 14px;
+}
+.toc > ul > li {
+ font-size: 15px;
+ font-weight: 500;
+}
+.toc .toc-link {
+ margin-bottom: 5px;
+ display: block;
+ color: var(--brand-dark-blue);
+}
+.toc .toc-link.active {
+ font-weight: 700;
+}
+
+/* Homepage Overrides */
+.homepage > .container {
+ max-width: none;
+}
+.homepage .toc {
+ display: none;
+}
+
+/* Comparison block */
+.compare {
+ background-color: var(--brand-lt-blue);
+ padding-top: 80px;
+ padding-bottom: 80px;
+ margin: 0px -15px;
+}
+.compare .heading {
+ margin-bottom: 30px;
+ margin-top: 0px;
+}
+.compare .bubble {
+ background: #fff;
+ border-radius: 10px;
+ padding: 30px;
+ height: 100%;
+}
+
+.compare .caption {
+ font-size: 15px;
+ line-height: 22px;
}
-body {
- font-family: "Arial","proxima-nova","Helvetica Neue","Arial","sans-serif";
-}
\ No newline at end of file
diff --git a/docs/docs/extend.md b/docs/docs/extend.md
index 3b49c33185..82895f6677 100644
--- a/docs/docs/extend.md
+++ b/docs/docs/extend.md
@@ -1,3 +1,5 @@
+# Extending Conductor
+
## Backend
Conductor provides a pluggable backend. The current implementation uses Dynomite.
diff --git a/docs/docs/externalpayloadstorage.md b/docs/docs/externalpayloadstorage.md
index 3d9c62dd34..097f0b8fca 100644
--- a/docs/docs/externalpayloadstorage.md
+++ b/docs/docs/externalpayloadstorage.md
@@ -1,3 +1,5 @@
+# External Payload Storage
+
!!!warning
The external payload storage is currently only implemented to be used to by the Java client. Client libraries in other languages need to be modified to enable this.
Contributions are welcomed.
diff --git a/docs/docs/faq.md b/docs/docs/faq.md
index 787b4f6623..d72a39a98b 100644
--- a/docs/docs/faq.md
+++ b/docs/docs/faq.md
@@ -20,14 +20,14 @@ Ensure all the tasks are registered via `/metadata/taskdefs` APIs. Add any miss
### Where does my worker run? How does conductor run my tasks?
Conductor does not run the workers. When a task is scheduled, it is put into the queue maintained by Conductor. Workers are required to poll for tasks using `/tasks/poll` API at periodic interval, execute the business logic for the task and report back the results using `POST /tasks` API call.
-Conductor, however will run [system tasks](../configuration/systask/) on the Conductor server.
+Conductor, however will run [system tasks](/configuration/systask.html) on the Conductor server.
### How can I schedule workflows to run at a specific time?
Netflix Conductor itself does not provide any scheduling mechanism. But there is a community project [_Schedule Conductor Workflows_](https://github.com/jas34/scheduledwf) which provides workflow scheduling capability as a pluggable module as well as workflow server.
Other way is you can use any of the available scheduling systems to make REST calls to Conductor to start a workflow. Alternatively, publish a message to a supported eventing system like SQS to trigger a workflow.
-More details about [eventing](../configuration/eventhandlers/).
+More details about [eventing](/configuration/eventhandlers.html).
### How do I setup Dynomite cluster?
@@ -65,11 +65,11 @@ When a workflow fails, you can configure a "failure workflow" to run using the``
You can also use the Workflow Status Listener:
-* Set the workflowStatusListenerEnabled field in your workflow definition to true which enables [notifications](https://netflix.github.io/conductor/configuration/workflowdef/#workflow-notifications).
-* Add a custom implementation of the Workflow Status Listener. Refer [this](https://netflix.github.io/conductor/extend/#workflow-status-listener).
-* This notification can be implemented in such a way as to either send a notification to an external system or to send an event on the conductor queue to complete/fail another task in another workflow as described [here](https://netflix.github.io/conductor/configuration/eventhandlers/#event-handler).
+* Set the workflowStatusListenerEnabled field in your workflow definition to true which enables [notifications](/configuration/workflowdef.html#workflow-notifications).
+* Add a custom implementation of the Workflow Status Listener. Refer [this](/extend.html#workflow-status-listener).
+* This notification can be implemented in such a way as to either send a notification to an external system or to send an event on the conductor queue to complete/fail another task in another workflow as described [here](/configuration/eventhandlers.html).
-Refer to this [documentation](../configuration/workflowdef/#workflow-notifications) to extend conductor to send out events/notifications upon workflow completion/failure.
+Refer to this [documentation](/configuration/workflowdef.html#workflow-notifications) to extend conductor to send out events/notifications upon workflow completion/failure.
diff --git a/docs/docs/gettingstarted/basicconcepts.md b/docs/docs/gettingstarted/basicconcepts.md
index c60a96c29c..6592c04239 100644
--- a/docs/docs/gettingstarted/basicconcepts.md
+++ b/docs/docs/gettingstarted/basicconcepts.md
@@ -1,3 +1,5 @@
+# Basic Concepts
+
## Definitions (aka Metadata or Blueprints)
Conductor definitions are like class definitions in OOP paradigm, or templates. You define this once, and use for each workflow execution. Definitions to Executions have 1:N relationship.
@@ -5,13 +7,13 @@ Conductor definitions are like class definitions in OOP paradigm, or templates.
Tasks are the building blocks of Workflow. There must be at least one task in a Workflow.
Tasks can be categorized into two types:
- * [System tasks](../../configuration/systask) - executed by Conductor server.
- * [Worker tasks](../../configuration/workerdef) - executed by your own workers.
+ * [System tasks](/configuration/systask.html) - executed by Conductor server.
+ * [Worker tasks](/configuration/workerdef.html) - executed by your own workers.
## Workflow
A Workflow is the container of your process flow. It could include several different types of Tasks, Sub-Workflows, inputs and outputs connected to each other, to effectively achieve the desired result. The tasks are either control tasks (fork, conditional etc) or application tasks (e.g. encode a file) that are executed on a remote machine.
-[Detailed description](../../configuration/workflowdef)
+[Detailed description](/configuration/workflowdef.html)
## Task Definition
Task definitions help define Task level parameters like inputs and outputs, timeouts, retries etc.
@@ -19,12 +21,12 @@ Task definitions help define Task level parameters like inputs and outputs, time
* All tasks need to be registered before they can be used by active workflows.
* A task can be re-used within multiple workflows.
-[Detailed description](../../configuration/taskdef)
+[Detailed description](/configuration/taskdef.html)
## System Tasks
System tasks are executed within the JVM of the Conductor server and managed by Conductor for its execution and scalability.
-See [Systems tasks](../../configuration/systask) for list of available Task types, and instructions for using them.
+See [Systems tasks](/configuration/systask.html) for list of available Task types, and instructions for using them.
!!! Note
Conductor provides an API to create user defined tasks that are executed in the same JVM as the engine. See [WorkflowSystemTask](https://github.com/Netflix/conductor/blob/main/core/src/main/java/com/netflix/conductor/core/execution/tasks/WorkflowSystemTask.java) interface for details.
diff --git a/docs/docs/gettingstarted/client.md b/docs/docs/gettingstarted/client.md
index 4ef4dce736..fd7f537bd7 100644
--- a/docs/docs/gettingstarted/client.md
+++ b/docs/docs/gettingstarted/client.md
@@ -1,3 +1,4 @@
+# Using the Client
Conductor tasks that are executed by remote workers communicate over HTTP endpoints/gRPC to poll for the task and update the status of the execution.
## Client APIs
diff --git a/docs/docs/gettingstarted/docker.md b/docs/docs/gettingstarted/docker.md
new file mode 100644
index 0000000000..c84b331fd4
--- /dev/null
+++ b/docs/docs/gettingstarted/docker.md
@@ -0,0 +1,148 @@
+
+# Running via Docker Compose
+
+In this article we will explore how you can set up Netflix Conductor on your local machine using Docker compose.
+The docker compose will bring up the following:
+1. Conductor API Server
+2. Conductor UI
+3. Elasticsearch for searching workflows
+
+## Prerequisites
+1. Docker: [https://docs.docker.com/get-docker/](https://docs.docker.com/get-docker/)
+2. Recommended host with CPU and RAM to be able to run multiple docker containers (at-least 16GB RAM)
+
+## Steps
+
+#### 1. Clone the Conductor Code
+
+```shell
+$ git clone https://github.com/Netflix/conductor.git
+```
+
+#### 2. Build the Docker Compose
+
+```shell
+$ cd conductor
+conductor $ cd docker
+docker $ docker-compose build
+```
+#### Note: Conductor supplies multiple docker compose templates that can be used with different configurations:
+
+| File | Containers |
+|--------------------------------|-----------------------------------------------------------------------------------------|
+| docker-compose.yaml | 1. In Memory Conductor Server 2. Elasticsearch 3. UI |
+| docker-compose-dynomite.yaml | 1. In Memory Conductor Server 2. Elasticsearch 3. UI 4. Dynomite Redis for persistence |
+| docker-compose-postgres.yaml | 1. In Memory Conductor Server 2. Elasticsearch 3. UI 4. Postgres persistence |
+| docker-compose-prometheus.yaml | Brings up Prometheus server |
+
+#### 3. Run Docker Compose
+
+```shell
+docker $ docker-compose up
+```
+
+Once up and running, you will see the following in your Docker dashboard:
+
+1. Elasticsearch
+2. Conductor UI
+3. Conductor Server
+
+You can access all three on your browser to verify that it is running correctly:
+
+Conductor Server URL: [http://localhost:8080/swagger-ui/index.html?configUrl=/api-docs/swagger-config](http://localhost:8080/swagger-ui/index.html?configUrl=/api-docs/swagger-config)
+
+
+
+Conductor UI URL: [http://localhost:5000/](http://localhost:5000/)
+
+
+
+
+### Exiting Compose
+`Ctrl+c` will exit docker compose.
+
+To ensure images are stopped execute: `docker-compose down`.
+
+## Standalone Server Image
+To build and run the server image, without using `docker-compose`, from the `docker` directory execute:
+```
+docker build -t conductor:server -f server/Dockerfile ../
+docker run -p 8080:8080 -d --name conductor_server conductor:server
+```
+This builds the image `conductor:server` and runs it in a container named `conductor_server`. The API should now be accessible at `localhost:8080`.
+
+To 'login' to the running container, use the command:
+```
+docker exec -it conductor_server /bin/sh
+```
+
+## Standalone UI Image
+From the `docker` directory,
+```
+docker build -t conductor:ui -f ui/Dockerfile ../
+docker run -p 5000:5000 -d --name conductor_ui conductor:ui
+```
+This builds the image `conductor:ui` and runs it in a container named `conductor_ui`. The UI should now be accessible at `localhost:5000`.
+
+### Note
+* In order for the UI to do anything useful the Conductor Server must already be running on port 8080, either in a Docker container (see above), or running directly in the local JRE.
+* Additionally, significant parts of the UI will not be functional without Elastisearch being available. Using the `docker-compose` approach alleviates these considerations.
+
+## Monitoring with Prometheus
+
+Start Prometheus with:
+`docker-compose -f docker-compose-prometheus.yaml up -d`
+
+Go to [http://127.0.0.1:9090](http://127.0.0.1:9090).
+
+
+## Potential problem when using Docker Images
+
+#### Not enough memory
+
+ 1. You will need at least 16 GB of memory to run everything. You can modify the docker compose to skip using
+ Elasticsearch if you have no option to run this with your memory options.
+ 2. To disable Elasticsearch using Docker Compose - follow the steps listed here: **TODO LINK**
+
+#### Elasticsearch fails to come up in arm64 based CPU machines
+
+ 1. As of writing this article, Conductor relies on 6.8.x version of Elasticsearch. This version doesn't have an
+ arm64 based Docker image. You will need to use Elasticsearch 7.x which requires a bit of customization to get up
+ and running
+
+#### Elasticsearch remains in Yellow health
+
+ 1. When you run Elasticsearch, sometimes the health remains in Yellow state. Conductor server by default requires
+ Green state to run when indexing is enabled. To work around this, you can use the following property:
+ `conductor.elasticsearch.clusteHealthColor=yellow` Reference: [Issue 2262](https://github.com/Netflix/conductor/issues/2262)
+
+
+
+#### Elasticsearch timeout
+Standalone(single node) elasticsearch has a yellow status which will cause timeout for conductor server (Required: Green).
+Spin up a cluster (more than one) to prevent timeout or use config option `conductor.elasticsearch.clusteHealthColor=yellow`.
+
+See issue: https://github.com/Netflix/conductor/issues/2262
+
+#### Changes in config-*.properties do not take effect
+Config is copy into image during docker build. You have to rebuild the image or better, link a volume to it to reflect new changes.
+
+#### To troubleshoot a failed startup
+Check the log of the server, which is located at `/app/logs` (default directory in dockerfile)
+
+#### Unable to access to conductor:server API on port 8080
+It may takes some time for conductor server to start. Please check server log for potential error.
+
+#### Elasticsearch
+Elasticsearch is optional, please be aware that disable it will make most of the conductor UI not functional.
+
+##### How to enable Elasticsearch
+* Set `workflow.indexing.enabled=true` in your_config.properties
+* Add config related to elasticsearch
+ E.g.: `conductor.elasticsearch.url=http://es:9200`
+
+##### How to disable Elasticsearch
+* Set `workflow.indexing.enabled=false` in your_config.properties
+* Comment out all the config related to elasticsearch
+E.g.: `conductor.elasticsearch.url=http://es:9200`
+
diff --git a/docs/docs/gettingstarted/intro.md b/docs/docs/gettingstarted/intro.md
new file mode 100644
index 0000000000..789a7db4b1
--- /dev/null
+++ b/docs/docs/gettingstarted/intro.md
@@ -0,0 +1,28 @@
+# Why Conductor?
+## Conductor was built to help Netflix orchestrate microservices based process flows with the following features:
+
+* A distributed server ecosystem, which stores workflow state information efficiently.
+* Allow creation of process / business flows in which each individual task can be implemented by the same / different microservices.
+* A DAG (Directed Acyclic Graph) based workflow definition.
+* Workflow definitions are decoupled from the service implementations.
+* Provide visibility and traceability into these process flows.
+* Simple interface to connect workers, which execute the tasks in workflows.
+* Workers are language agnostic, allowing each microservice to be written in the language most suited for the service.
+* Full operational control over workflows with the ability to pause, resume, restart, retry and terminate.
+* Allow greater reuse of existing microservices providing an easier path for onboarding.
+* User interface to visualize, replay and search the process flows.
+* Ability to scale to millions of concurrently running process flows.
+* Backed by a queuing service abstracted from the clients.
+* Be able to operate on HTTP or other transports e.g. gRPC.
+* Event handlers to control workflows via external actions.
+* Client implementations in Java, Python and other languages.
+* Various configurable properties with sensible defaults to fine tune workflow and task executions like rate limiting, concurrent execution limits etc.
+
+## Why not peer to peer choreography?
+
+With peer to peer task choreography, we found it was harder to scale with growing business needs and complexities.
+Pub/sub model worked for simplest of the flows, but quickly highlighted some of the issues associated with the approach:
+
+* Process flows are “embedded” within the code of multiple application.
+* Often, there is tight coupling and assumptions around input/output, SLAs etc, making it harder to adapt to changing needs.
+* Almost no way to systematically answer “How much are we done with process X”?
diff --git a/docs/docs/server.md b/docs/docs/gettingstarted/local.md
similarity index 89%
rename from docs/docs/server.md
rename to docs/docs/gettingstarted/local.md
index bb1bd1bdb5..ef3943762e 100644
--- a/docs/docs/server.md
+++ b/docs/docs/gettingstarted/local.md
@@ -45,6 +45,12 @@ protobuf {
...
}
```
+You may also need to install rosetta:
+
+ ```bash
+ softwareupdate --install-rosetta
+ ```
+
```shell
$ cd conductor
@@ -55,7 +61,7 @@ server $ ../gradlew bootRun
Navigate to the swagger API docs:
[http://localhost:8080/swagger-ui/index.html?configUrl=/api-docs/swagger-config](http://localhost:8080/swagger-ui/index.html?configUrl=/api-docs/swagger-config)
-
+
## Build and Run UI
@@ -72,9 +78,9 @@ ui $ yarn run start
Launch UI [http://localhost:5000](http://localhost:5000)
-
+
## Summary
1. All the data is stored in memory, so any workflows created or excuted will be wiped out once the server is terminated.
2. Indexing is disabled, so search functionality in UI will not work and will result an empty set.
-3. See how to install Conductor using [Docker](running-locally-docker.md) with persistence and indexing.
\ No newline at end of file
+3. See how to install Conductor using [Docker](docker.md) with persistence and indexing.
\ No newline at end of file
diff --git a/docs/docs/gettingstarted/startworkflow.md b/docs/docs/gettingstarted/startworkflow.md
index d438378a29..2b285c0a75 100644
--- a/docs/docs/gettingstarted/startworkflow.md
+++ b/docs/docs/gettingstarted/startworkflow.md
@@ -1,16 +1,16 @@
-## Start Workflow Request
-
-When starting a Workflow execution with a registered definition, Workflow accepts following parameters:
+# Starting a Workflow
+## Start Workflow Endpoint
+When starting a Workflow execution with a registered definition, `/workflow` accepts following parameters:
| Field | Description | Notes |
|:--------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------|
| name | Name of the Workflow. MUST be registered with Conductor before starting workflow | |
| version | Workflow version | defaults to latest available version |
-| input | JSON object with key value params, that can be used by downstream tasks | See [Wiring Inputs and Outputs](../../configuration/workflowdef/#wiring-inputs-and-outputs) for details |
+| input | JSON object with key value params, that can be used by downstream tasks | See [Wiring Inputs and Outputs](/configuration/workflowdef.html#wiring-inputs-and-outputs) for details |
| correlationId | Unique Id that correlates multiple Workflow executions | optional |
-| taskToDomain | See [Task Domains](../../configuration/taskdomains/#task-domains) for more information. | optional |
-| workflowDef | An adhoc [Workflow Definition](../../configuration/workflowdef) to run, without registering. See [Dynamic Workflows](#dynamic-workflows). | optional |
-| externalInputPayloadStoragePath | This is taken care of by Java client. See [External Payload Storage](../../externalpayloadstorage/) for more info. | optional |
+| taskToDomain | See [Task Domains](/configuration/taskdomains.html) for more information. | optional |
+| workflowDef | An adhoc [Workflow Definition](/configuration/workflowdef.html) to run, without registering. See [Dynamic Workflows](#dynamic-workflows). | optional |
+| externalInputPayloadStoragePath | This is taken care of by Java client. See [External Payload Storage](/externalpayloadstorage.html) for more info. | optional |
| priority | Priority level for the tasks within this workflow execution. Possible values are between 0 - 99. | optional |
**Example:**
diff --git a/docs/docs/gettingstarted/steps.md b/docs/docs/gettingstarted/steps.md
new file mode 100644
index 0000000000..695aa62779
--- /dev/null
+++ b/docs/docs/gettingstarted/steps.md
@@ -0,0 +1,36 @@
+
+# High Level Steps
+Steps required for a new workflow to be registered and get executed
+
+1. Define task definitions used by the workflow.
+2. Create the workflow definition
+3. Create task worker(s) that polls for scheduled tasks at regular interval
+
+### Trigger Workflow Execution
+
+```
+POST /workflow/{name}
+{
+ ... //json payload as workflow input
+}
+```
+
+### Polling for a task
+
+```
+GET /tasks/poll/batch/{taskType}
+```
+
+### Update task status
+
+```
+POST /tasks
+{
+ "outputData": {
+ "encodeResult":"success",
+ "location": "http://cdn.example.com/file/location.png"
+ //any task specific output
+ },
+ "status": "COMPLETED"
+}
+```
diff --git a/docs/docs/how-tos/Tasks/creating-tasks.md b/docs/docs/how-tos/Tasks/creating-tasks.md
index 82a44013ac..54ae74e91b 100644
--- a/docs/docs/how-tos/Tasks/creating-tasks.md
+++ b/docs/docs/how-tos/Tasks/creating-tasks.md
@@ -1,14 +1,8 @@
----
-sidebar_position: 1
----
-
# Creating Task Definitions
Tasks can be created using the tasks metadata API
-```http request
-POST /api/metadata/taskdefs
-```
+`POST /api/metadata/taskdefs`
This API takes an array of new task definitions.
@@ -36,6 +30,6 @@ fetch("http://localhost:8080/api/metadata/taskdefs", {
## Best Practices
1. You can update a set of tasks together in this API
-2. Task configurations are important attributes that controls the behavior of this task in a Workflow. Refer to [Task Configurations](/content/docs/how-tos/task-configurations) for all the options and details'
+2. Task configurations are important attributes that controls the behavior of this task in a Workflow. Refer to [Task Configurations](/configuration/taskdef.html) for all the options and details'
3. You can also use the Conductor Swagger UI to update the tasks
diff --git a/docs/docs/how-tos/Tasks/dynamic-vs-switch-tasks.md b/docs/docs/how-tos/Tasks/dynamic-vs-switch-tasks.md
index f9e1e4e6d0..e81327d494 100644
--- a/docs/docs/how-tos/Tasks/dynamic-vs-switch-tasks.md
+++ b/docs/docs/how-tos/Tasks/dynamic-vs-switch-tasks.md
@@ -6,11 +6,11 @@ sidebar_position: 1
Learn more about
-1. [Dynamic Tasks](../reference-docs/dynamic-task)
-2. [Switch Tasks](../reference-docs/switch-task)
+1. [Dynamic Tasks](/reference-docs/dynamic-task.html)
+2. [Switch Tasks](/reference-docs/switch-task.html)
Dynamic Tasks are useful in situations when need to run a task of which the task type is determined at runtime instead
-of during the configuration. It is similar to the [SWITCH](../reference-docs/switch-task) use case but with `DYNAMIC`
+of during the configuration. It is similar to the [SWITCH](/reference-docs/switch-task.html) use case but with `DYNAMIC`
we won't need to preconfigure all case options in the workflow definition itself. Instead, we can mark the task
as `DYNAMIC` and determine which underlying task does it run during the workflow execution itself.
diff --git a/docs/docs/how-tos/Tasks/extending-system-tasks.md b/docs/docs/how-tos/Tasks/extending-system-tasks.md
new file mode 100644
index 0000000000..5661d040c2
--- /dev/null
+++ b/docs/docs/how-tos/Tasks/extending-system-tasks.md
@@ -0,0 +1,98 @@
+# Extending System Tasks
+
+[System tasks](/configuration/systask.html) allow Conductor to run simple tasks on the server - removing the need to build (and deploy) workers for basic tasks. This allows for automating more mundane tasks without building specific microservices for them.
+
+However, sometimes it might be necessary to add additional parameters to a System Task to gain the behavior that is desired.
+
+## Example HTTP Task
+
+```json
+{
+ "name": "get_weather_90210",
+ "version": 1,
+ "tasks": [
+ {
+ "name": "get_weather_90210",
+ "taskReferenceName": "get_weather_90210",
+ "inputParameters": {
+ "http_request": {
+ "uri": "https://weatherdbi.herokuapp.com/data/weather/90210",
+ "method": "GET",
+ "connectionTimeOut": 1300,
+ "readTimeOut": 1300
+ }
+ },
+ "type": "HTTP",
+ "decisionCases": {},
+ "defaultCase": [],
+ "forkTasks": [],
+ "startDelay": 0,
+ "joinOn": [],
+ "optional": false,
+ "defaultExclusiveJoinTask": [],
+ "asyncComplete": false,
+ "loopOver": []
+ }
+ ],
+ "inputParameters": [],
+ "outputParameters": {
+ "data": "${get_weather_ref.output.response.body.currentConditions.comment}"
+ },
+ "schemaVersion": 2,
+ "restartable": true,
+ "workflowStatusListenerEnabled": false,
+ "ownerEmail": "conductor@example.com",
+ "timeoutPolicy": "ALERT_ONLY",
+ "timeoutSeconds": 0,
+ "variables": {},
+ "inputTemplate": {}
+}
+
+```
+
+This very simple workflow has a single HTTP Task inside. No parameters need to be passed, and when run, the HTTP task will return the weather in Beverly Hills, CA (Zip code = 90210).
+
+> This API has a very slow response time. In the HTTP task, the connection is set to time out after 1300ms, which is *too short* for this API, resulting in a timeout. This API *will* work if we allowed for a longer timeout, but in order to demonstrate adding retries to the HTTP Task, we will artificially force the API call to fail.
+
+When this workflow is run - it fails, as expected.
+
+Now, sometimes an API call might fail due to an issue on the remote server, and retrying the call will result in a response. With many Conductor tasks, ```retryCount```, ```retryDelaySeconds``` and ```retryLogic``` fields can be applied to retry the worker (with the desired parameters).
+
+By default, the [HTTP Task](/reference-docs/http-task.html) does not have ```retryCount```, ```retryDelaySeconds``` or ```retryLogic``` built in. Attempting to add these parameters to a HTTP Task results in an error.
+
+## The Solution
+
+We can create a task with the same name with the desired parameters. Defining the following task (note that the ```name``` is identical to the one in the workflow):
+
+```json
+{
+
+ "createdBy": "",
+ "name": "get_weather_90210",
+ "description": "editing HTTP task",
+ "retryCount": 3,
+ "timeoutSeconds": 5,
+ "inputKeys": [],
+ "outputKeys": [],
+ "timeoutPolicy": "TIME_OUT_WF",
+ "retryLogic": "FIXED",
+ "retryDelaySeconds": 5,
+ "responseTimeoutSeconds": 5,
+ "inputTemplate": {},
+ "rateLimitPerFrequency": 0,
+ "rateLimitFrequencyInSeconds": 1
+}
+
+```
+
+We've added the three parameters: ```retryCount: 3, retryDelaySeconds: 5, retryLogic: FIXED```
+
+The ```get_weather_90210``` task will now run 4 times (it will fail once, and then retry 3 times), with a ```FIXED``` 5 second delay between attempts.
+
+Re-running the task (and looking at the timeline view) shows that this is what occurs. There are 4 attempts, with a 5 second delay between them.
+
+If we change the ```retryLogic``` to EXPONENTIAL_BACKOFF, the delay between attempts grows exponentially:
+
+1. 5*2^0 = 5 seconds
+2. 5*2^1 = 10 seconds
+3. 5*2^2 = 20 seconds
diff --git a/docs/docs/how-tos/Tasks/task-configurations.md b/docs/docs/how-tos/Tasks/task-configurations.md
index 5177833685..1fef6c7095 100644
--- a/docs/docs/how-tos/Tasks/task-configurations.md
+++ b/docs/docs/how-tos/Tasks/task-configurations.md
@@ -4,7 +4,7 @@ sidebar_position: 1
# Task Configurations
-Refer to [Task Definitions](../getting-started/concepts/tasks-and-workers#task-definitions) for details on how to configure task definitions
+Refer to [Task Definitions](/configuration/taskdef.html) for details on how to configure task definitions
### Example
@@ -31,5 +31,5 @@ Here is a task template payload with commonly used fields:
### Best Practices
-1. Refer to [Task Timeouts](./task-timeouts) for additional information on how the various timeout settings work
-2. Refer to [Monitoring Task Queues](./monitoring-task-queues) on how to monitor task queues
+1. Refer to [Task Timeouts](/how-tos/Tasks/task-timeouts.html) for additional information on how the various timeout settings work
+2. Refer to [Monitoring Task Queues](/how-tos/Tasks/monitoring-task-queues.html) on how to monitor task queues
diff --git a/docs/docs/how-tos/Tasks/updating-tasks.md b/docs/docs/how-tos/Tasks/updating-tasks.md
index b786d809a5..4978e80b37 100644
--- a/docs/docs/how-tos/Tasks/updating-tasks.md
+++ b/docs/docs/how-tos/Tasks/updating-tasks.md
@@ -39,4 +39,4 @@ fetch("http://localhost:8080/api/metadata/taskdefs", {
## Best Practices
1. You can also use the Conductor Swagger UI to update the tasks
-2. Task configurations are important attributes that controls the behavior of this task in a Workflow. Refer to [Task Configurations](task-configurations) for all the options and details'
+2. Task configurations are important attributes that controls the behavior of this task in a Workflow. Refer to [Task Configurations](/how-tos/Tasks/task-configurations.html) for all the options and details'
diff --git a/docs/docs/how-tos/Workflows/create-workflow.md b/docs/docs/how-tos/Workflows/create-workflow.md
deleted file mode 100644
index 1b5368a58b..0000000000
--- a/docs/docs/how-tos/Workflows/create-workflow.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-sidebar_position: 1
----
-
-# Creating Workflows
-
-TODO
-
-## Summary
-
-TODO
diff --git a/docs/docs/how-tos/Workflows/handling-errors.md b/docs/docs/how-tos/Workflows/handling-errors.md
index 7f8d4e20a6..b58fe1e91e 100644
--- a/docs/docs/how-tos/Workflows/handling-errors.md
+++ b/docs/docs/how-tos/Workflows/handling-errors.md
@@ -1,20 +1,59 @@
----
-sidebar_position: 1
----
-
# Handling Errors
When a workflow fails, there are 2 ways to handle the exception.
-## ```failureWorkflow```
+## Set ```failureWorkflow``` in Workflow Definition
+
+In your main workflow definition, you can configure a workflow to run upon failure, by adding the following parameter to the workflow:
+
+```json
+"failureWorkflow": "
+ 
Open Source
+ 
Modular
+ 
Proven
+ 
Control
+ 
Polyglot
+ 
Scalable
+ 
Service Orchestration
+ Workflow definitions are decoupled from task implementations. This allows the creation of process flows in which each individual task can be implemented + by an encapsulated microservice.
+Desiging a workflow orchestrator that is resilient and horizontally scalable is not a simple problem. At Netflix we have developed a solution in Conductor.
+
Service Choreography
+ 
+