Delete all Mesos-using code (+ misc. cleanups)

Makefile

@@ -5,24 +5,27 @@ else
 	BUILD_ENV?=$(shell hostname -f)
 endif
 
+.PHONY: venv
 venv:
 	tox -e venv
 
+.PHONY: test
 test:
 	tox
 
+.PHONY: tox_%
 tox_%:
 	tox -e $*
 
-itest:
-	tox -e integration
-
+.PHONY: docs
 docs:
 	tox -e docs
 
+.PHONY: pypi
 pypi:
 	tox -e pypi
 
+.PHONY: clean
 clean:
 	rm -rf docs/build
 	find . -name '*.pyc' -delete

README.md

@@ -6,47 +6,9 @@ Interfaces and shared infrastructure for generic task processing (also known as
 
 ### Pre-requisites
 
-+ [Docker](https://www.docker.com/get-docker)
 + [Python 3.8](https://www.python.org/downloads/)
 + [Virtualenv](https://virtualenv.pypa.io/en/stable/installation/)
 
-### Running examples
-
-[hello-world.py](/examples/hello-world/py) is a very simple annotated example that launches a task to echo `hello world`. From the root of the repository, run:
-
-    docker-compose -f examples/cluster/docker-compose.yaml \
-      run playground examples/hello-world.py
-
-This will bring up a single master, single agent Mesos cluster using [Docker Compose](https://docs.docker.com/compose/) and launch a single task which will print "hello world" to the sandbox's stdout before terminating.
-
-Other examples available include:
-+ async.py
-Example of the [async](#async) task runner.
-
-+ dynamo_persistence.py
-Example that shows how task events may be persisted to [DynamoDB](https://aws.amazon.com/dynamodb) using the `stateful` plugin.
-
-+ file_persistence.py
-Example that shows how task events may be persisted to disk using the `stateful` plugin.
-
-+ promise.py
-Example that shows how the [promise/future](#Promise/Future) task runner (not yet implemented) may be used.
-
-+ subscription.py
-Example of the [subscription](#subscription) task runner.
-
-+ sync.py
-Brief example using the [sync](#sync) task runner.
-
-+ timeout.py
-Example that shows how to timeout a task execution using the `timeout` plugin.
-
-+ retry.py
-Example that shows how to retry a task on failure using the `retry` plugin.
-
-+ task_logging.py
-Example that shows how to fetch task logs from Mesos agents using the `logging` plugin.
-
 ### Running tests
 
 From the root of the repository, run:
@@ -65,30 +27,10 @@ From the root of the repository, run:
 
 ### /plugins
 
-Plugins can be chained to create a task execution pipeline with more than one property. Please refer to persistence/retry/timeout examples.
-
-#### mesos
-Implements all required interfaces to talk to Mesos deployment. This plugin uses [PyMesos](https://github.com/douban/pymesos) to communicate with Mesos.
-
-#### timeout
-Implements an executor to timeout task execution.
-
-#### retrying
-Implements an executor to retry task execution upon failure.
-
-#### logging
-Implements an executor to retrieve task logs from Mesos agents. Note that it has to be the immediate upstream executor of the mesos executor.
-
-##### Configuration options
-
-- authentication\_principal Mesos principal
-- credential\_secret\_file path to file containing Mesos secret
-- mesos\_address host:port to connect to Mesos cluster
-- event_translator a fucntion that maps Mesos-specific events to `Event` objects
-
-#### stateful
+Plugins can be chained to create a task execution pipeline with more than one property.
 
-TODO: documentation
+#### Kubernetes
+Implements all required interfaces to talk to Kubernetes. This plugin uses [kubernetes-client](https://github.com/kubernetes-client/python) to communicate with Kubernetes.
 
 ### /runners
 

docs/source/generated/task_processing.interfaces.event.rst

@@ -1,7 +1,7 @@
-task\_processing\.interfaces\.event module
-==========================================
+task\_processing.interfaces.event module
+========================================
 
 .. automodule:: task_processing.interfaces.event
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.interfaces.persistence.rst

@@ -0,0 +1,7 @@
+task\_processing.interfaces.persistence module
+==============================================
+
+.. automodule:: task_processing.interfaces.persistence
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.interfaces.rst

@@ -1,19 +1,21 @@
-task\_processing\.interfaces package
-====================================
+task\_processing.interfaces package
+===================================
 
 Submodules
 ----------
 
 .. toctree::
+   :maxdepth: 4
 
    task_processing.interfaces.event
+   task_processing.interfaces.persistence
    task_processing.interfaces.runner
    task_processing.interfaces.task_executor
 
 Module contents
 ---------------
 
 .. automodule:: task_processing.interfaces
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.interfaces.runner.rst

@@ -1,7 +1,7 @@
-task\_processing\.interfaces\.runner module
-===========================================
+task\_processing.interfaces.runner module
+=========================================
 
 .. automodule:: task_processing.interfaces.runner
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.interfaces.task_executor.rst

@@ -1,7 +1,7 @@
-task\_processing\.interfaces\.task\_executor module
-===================================================
+task\_processing.interfaces.task\_executor module
+=================================================
 
 .. automodule:: task_processing.interfaces.task_executor
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.metrics.rst

@@ -0,0 +1,7 @@
+task\_processing.metrics module
+===============================
+
+.. automodule:: task_processing.metrics
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.kube_client.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.kube\_client module
+=======================================================
+
+.. automodule:: task_processing.plugins.kubernetes.kube_client
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.kubernetes_pod_executor.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.kubernetes\_pod\_executor module
+====================================================================
+
+.. automodule:: task_processing.plugins.kubernetes.kubernetes_pod_executor
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.rst

@@ -0,0 +1,23 @@
+task\_processing.plugins.kubernetes package
+===========================================
+
+Submodules
+----------
+
+.. toctree::
+   :maxdepth: 4
+
+   task_processing.plugins.kubernetes.kube_client
+   task_processing.plugins.kubernetes.kubernetes_pod_executor
+   task_processing.plugins.kubernetes.task_config
+   task_processing.plugins.kubernetes.task_metadata
+   task_processing.plugins.kubernetes.types
+   task_processing.plugins.kubernetes.utils
+
+Module contents
+---------------
+
+.. automodule:: task_processing.plugins.kubernetes
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.task_config.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.task\_config module
+=======================================================
+
+.. automodule:: task_processing.plugins.kubernetes.task_config
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.task_metadata.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.task\_metadata module
+=========================================================
+
+.. automodule:: task_processing.plugins.kubernetes.task_metadata
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.types.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.types module
+================================================
+
+.. automodule:: task_processing.plugins.kubernetes.types
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.utils.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.utils module
+================================================
+
+.. automodule:: task_processing.plugins.kubernetes.utils
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.rst

@@ -1,17 +1,18 @@
-task\_processing\.plugins package
-=================================
+task\_processing.plugins package
+================================
 
 Subpackages
 -----------
 
 .. toctree::
+   :maxdepth: 4
 
-    task_processing.plugins.mesos
+   task_processing.plugins.kubernetes
 
 Module contents
 ---------------
 
 .. automodule:: task_processing.plugins
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.rst

@@ -5,15 +5,26 @@ Subpackages
 -----------
 
 .. toctree::
+   :maxdepth: 4
 
-    task_processing.interfaces
-    task_processing.plugins
-    task_processing.runners
+   task_processing.interfaces
+   task_processing.plugins
+   task_processing.runners
+
+Submodules
+----------
+
+.. toctree::
+   :maxdepth: 4
+
+   task_processing.metrics
+   task_processing.task_processor
+   task_processing.utils
 
 Module contents
 ---------------
 
 .. automodule:: task_processing
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.runners.async_runner.rst

@@ -0,0 +1,7 @@
+task\_processing.runners.async\_runner module
+=============================================
+
+.. automodule:: task_processing.runners.async_runner
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.runners.promise.rst

@@ -1,7 +1,7 @@
-task\_processing\.runners\.promise module
-=========================================
+task\_processing.runners.promise module
+=======================================
 
 .. automodule:: task_processing.runners.promise
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance: