Skip to content

Latest commit

 

History

History
204 lines (161 loc) · 13.3 KB

README.en.md

File metadata and controls

204 lines (161 loc) · 13.3 KB
Raw

Logo

A distributed task scheduling and distributed computing framework

License JDK8+ Maven Central Build status Demo

online demo：http://ponfee.cn:8000/

English | 简体中文

A distributed task scheduling and distributed computing framework, in addition to having regular task scheduling functions, also provides: pausing/cancelling running tasks, resuming paused tasks, task splitting, failure retries, broadcast tasks, task dependencies, workflow tasks (DAG), separate deployment of managers and executors, Web management backend, and other capabilities.

Lightweight and easy to use, it is particularly suitable for the execution of long tasks. Powerful and reliable, it has been tested in production environments.

Architecture

  • Overall Process

Overall Process

  • Code Structure
disjob                                        # Main project①
├── disjob-admin                              # Management backend project② (Based on RuoYi framework secondary development)
├── disjob-bom                                # Maven project bom module
├── disjob-common                             # Common utility class module
├── disjob-core                               # Task scheduling related core classes (such as data models, enumeration classes, abstract layer interfaces, etc.)
├── disjob-dispatch                           # Task dispatch module
│   ├── disjob-dispatch-api                   # Abstract interface layer of task dispatch
│   ├── disjob-dispatch-http                  # Http implementation of task dispatch
│   └── disjob-dispatch-redis                 # Redis implementation of task dispatch
├── disjob-id                                 # Distributed ID generation module
├── disjob-registry                           # Server (Supervisor & Worker) registration module
│   ├── disjob-registry-api                   # Abstract interface layer of Server registration center
│   ├── disjob-registry-consul                # Server registry center: Consul implementation
│   ├── disjob-registry-database              # Server registry center: Database implementation
│   ├── disjob-registry-etcd                  # Server registry center: Etcd implementation
│   ├── disjob-registry-nacos                 # Server registry center: Nacos implementation
│   ├── disjob-registry-redis                 # Server registry center: Redis implementation
│   └── disjob-registry-zookeeper             # Server registry center: Zookeeper implementation
├── disjob-reports                            # Aggregate test coverage reports of various modules
├── disjob-samples                            # Samples project③
│   ├── disjob-samples-conf-common            # Samples common config (log4j2.xml)
│   ├── disjob-samples-conf-supervisor        # Samples Supervisor config
│   ├── disjob-samples-conf-worker            # Samples Worker config
│   ├── disjob-samples-frameless-worker       # Example of Worker deployed separately (ordinary Java-main application)
│   ├── disjob-samples-springboot-common      # Samples Spring-boot common module
│   ├── disjob-samples-springboot-merged      # Example of Supervisor and Worker deployed together (Spring-boot application)
│   ├── disjob-samples-springboot-supervisor  # Example of Supervisor deployed separately (Spring-boot application)
│   └── disjob-samples-springboot-worker      # Example of Worker deployed separately (Spring-boot application)
├── disjob-supervisor                         # Supervisor code
├── disjob-test                               # Used for auxiliary testing
└── disjob-worker                             # Worker code

Features

  • Divided into two roles: Manager (Supervisor) and Executor (Worker), Supervisor and Worker can be deployed separately
  • Supervisor and Worker discover each other through the registry center, supported: Database, Redis, Consul, Nacos, Zookeeper, Etcd
  • Supervisor is responsible for generating tasks and dispatching them to Worker for execution, supported: Redis, Http
  • Need to specify the group, Job tasks will only be dispatched to the specified group of Workers for execution
  • Provides the ability to split tasks, override the method JobSplitter#split to split many tasks, then distributed and parallel execution
  • Supports pausing and cancelling running tasks, paused tasks can be resumed for execution, failed tasks support retry
  • Supports savepoint task execution snapshot, so that manually or abnormally paused tasks can be resumed from the savepoint
  • If a task throw PauseTaskException at executing, then will pause all instance tasks (even if dispatched other worker machine tasks)
  • Supports broadcast tasks, broadcast tasks will be dispatched to all workers under the group for execution
  • Supports dependencies jobs, multiple Jobs configured with dependencies will be executed in the established dependency order
  • Supports DAG workflows, can configure jobExecutor as a complex DAG expression, such as: A->B,C,(D->E)->F,G->H
  • Provides a Web management backend, job configuration, task monitoring, etc.

Download From Maven Central

<dependency>
  <groupId>cn.ponfee</groupId>
  <artifactId>disjob-{xxx}</artifactId>
  <version>2.1.6</version>
</dependency>

Build From Source

./mvnw clean install -DskipTests -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true -U

Quick Start

  1. Import the projects to IDE (three independent projects all in one git repository)
  1. Start the following applications to form a distributed scheduling cluster

Configured different ports, run the java main class in IDE, or run the built jar package with the java -jar command.

  1. Login to the Admin
  • Open【 http://127.0.0.1:80/ 】in your browser and login with username/password: admin/123456
  • Find the Scheduling Management menu in the left sidebar to use the scheduling management function
  • Scheduling Configuration: Job configuration, including viewing, adding, modifying, deleting, triggering, disabling, etc.
  • Scheduling Instances: job trigger time generate an instance, the instance split to many tasks.
  1. Link Address
  • Admin demo:【 http://ponfee.cn:8000/ 】, username/password: disjob/123456
  • Online documentation: Under construction, please stay tuned!

User Guide

  1. The project includes two SQL scripts
  1. Change the specific implementation of disjob-registry-{xxx} and disjob-dispatch-{xxx} in the maven pom file
  • Samples project
  • Admin project
  • By default, disjob-registry-redis is used as the server registry and disjob-dispatch-http is used for task dispatch
  1. Samples project configuration files
  1. Admin project configuration files
  1. Reference for various registry configuration classes (Redis uses the configuration method provided by Spring-boot)

For example, if using Consul as the registry center, you can add the following configuration

disjob.registry.consul:
  namespace: disjob_registry
  host: localhost
  port: 8500
  token:
  1. Some embedded local servers

Contributing

If you find bugs, have better implementation solutions, new features and so on, you can submit a PR or create newIssues

Communication

Scan the QR code to add WeChat friends, note disjob, and invite you to join the group WeChat.

Wechat group

TODO List

  • Extend registry: Zookeeper, Etcd, Nacos
  • Workflow tasks (DAG)
  • Admin console: accounting, authority, permission and task manage
  • Build a project document web site
  • Monitor real-time executing logs of tasks online
  • Alarm subscribe: Email, SMS, Voice, Lark, Ding Talk, WeChat
  • Visual monitoring BI dashboard