Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

Add an API to view current scheduled jobs #80

Closed
wants to merge 63 commits into from
Closed

Add an API to view current scheduled jobs #80

wants to merge 63 commits into from

Conversation

ftianli-amzn
Copy link

@ftianli-amzn ftianli-amzn commented Oct 21, 2020

Issue #, if available:
#73
Description of changes:

(This is a draft, and some details are going to be written down.)
Create a new API to get the information for all the in-memory jobs in every job index:

GET /_opendistro/_jobscheduler/{nodeId}/jobs
GET /_opendistro/_jobscheduler/jobs

The jobs information contains:

  • The last execution time
  • The expected next execution time
  • The actual duration (in milliseconds) to next execution time

The implementation is Inspired by the stats API in ODFE Alerting plugin (https://github.com/opendistro-for-elasticsearch/alerting/blob/master/core/src/main/kotlin/com/amazon/opendistroforelasticsearch/alerting/core/resthandler/RestScheduledJobStatsHandler.kt)

The structure of the response is:

Response Body
+--- _nodes : Contains statistics about the number of nodes selected by the request. 
+--- cluster_name : Name of the cluster.: 
+--- nodes : Contains the job information for the nodes selected by the request.
         +--- Node ID
         |        +--- Job Index Name
         |        |        +--- Job ID
         |        |        |      +---  last_execution_time : The actual last running time of the job. (Unix time)
         |        |        |      +---  next_execution_time : The expected next running time of the job. (Unix time)
         |        |        |      +---  duration_to_next_execution_in_ms : The actual duration to next job runs
         |        |        \--- Job ID 2
         |        \--- Job Index Name 2
         \---Node ID 2
...

Samples:
When there is no scheduled job:

{
    "_nodes": {
        "total": 1,
        "successful": 1,
        "failed": 0
    },
    "cluster_name": "integTest",
    "nodes": {
        "PPgOgvJkRf-agf7SXAkgIw": {
            ".scheduler_sample_extension": {}
        }
    }
}

When there are multiple scheduled jobs:

{
    "_nodes": {
        "total": 1,
        "successful": 1,
        "failed": 0
    },
    "cluster_name": "integTest",
    "nodes": {
        "PPgOgvJkRf-agf7SXAkgIw": {
            ".scheduler_sample_extension": {
                "kibana-job-01": {
                    "last_execution_time": 1603240231680,
                    "next_execution_time": 1603240291675,
                    "duration_to_next_execution_in_ms": 59409
                },
                "kibana-job-02": {
                    "next_execution_time": 1603240243523,
                    "duration_to_next_execution_in_ms": 11257
                }
            }
        }
    }
}

Todo: unit tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

hyandell and others added 30 commits March 8, 2019 09:38
Update code of conduct link
* Support ES7.0, use primary_term and seq_no for job doc versioning
* support scheduledJob with locks.

* Use ClusterService instead of Client to check the index exists.

address some minor comment.

* Updated through the comment.

1. Move the LockService in the SPI package
1. Added deleteLock to test.
1. Updated naming from job_type to job_index_name
1. updated delete lock to be safe with non existing index name.

* Use ClusterStateService for lockIndexExist

* Update acquireLock to take JobParam and JobContext

* add some documentation to LockService
* Update release-note for OpenDistro 1.0.0 release
* Add Debian package build
* Update Rpm build
* Filter clusterChangedEvent when doing sweeping
* Format code
* Update release lock log level
* Update sweeper version compare log level
* update ScheduledJobParser to use the JobDocVersion
* support ES 7.1.1 and upgrade to Opendistro 1.1
* Add spi subproject maven publish task
* Support Elasticsearch 7.2
* Support ES version 7.4.2

* change LockServiceIT timing

* give unique table name and job id for each test

* upload artifacts if failed

* include spi build upload artifacts
* Updates release notes

* Changes runs-on to Mac for github workflow
* add jitter in job parameters
* add jitter limit
* set default jitter limit as 0.6; add more comments
* Create push-jdbc-jar.yml

* Update and rename push-jdbc-jar.yml to push-job-sched-jar.yml

* Update push-job-sched-jar.yml

* Update push-job-sched-jar.yml

* Update push-job-sched-jar.yml
gaiksaya and others added 27 commits May 15, 2020 09:44
* Support ES version 7.7.0

* A hack to reorder the plugins installation in testCluster

* Update the requirement of JDK 14 in README and workflow files

* Update the Release Notes
… interface in Elasticsearch 7.9.0 (#67)

* Support Elasticsearch 7.9.0
* Update the release note for 1.10.0.0 to satisfy the unified format
…e notes (#68)

* Add a config file to organize all labeled PRs and place under headers in the release notes
* Add a workflow to create draft release notes under ‘Releases’ tab in GitHub
* Reformat old release notes to meet the unified standard
* Making Schedule interface Writeable and relevant changes for CronScheduler and IntervalScheduler classes.

- This is needed for converting Anomaly Detection API's to integrate with RestActions.

* Removing intervalInMillis from Transport

* Adding Unit Tests for Cron and Interval Schedules
* Add a function to renew lock in LockService class and add integration test for it
* return the renewed lock instead of a boolean value when calling 'renewLock()'
* Bump plugin version and update release notes
@ftianli-amzn
Copy link
Author

I'm thinking if it's better to copy all the functionality in the stats API of Alerting to here.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.