[PROPOSAL] Relocate Job Scheduler into core as a native plugin

[joshpalis]

Based on this discussion, Job Scheduler will be relocated to core as a native plugin. This issue shall be used to track the associated work items and discuss any questions / concerns.

Draft PR to relocate Job Scheduler to native plugins : #5453

Update :

Previously, this change has been slated for the 2.6.0 release now that we have ensured that rolling/restart upgrades are not detrimentally affected by this relocation. However, upon consolidating the necessary build.gradle modifications for dependent plugins [3], the move to relocate Job Scheduler to native plugins has been determined to be a breaking change. In our efforts to adhere to semantic versioning, we shall relocate Job Scheduler to OpenSearch core as a native plugin on the 3.x branch of OpenSearch, and in order to backport this change to 2.x, we shall forgo any dependent plugin
build.gradle modifications for 2.x and instead publish Job Scheduler artifacts to two separate Maven coordinates.

[1] Native Job Scheduler Maven Coordinates :

• org.opensearch.plugin:opensearch-job-scheduler-(3 digit version number)
• org.opensearch.plugin:opensearch-job-scheduler-spi-(3 digit version number)

[2] Original Job Scheduler Maven Coordinates :

• org.opensearch:opensearch-job-scheduler-(4 digit version number)
• org.orpensearch:opensearch-job-scheduler-spi-(4 digit version number)

The original Job Scheduler Repository shall consume JS Native plugin artifacts[1] and repackage and publish these artifacts to the original Job Scheduler Maven coordinates[2]. By doing so, we can achieve both the relocation of Job Scheduler into native plugins, whilst adhering to semantic versioning by circumnavigating any modifications to how dependent plugins consume Job Scheduler for 2.x.

Build and Assemble Workflow Modifications

Update Now that the Job Scheduler relocation to native plugins will target the 3.0.0 release, modifications for the input manifest yml schema will now only be necessary for 3.0.0 and onwards.

Since Job Scheduler acts as a dependency for other plugins (AD, IM, Reporting), relocating this project within core as a native plugin will necessitate changes to the yml schema of the input manifest used to assemble the full distribution of OpenSearch.

Discussions for the input manifest changes has been started here .

Dependent Plugin Modifications
Now that the modifications for dependent plugin build.gradles[3] has been determined to be a breaking change, moving forward, dependent plugins will have no modifications for 2.x in order to adhere to semantic versioning, however this change will be raised for 3.x.

Simply put, for 2.x, dependent plugins will still consume Job Scheduler artifacts from the original maven coordinates[2], for 3.x, dependent plugins will consume Job Scheduler artifacts from the native JS maven coordinates[1].

• [3] Job Scheduler SPI artifacts are currently uploaded with and consumed from the following maven coordinates org.opensearch:opensearch-job-scheduler-spi:${job_scheduler_version}. Upon relocation, artifacts will be uploaded as part of the OpenSearch component with the maven coordinates org.opensearch.plugin:opensearch-job-scheduler-spi:${job_scheduler_version}

Plans to support previous versions of Job Scheduler
In order to support security patches to previous versions of Job Scheduler, upon relocation, the original Job Scheduler repository will not be archived.

[joshpalis]

@prudhvigodithi @nknize @dblock @vibrantvarun @praveensameneni @sean-zheng-amazon @anirudha @bbarani @gaiksaya @peterzhuamazon

For the dependent plugin owners, please let us know if you have any questions / concerns regarding items [1] and [2]. We shall move forward with raising these PRs starting December 5th.

[praveensameneni]

LGTM

[minalsha]

@sean-zheng-amazon @anirudha can i please get your signoff here?

[sean-zheng-amazon]

I think this is a great move, long waiting for. Just one thing to confirm, the gradle change is the only change needed from plugin side right? and also assumed this shouldn't cause any backward compatibility issue.

[joshpalis]

Hi @sean-zheng-amazon, yes the aforementioned gradle changes are the only modifications needed from the plugin side. As for any backward compatibility issues that may come up (if any), we shall also take care of resolving.

[rupal-bq]

Reporting signing off

[prudhvigodithi]

Hey @saratvemulapalli @joshpalis quick question from the comment, what about for the users who do upgrade/rolling-upgrade from 1.x to 2.x? In this case 1.x the job scheduler is pre-installed with the bundle, but for 2.x (where job-scheduler is now a native plugin) the job scheduler should be explicitly installed, how do we plan to handle this? Can this be a red flag and break anything in the running cluster?
Also Upgrade from one major/minor to next major/minor version where job-scheduler is now a native plugin would also fall under same scenario.
@bbarani @peterzhuamazon @dblock @CEHENKLE

curl -XGET https://localhost:9200/_cat/plugins -ku 'admin:admin'
039cd4094188 opensearch-alerting                  2.4.0.0
039cd4094188 opensearch-anomaly-detection         2.4.0.0
039cd4094188 opensearch-asynchronous-search       2.4.0.0
039cd4094188 opensearch-cross-cluster-replication 2.4.0.0
039cd4094188 opensearch-geospatial                2.4.0.0
039cd4094188 opensearch-index-management          2.4.0.0
039cd4094188 opensearch-job-scheduler             2.4.0.0
039cd4094188 opensearch-knn                       2.4.0.0
039cd4094188 opensearch-ml                        2.4.0.0
039cd4094188 opensearch-neural-search             2.4.0.0
039cd4094188 opensearch-notifications             2.4.0.0
039cd4094188 opensearch-notifications-core        2.4.0.0
039cd4094188 opensearch-observability             2.4.0.0
039cd4094188 opensearch-performance-analyzer      2.4.0.0
039cd4094188 opensearch-reports-scheduler         2.4.0.0
039cd4094188 opensearch-security                  2.4.0.0
039cd4094188 opensearch-security-analytics        2.4.0.0
039cd4094188 opensearch-sql                       2.4.0.0

[saratvemulapalli]

@prudhvigodithi it really doesnt matter how the plugin is installed (i.e native vs non-native), after the installation the code is the same from OpenSearch point of view. So rolling upgrade will work as it was before.
Customers using non native JS with OpenSearch 1.x should be able to upgrade to native JS with OpenSearch 2.x with no problems.
The only difference between native and non-native plugins is:

• They are located within OpenSearch repo.
• Can be installed seamlessly.
  I hope that helps.

[prudhvigodithi]

Hey @saratvemulapalli thanks for the info, what I was curious as part of the rolling upgrade is, since the native plugins are default not installed in the bundle, how do we plan to handle this? Do we inform to re-install the plugin manually after the upgrade is done?
As the old version would have the job-scheduler installed (that comes out of the box with the bundle) but not when its a native plugin.

[minalsha]

Hi @prudhvigodithi one clarification: If JS is part of core/modules folder, then it is installed out of the box and it becomes a mandated install for customers who may not even use it. We are proposing to move Job Scheduler as a native plugin so that it is part of the Core/Plugins folder since we do not want it to be a mandated installation and customers can determine if they want to install it or not. (All the plugins in the Plugins folder are optional and can be installed optionally by a user). @saratvemulapalli / @joshpalis please confirm if I am mistaken. Thanks

[joshpalis]

@minalsha You are correct, native plugins are optionally installed. @prudhvigodithi is referring to the full bundle of OpenSearch, which installs all the components listed in the input build manifest, which currently includes Job Scheduler. While native plugins are part of the OpenSearch component, these are not installed by default but their artifacts are built and published as part of build/assemble. Once Job Scheduler is relocated to native plugins, since it is a dependency of other component plugins, it is necessary for Job Scheduler to be pre-installed, despite it being a native plugin. This issue is being addressed here : opensearch-project/opensearch-build#2849

[saratvemulapalli]

Hey @saratvemulapalli thanks for the info, what I was curious as part of the rolling upgrade is, since the native plugins are default not installed in the bundle, how do we plan to handle this? Do we inform to re-install the plugin manually after the upgrade is done? As the old version would have the job-scheduler installed (that comes out of the box with the bundle) but not when its a native plugin.

@prudhvigodithi our distributions will have JS installed (doesn't matter if its installed via native/non-native), because there are other dependent plugins. i.e we should install JS within the bundle, if not we cannot ship AD, ISM etc. So, rolling upgrade won't be any different.
OpenSearch min distribution will not have JS installed by default, if customers need it they could install it.

[joshpalis]

In order to ensure that relocating Job Scheduler to native plugins does not affect rolling upgrades we will perform rolling upgrades for 2.x -> 3.x and 1.x -> 2.x. Each experiment will first use the original cluster to start an Anomaly Detector and upon completing the rolling upgrade, we will utilize the Get Detector API to retrieve real time analysis information for this particular detector. By retrieving real time analysis information after executing the rolling upgrade, we can confirm that Job Scheduler is still working as a native plugin

Rolling Upgrade Experiment (2.x -> 3.x)

2.x cluster configuration:

• 3 nodes (1 manager, 2 data)
• installed opensearch-anomaly-detection
• installed opensearch-job-scheduler (Plugin)

2.x Logs

• clustermanagernode.log.log
• datanode1.log
• datanode2.log

3.x cluster configuration:

• 3 nodes (1 manager, 2 data)
• installed opensearch-anomaly-detection
• installed opensearch-job-scheduler (Native Plugin)

3.x Logs

• clustermanagernode.log
• datanode1.log
• datanode2.log

Result :
Success

Creating Anomaly Detector (ID : J7N6CIUB02zLoZpx4Nd8 )

curl -XPOST "localhost:9200/_plugins/_anomaly_detection/detectors?pretty" -H "Content-Type:application/json" --data '{"name":"test-detector","description":"Test detector","time_field":"timestamp","indices":["server_log*"],"feature_attributes":[{"feature_name":"test","feature_enabled":true,"aggregation_query":{"test":{"sum":{"field":"value"}}}}],"filter_query":{"bool":{"filter":[{"range":{"value":{"gt":1}}}],"adjust_pure_negative":true,"boost":1}},"detection_interval":{"period":{"interval":1,"unit":"Minutes"}},"window_delay":{"period":{"interval":1,"unit":"Minutes"}},"result_index":"opensearch-ad-plugin-result-test"}'
{
  "_id" : "J7N6CIUB02zLoZpx4Nd8",
  "_version" : 1,
  "_seq_no" : 0,
  "anomaly_detector" : {
    "name" : "test-detector",
    "description" : "Test detector",
    "time_field" : "timestamp",
    "indices" : [
      "server_log*"
    ],
    "filter_query" : {
      "bool" : {
        "filter" : [
          {
            "range" : {
              "value" : {
                "from" : 1,
                "to" : null,
                "include_lower" : false,
                "include_upper" : true,
                "boost" : 1.0
              }
            }
          }
        ],
        "adjust_pure_negative" : true,
        "boost" : 1.0
      }
    },
    "detection_interval" : {
      "period" : {
        "interval" : 1,
        "unit" : "Minutes"
      }
    },
    "window_delay" : {
      "period" : {
        "interval" : 1,
        "unit" : "Minutes"
      }
    },
    "shingle_size" : 8,
    "schema_version" : 0,
    "feature_attributes" : [
      {
        "feature_id" : "JrN6CIUB02zLoZpx3ddb",
        "feature_name" : "test",
        "feature_enabled" : true,
        "aggregation_query" : {
          "test" : {
            "sum" : {
              "field" : "value"
            }
          }
        }
      }
    ],
    "last_update_time" : 1670884548730,
    "detector_type" : "SINGLE_ENTITY",
    "result_index" : "opensearch-ad-plugin-result-test"
  },
  "_primary_term" : 1
}

Starting Detector (ID : J7N6CIUB02zLoZpx4Nd8)

curl -X POST "localhost:9200/_plugins/_anomaly_detection/detectors/J7N6CIUB02zLoZpx4Nd8/_start"
{"_id":"J7N6CIUB02zLoZpx4Nd8","_version":1,"_seq_no":0,"_primary_term":1}%

Retrieving Real Time Analysis Information for Detector (ID : J7N6CIUB02zLoZpx4Nd8) post upgrade

curl -X GET "localhost:9200/_plugins/_anomaly_detection/detectors/J7N6CIUB02zLoZpx4Nd8?job=true"
{
  "_id": "J7N6CIUB02zLoZpx4Nd8",
  "_version": 1,
  "_primary_term": 1,
  "_seq_no": 0,
  "anomaly_detector": {
    "name": "test-detector",
    "description": "Test detector",
    "time_field": "timestamp",
    "indices": [
      "server_log*"
    ],
    "filter_query": {
      "bool": {
        "filter": [
          {
            "range": {
              "value": {
                "from": 1,
                "to": null,
                "include_lower": false,
                "include_upper": true,
                "boost": 1
              }
            }
          }
        ],
        "adjust_pure_negative": true,
        "boost": 1
      }
    },
    "detection_interval": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "shingle_size": 8,
    "schema_version": 0,
    "feature_attributes": [
      {
        "feature_id": "JrN6CIUB02zLoZpx3ddb",
        "feature_name": "test",
        "feature_enabled": true,
        "aggregation_query": {
          "test": {
            "sum": {
              "field": "value"
            }
          }
        }
      }
    ],
    "last_update_time": 1670884548730,
    "detector_type": "SINGLE_ENTITY",
    "result_index": "opensearch-ad-plugin-result-test"
  },
  "anomaly_detector_job": {
    "name": "J7N6CIUB02zLoZpx4Nd8",
    "schedule": {
      "interval": {
        "start_time": 1670884573449,
        "period": 1,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "enabled": true,
    "enabled_time": 1670884573449,
    "last_update_time": 1670884573449,
    "lock_duration_seconds": 60,
    "result_index": "opensearch-ad-plugin-result-test"
  }
}

Rolling Upgrade Experiment (1.x -> 2.x)

1.x cluster configuration:

• 3 nodes (1 manager, 2 data)
• installed opensearch-anomaly-detection
• installed opensearch-job-scheduler (Plugin)

1.x Logs

• clustermasternode.log
• datanode1.log
• datanode2.log

2.x cluster configuration:

• 3 nodes (1 manager, 2 data)
• installed opensearch-anomaly-detection
• installed opensearch-job-scheduler (Native Plugin)

2.x Logs

• clustermanagernode.log
• datanode1.log
• datanode2.log

Result :
Success

Creating Anomaly Detector (ID : oBmvDYUB0LodPD1ay_uK )

curl -XPOST "localhost:9200/_plugins/_anomaly_detection/detectors?pretty" -H "Content-Type:application/json" --data '{"name":"test-detector","description":"Test detector","time_field":"timestamp","indices":["server_log*"],"feature_attributes":[{"feature_name":"test","feature_enabled":true,"aggregation_query":{"test":{"sum":{"field":"value"}}}}],"filter_query":{"bool":{"filter":[{"range":{"value":{"gt":1}}}],"adjust_pure_negative":true,"boost":1}},"detection_interval":{"period":{"interval":1,"unit":"Minutes"}},"window_delay":{"period":{"interval":1,"unit":"Minutes"}},"result_index":"opensearch-ad-plugin-result-test"}'
{
  "_id" : "oBmvDYUB0LodPD1ay_uK",
  "_version" : 1,
  "_seq_no" : 0,
  "anomaly_detector" : {
    "name" : "test-detector",
    "description" : "Test detector",
    "time_field" : "timestamp",
    "indices" : [
      "server_log*"
    ],
    "filter_query" : {
      "bool" : {
        "filter" : [
          {
            "range" : {
              "value" : {
                "from" : 1,
                "to" : null,
                "include_lower" : false,
                "include_upper" : true,
                "boost" : 1.0
              }
            }
          }
        ],
        "adjust_pure_negative" : true,
        "boost" : 1.0
      }
    },
    "detection_interval" : {
      "period" : {
        "interval" : 1,
        "unit" : "Minutes"
      }
    },
    "window_delay" : {
      "period" : {
        "interval" : 1,
        "unit" : "Minutes"
      }
    },
    "shingle_size" : 8,
    "schema_version" : 0,
    "feature_attributes" : [
      {
        "feature_id" : "nxmvDYUB0LodPD1ayPuC",
        "feature_name" : "test",
        "feature_enabled" : true,
        "aggregation_query" : {
          "test" : {
            "sum" : {
              "field" : "value"
            }
          }
        }
      }
    ],
    "last_update_time" : 1670971902856,
    "detector_type" : "SINGLE_ENTITY",
    "result_index" : "opensearch-ad-plugin-result-test"
  },
  "_primary_term" : 1
}

Starting Detector (ID : oBmvDYUB0LodPD1ay_uK )

curl -X POST "localhost:9200/_plugins/_anomaly_detection/detectors/oBmvDYUB0LodPD1ay_uK/_start"
{"_id":"oBmvDYUB0LodPD1ay_uK","_version":1,"_seq_no":0,"_primary_term":1}% 

Retrieving Real Time Analysis Information for Detector (ID : oBmvDYUB0LodPD1ay_uK) post upgrade

curl -X GET "localhost:9200/_plugins/_anomaly_detection/detectors/oBmvDYUB0LodPD1ay_uK"
{
  "_id": "oBmvDYUB0LodPD1ay_uK",
  "_version": 1,
  "_primary_term": 1,
  "_seq_no": 0,
  "anomaly_detector": {
    "name": "test-detector",
    "description": "Test detector",
    "time_field": "timestamp",
    "indices": [
      "server_log*"
    ],
    "filter_query": {
      "bool": {
        "filter": [
          {
            "range": {
              "value": {
                "from": 1,
                "to": null,
                "include_lower": false,
                "include_upper": true,
                "boost": 1
              }
            }
          }
        ],
        "adjust_pure_negative": true,
        "boost": 1
      }
    },
    "detection_interval": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "shingle_size": 8,
    "schema_version": 0,
    "feature_attributes": [
      {
        "feature_id": "nxmvDYUB0LodPD1ayPuC",
        "feature_name": "test",
        "feature_enabled": true,
        "aggregation_query": {
          "test": {
            "sum": {
              "field": "value"
            }
          }
        }
      }
    ],
    "last_update_time": 1670971902856,
    "detector_type": "SINGLE_ENTITY",
    "result_index": "opensearch-ad-plugin-result-test"
  }
}

[joshpalis]

In order to ensure that relocating Job Scheduler to native plugins does not affect restart upgrades we will perform restart upgrades for 1.x -> 2.x and 2.x -> 3.x. Similarly, each experiment will first use the original cluster to start an Anomaly Detector and upon completing the restart upgrade, we will utilize the Get Detector API to retrieve real time analysis information for this particular detector. By retrieving real time analysis information after executing the rolling upgrade, we can confirm that Job Scheduler is still working as a native plugin

Restart Upgrade Experiment (1.x -> 2.x)

1.x cluster configuration:

• 1 node
• installed opensearch-anomaly-detection
• installed opensearch-job-scheduler (Plugin)

1.x Log

• opensearch.log

2.x cluster configuration:

• 1 node
• installed opensearch-anomaly-detection
• installed opensearch-job-scheduler (Native Plugin)

2.x Log

• opensearch.log

Result :
Success

Creating Anomaly Detector (ID : NwnzDYUBxGSlphqwAlxq )

curl -XPOST "localhost:9200/_plugins/_anomaly_detection/detectors?pretty" -H "Content-Type:application/json" --data '{"name":"test-detector","description":"Test detector","time_field":"timestamp","indices":["server_log*"],"feature_attributes":[{"feature_name":"test","feature_enabled":true,"aggregation_query":{"test":{"sum":{"field":"value"}}}}],"filter_query":{"bool":{"filter":[{"range":{"value":{"gt":1}}}],"adjust_pure_negative":true,"boost":1}},"detection_interval":{"period":{"interval":10,"unit":"Minutes"}},"window_delay":{"period":{"interval":10,"unit":"Minutes"}},"result_index":"opensearch-ad-plugin-result-test"}'
{
  "_id" : "NwnzDYUBxGSlphqwAlxq",
  "_version" : 1,
  "_seq_no" : 0,
  "anomaly_detector" : {
    "name" : "test-detector",
    "description" : "Test detector",
    "time_field" : "timestamp",
    "indices" : [
      "server_log*"
    ],
    "filter_query" : {
      "bool" : {
        "filter" : [
          {
            "range" : {
              "value" : {
                "from" : 1,
                "to" : null,
                "include_lower" : false,
                "include_upper" : true,
                "boost" : 1.0
              }
            }
          }
        ],
        "adjust_pure_negative" : true,
        "boost" : 1.0
      }
    },
    "detection_interval" : {
      "period" : {
        "interval" : 10,
        "unit" : "Minutes"
      }
    },
    "window_delay" : {
      "period" : {
        "interval" : 10,
        "unit" : "Minutes"
      }
    },
    "shingle_size" : 8,
    "schema_version" : 0,
    "feature_attributes" : [
      {
        "feature_id" : "NgnzDYUBxGSlphqwAVwu",
        "feature_name" : "test",
        "feature_enabled" : true,
        "aggregation_query" : {
          "test" : {
            "sum" : {
              "field" : "value"
            }
          }
        }
      }
    ],
    "last_update_time" : 1670976307817,
    "detector_type" : "SINGLE_ENTITY",
    "result_index" : "opensearch-ad-plugin-result-test"
  },
  "_primary_term" : 1
}

Starting Detector (ID : NwnzDYUBxGSlphqwAlxq )

curl -X POST "localhost:9200/_plugins/_anomaly_detection/detectors/NwnzDYUBxGSlphqwAlxq/_start"
{"_id":"NwnzDYUBxGSlphqwAlxq","_version":1,"_seq_no":0,"_primary_term":1}%   

Retrieving Real Time Analysis Information for Detector (ID : NwnzDYUBxGSlphqwAlxq) post upgrade

 curl -X GET "localhost:9200/_plugins/_anomaly_detection/detectors/NwnzDYUBxGSlphqwAlxq?pretty"
{
  "_id" : "NwnzDYUBxGSlphqwAlxq",
  "_version" : 1,
  "_primary_term" : 1,
  "_seq_no" : 0,
  "anomaly_detector" : {
    "name" : "test-detector",
    "description" : "Test detector",
    "time_field" : "timestamp",
    "indices" : [
      "server_log*"
    ],
    "filter_query" : {
      "bool" : {
        "filter" : [
          {
            "range" : {
              "value" : {
                "from" : 1,
                "to" : null,
                "include_lower" : false,
                "include_upper" : true,
                "boost" : 1.0
              }
            }
          }
        ],
        "adjust_pure_negative" : true,
        "boost" : 1.0
      }
    },
    "detection_interval" : {
      "period" : {
        "interval" : 10,
        "unit" : "Minutes"
      }
    },
    "window_delay" : {
      "period" : {
        "interval" : 10,
        "unit" : "Minutes"
      }
    },
    "shingle_size" : 8,
    "schema_version" : 0,
    "feature_attributes" : [
      {
        "feature_id" : "NgnzDYUBxGSlphqwAVwu",
        "feature_name" : "test",
        "feature_enabled" : true,
        "aggregation_query" : {
          "test" : {
            "sum" : {
              "field" : "value"
            }
          }
        }
      }
    ],
    "last_update_time" : 1670976307817,
    "detector_type" : "SINGLE_ENTITY",
    "result_index" : "opensearch-ad-plugin-result-test"
  }
}

Restart Upgrade Experiment (2.x -> 3.x)

2.x cluster configuration:

• 1 node
• installed opensearch-anomaly-detection
• installed opensearch-job-scheduler (Plugin)

2.x Log

• opensearch.log

3.x cluster configuration:

• 1 node
• installed opensearch-anomaly-detection
• installed opensearch-job-scheduler (Native Plugin)

3.x Log

• opensearch.log

Result :
Success

Creating Anomaly Detector (ID : 5LoBDoUBG1y14YzvlquP)

curl -XPOST "localhost:9200/_plugins/_anomaly_detection/detectors?pretty" -H "Content-Type:application/json" --data '{"name":"test-detector","description":"Test detector","time_field":"timestamp","indices":["server_log*"],"feature_attributes":[{"feature_name":"test","feature_enabled":true,"aggregation_query":{"test":{"sum":{"field":"value"}}}}],"filter_query":{"bool":{"filter":[{"range":{"value":{"gt":1}}}],"adjust_pure_negative":true,"boost":1}},"detection_interval":{"period":{"interval":10,"unit":"Minutes"}},"window_delay":{"period":{"interval":10,"unit":"Minutes"}},"result_index":"opensearch-ad-plugin-result-test"}'
{
  "_id" : "5LoBDoUBG1y14YzvlquP",
  "_version" : 1,
  "_seq_no" : 0,
  "anomaly_detector" : {
    "name" : "test-detector",
    "description" : "Test detector",
    "time_field" : "timestamp",
    "indices" : [
      "server_log*"
    ],
    "filter_query" : {
      "bool" : {
        "filter" : [
          {
            "range" : {
              "value" : {
                "from" : 1,
                "to" : null,
                "include_lower" : false,
                "include_upper" : true,
                "boost" : 1.0
              }
            }
          }
        ],
        "adjust_pure_negative" : true,
        "boost" : 1.0
      }
    },
    "detection_interval" : {
      "period" : {
        "interval" : 10,
        "unit" : "Minutes"
      }
    },
    "window_delay" : {
      "period" : {
        "interval" : 10,
        "unit" : "Minutes"
      }
    },
    "shingle_size" : 8,
    "schema_version" : 0,
    "feature_attributes" : [
      {
        "feature_id" : "47oBDoUBG1y14YzvlatY",
        "feature_name" : "test",
        "feature_enabled" : true,
        "aggregation_query" : {
          "test" : {
            "sum" : {
              "field" : "value"
            }
          }
        }
      }
    ],
    "last_update_time" : 1670977263246,
    "detector_type" : "SINGLE_ENTITY",
    "result_index" : "opensearch-ad-plugin-result-test"
  },
  "_primary_term" : 1
}

Starting Detector (ID : 5LoBDoUBG1y14YzvlquP)

curl -X POST "localhost:9200/_plugins/_anomaly_detection/detectors/5LoBDoUBG1y14YzvlquP/_start"
{"_id":"5LoBDoUBG1y14YzvlquP","_version":1,"_seq_no":0,"_primary_term":1}

Retrieving Real Time Analysis Information for Detector (ID : 5LoBDoUBG1y14YzvlquP) post upgrade

curl -X GET "localhost:9200/_plugins/_anomaly_detection/detectors/5LoBDoUBG1y14YzvlquP?pretty"
{
  "_id" : "5LoBDoUBG1y14YzvlquP",
  "_version" : 1,
  "_primary_term" : 1,
  "_seq_no" : 0,
  "anomaly_detector" : {
    "name" : "test-detector",
    "description" : "Test detector",
    "time_field" : "timestamp",
    "indices" : [
      "server_log*"
    ],
    "filter_query" : {
      "bool" : {
        "filter" : [
          {
            "range" : {
              "value" : {
                "from" : 1,
                "to" : null,
                "include_lower" : false,
                "include_upper" : true,
                "boost" : 1.0
              }
            }
          }
        ],
        "adjust_pure_negative" : true,
        "boost" : 1.0
      }
    },
    "detection_interval" : {
      "period" : {
        "interval" : 10,
        "unit" : "Minutes"
      }
    },
    "window_delay" : {
      "period" : {
        "interval" : 10,
        "unit" : "Minutes"
      }
    },
    "shingle_size" : 8,
    "schema_version" : 0,
    "feature_attributes" : [
      {
        "feature_id" : "47oBDoUBG1y14YzvlatY",
        "feature_name" : "test",
        "feature_enabled" : true,
        "aggregation_query" : {
          "test" : {
            "sum" : {
              "field" : "value"
            }
          }
        }
      }
    ],
    "last_update_time" : 1670977263246,
    "detector_type" : "SINGLE_ENTITY",
    "result_index" : "opensearch-ad-plugin-result-test"
  }
}

[penghuo]

signoff for observability plugin. because of

1. only gradle change required.
2. observability plugin only use job-scheduler for PoC.

[joshpalis]

@penghuo @sean-zheng-amazon @praveensameneni @rupal-bq @anirudha . Please note that due to this change, while we will no longer require modifications to your respective plugin build.gradles for 2.x, we shall be raising PRs to update the maven coordinates for 3.x.

[saratvemulapalli]

@joshpalis dropping 2.5 label and adding in 2.6 as we are shooting for that train.

[bbarani]

@saratvemulapalli We should tag this issue to 3.x based on the latest updates as mentioned here?

[joshpalis]

An update regarding the effort relocate Job Scheduler into core as a native plugin.

Moving Job Scheduler into core has been determined to be a breaking change (modifies the location in which job scheduler artifacts are published to), thus, in order to adhere to semver, Job Scheduler will not be moved into native plugins until 3.0.0.

It is necessary to modify the assemble workflow for the full-bundle OpenSearch distribution, and there is an ongoing effort to determine how the pre-installation of native plugins will be supported. This discussion can be found here : opensearch-project/opensearch-build#2849