Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix for MapperException[the [enabled] parameter can't be updated for the object mapping [metadata.source_to_query_index_mapping] #1432

Merged
merged 1 commit into from
Feb 16, 2024

Conversation

sbcd90
Copy link
Collaborator

@sbcd90 sbcd90 commented Feb 16, 2024

Issue #, if available:

Description of changes:

when the source_to_query_index_mapping field is upserted without the .opendistro-alerting-config index present, the mapping for it gets corrupted.

"source_to_query_index_mapping": {
  "properties": {
    "ocsf_api_activityXnRAfI0B9Qgmh2Pzq4q4": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    }
  }
}

the correct mapping for the field is,

"source_to_query_index_mapping": {
  "type": "object",
  "enabled": false
}

this pr ensures that the upsert doesn't happen if the index is not present.

CheckList:

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…the object mapping [metadata.source_to_query_index_mapping]
@sbcd90 sbcd90 merged commit 5b3e1a2 into opensearch-project:main Feb 16, 2024
17 of 19 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Feb 16, 2024
…the object mapping [metadata.source_to_query_index_mapping] (#1432)

(cherry picked from commit 5b3e1a2)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sbcd90 pushed a commit that referenced this pull request Feb 22, 2024
…the object mapping [metadata.source_to_query_index_mapping] (#1432) (#1434)

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sbcd90 pushed a commit to sbcd90/alerting that referenced this pull request Mar 10, 2024
…ject#1307)

* Added 2.11.1 release notes.

* Added 2.11.1 release notes.

---------

(cherry picked from commit 06c1b8a)

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

fix workflow security tests in alerting (opensearch-project#1310) (opensearch-project#1311)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

Increment version to 2.12.0-SNAPSHOT (opensearch-project#1239)

Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com>
Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com>

[Backport 2.x] Reference get monitor and search monitor action / request / responses from common-utils (opensearch-project#1315)

* Use get monitor action / req / resp from common-utils

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

* Dummy commit to retrigger

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

---------

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

optimize doc-level monitor execution workflow for datastreams (opensearch-project#1302) (opensearch-project#1322)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

Update to Gradle 8.5 (opensearch-project#1369) (opensearch-project#1371)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

[Backport 2.x] Inject namedWriteableRegistry during ser/deser of SearchMonitorAction (opensearch-project#1382) (opensearch-project#1384)

* Inject namedWriteableRegistry during ser/deser of SearchMonitorAction (opensearch-project#1382)

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

* remove bin files

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

* remove core bin

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

---------

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

Don't attempt to parse workflow if it doesn't exist (opensearch-project#1346) (opensearch-project#1359)

(cherry picked from commit 733fd4e)

Signed-off-by: Chase Engelbrecht <engechas@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Set docData to empty string if actual is null (opensearch-project#1325) (opensearch-project#1334)

(cherry picked from commit 008e076)

Signed-off-by: Chase Engelbrecht <engechas@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

removed default admin credentials for alerting (opensearch-project#1399) (opensearch-project#1400)

(cherry picked from commit 3c50f7d)

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Dennis Toepker <toepkerd@amazon.com>

ipaddress lib upgrade as part of cve fix (opensearch-project#1397) (opensearch-project#1407)

(cherry picked from commit 8d59060)

Signed-off-by: Riya Saxena <riysaxen@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Bulk index findings and sequentially invoke auto-correlations (opensearch-project#1355) (opensearch-project#1410)

* Bulk index findings and sequentially invoke auto-correlations

* Bulk index findings in batches of 10000 and make it configurable

* Addressing review comments

* Add integ tests to test bulk index findings

* Fix ktlint formatting

---------

(cherry picked from commit b561965)

Signed-off-by: Megha Goyal <goyamegh@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Add 2.12 release notes (opensearch-project#1408) (opensearch-project#1413)

* Add 2.12 release notes

* Fix release notes PR

* Add 2 more PRs

---------

(cherry picked from commit b10eaad)

Signed-off-by: Chase Engelbrecht <engechas@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

[Backport 2.x] Implemented cross-cluster monitor support (opensearch-project#1404) (opensearch-project#1412)

* Implemented cross-cluster monitor support (opensearch-project#1404)

* Updated alert mappings to accommodate cross-cluster cluster metrics monitors.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Implemented support for cross-cluster cluster metrics monitors. Implemented GetRemoteIndexes API to populate the frontend UI with details regarding the remote clusters, and indexes.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Fixed a writeable test after changing QueryLevelTriggerRunResult from a data class to an open class for inheritability.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Fixed ktlint errors.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Removed changes to IndexUtils as they're only needed by doc monitors.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

---------

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

(cherry picked from commit ea36996)
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* Fixed a test.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

---------

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

Add publishToMavenLocal in build.sh (opensearch-project#1418) (opensearch-project#1419)

(cherry picked from commit 4cdc1d1)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

fix for MapperException[the [enabled] parameter can't be updated for the object mapping [metadata.source_to_query_index_mapping] (opensearch-project#1432) (opensearch-project#1434)

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

bacport PRs opensearch-project#1445, opensearch-project#1430, opensearch-project#1441, opensearch-project#1435 to 2.x (opensearch-project#1452)

* Add jvm aware setting and max num docs settings for batching docs for percolate queries (opensearch-project#1435)

* add jvm aware and max docs settings for batching docs for percolate queries

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix stats logging

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add queryfieldnames field in findings mapping

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (opensearch-project#1441)

* optimize to fetch only fields relevant to doc level queries in doc level monitor

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix test for settings check

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix ktlint

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* clean up doc level queries on dry run (opensearch-project#1430)

Signed-off-by: Joanne Wang <jowg@amazon.com>

* optimize sequence number calculation and reduce search requests in doc level monitor execution (opensearch-project#1445)

* optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix tests

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* optimize check indices and execute to query only write index of aliases and datastreams during monitor creation

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix test

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add javadoc

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add tests to verify seq_no calculation

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Joanne Wang <jowg@amazon.com>
Co-authored-by: Joanne Wang <jowg@amazon.com>

[Backport 2.x] Add an _exists_ check to document level monitor queries (opensearch-project#1425) (opensearch-project#1456)

* Add an _exists_ check to document level monitor queries (opensearch-project#1425)

* clean up and add integ tests

Signed-off-by: Joanne Wang <jowg@amazon.com>

* refactored out common method and renamed test

Signed-off-by: Joanne Wang <jowg@amazon.com>

* remove _exists_ flag

Signed-off-by: Joanne Wang <jowg@amazon.com>

---------

Signed-off-by: Joanne Wang <jowg@amazon.com>

* fix integ test

Signed-off-by: Joanne Wang <jowg@amazon.com>

---------

Signed-off-by: Joanne Wang <jowg@amazon.com>

add distributed locking to jobs in alerting (opensearch-project#1403) (opensearch-project#1458)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
eirsep pushed a commit to eirsep/alerting that referenced this pull request Mar 13, 2024
…the object mapping [metadata.source_to_query_index_mapping] (opensearch-project#1432) (opensearch-project#1434)

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
eirsep pushed a commit to eirsep/alerting that referenced this pull request Mar 14, 2024
…the object mapping [metadata.source_to_query_index_mapping] (opensearch-project#1432) (opensearch-project#1434)

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
eirsep added a commit that referenced this pull request Mar 14, 2024
…, #1441 to 2.9 (#1469)

* optimize doc-level monitor execution workflow for datastreams (#1302)

* optimize doc-level monitor execution for datastreams

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* add more tests to address comments

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* add integTest for multiple datastreams inside a single index pattern

* add integTest for multiple datastreams inside a single index pattern

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

---------

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* Bulk index findings and sequentially invoke auto-correlations (#1355)

* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

* Addressing review comments

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

---------

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

* fix for MapperException[the [enabled] parameter can't be updated for the object mapping [metadata.source_to_query_index_mapping] (#1432) (#1434)

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add jvm aware setting and max num docs settings for batching docs for percolate queries (#1435)

* add jvm aware and max docs settings for batching docs for percolate queries

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix stats logging

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add queryfieldnames field in findings mapping

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* clean up doc level queries on dry run (#1430)

Signed-off-by: Joanne Wang <jowg@amazon.com>

* optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (#1441)

* optimize to fetch only fields relevant to doc level queries in doc level monitor

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix test for settings check

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix ktlint

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* optimize sequence number calculation and reduce search requests in doc level monitor execution (#1445)

* optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix tests

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* optimize check indices and execute to query only write index of aliases and datastreams during monitor creation

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix test

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add javadoc

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add tests to verify seq_no calculation

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add distributed locking to jobs in alerting (#1403)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* 2.9.1 version bump

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix compilation issues

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* dummy commit

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix findings index schema version tsts

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: Megha Goyal <goyamegh@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Joanne Wang <jowg@amazon.com>
Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: Joanne Wang <jowg@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants