Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dot-kibana-split] Update ES archiver to support multiple SO indices #154882

Merged
merged 3 commits into from
Apr 18, 2023
Merged

[dot-kibana-split] Update ES archiver to support multiple SO indices #154882

merged 3 commits into from
Apr 18, 2023

Conversation

gsoldevila
Copy link
Contributor

@gsoldevila gsoldevila commented Apr 13, 2023 •
edited
Loading

Part of #154888

ES archiver logic assumes that there are only two SO indices: '.kibana' and '.kibana_task_manager'.
This PR updates ES archiver so that it works with a list of SO indices instead.

@gsoldevila gsoldevila added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Team:Operations Team label for Operations Team enhancement New value added to drive a business result technical debt Improvement of the software architecture and operational architecture release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting v8.8.0 labels Apr 13, 2023
@gsoldevila gsoldevila requested review from a team as code owners April 13, 2023 04:09
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

@gsoldevila gsoldevila mentioned this pull request Apr 13, 2023
Merged
@gsoldevila gsoldevila changed the title Update ES archiver to support multiple SO indices [dot-kibana-split] Update ES archiver to support multiple SO indices Apr 13, 2023
rudolf
rudolf reviewed Apr 17, 2023
View reviewed changes
packages/kbn-es-archiver/src/lib/indices/kibana_index.ts Outdated
@@ -29,8 +34,10 @@ export async function deleteKibanaIndices({
onlyTaskManager?: boolean;
log: ToolingLog;
}) {
const indexPattern = onlyTaskManager ? '.kibana_task_manager*' : '.kibana*';
const indexNames = await fetchKibanaIndices(client, indexPattern);
// WARNING note that we are deleting ALL .kibana* indices here, NOT only the saved object ones
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on INDICES_REGEXP this seems to no longer be true

packages/kbn-es-archiver/src/lib/indices/kibana_index.ts Outdated

const LEGACY_INDICES_REGEXP = new RegExp(`^(${SavedObjectsIndexPatterns.join('|')})(:?_\\d*)?$`);
const INDICES_REGEXP = new RegExp(`^(${SavedObjectsIndexPatterns.join('|')})_(pre)?\\d+.\\d+.\\d+`);

function isKibanaIndex(index?: string): index is string {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we call this isSavedObjectIndex?

packages/kbn-es-archiver/src/lib/indices/kibana_index.ts Outdated
}

async function fetchKibanaIndices(client: Client, indexPattern: string) {
async function fetchKibanaIndices(client: Client) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we rename to fetchSavedObjectIndices ?

pgayvallet
pgayvallet approved these changes Apr 18, 2023
View reviewed changes
Copy link
Contributor

@pgayvallet pgayvallet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (especially the last commit :trollface:)

gsoldevila
gsoldevila commented Apr 18, 2023
View reviewed changes
packages/kbn-es-archiver/src/lib/indices/kibana_index.ts
import { Stats } from '../stats';
import { deleteIndex } from './delete_index';
import { ES_CLIENT_HEADERS } from '../../client_headers';

/**
* Deletes all indices that start with `.kibana`, or if onlyTaskManager==true, all indices that start with `.kibana_task_manager`
* Deletes all saved object indices, or if onlyTaskManager==true, it deletes task_manager indices
*/
export async function deleteKibanaIndices({
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT similarly, we could rename this to deleteSavedObjectIndices.

@gsoldevila gsoldevila merged commit 00edae1 into elastic:dot-kibana-split Apr 18, 2023
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/core-saved-objects-server 100 101 +1
Unknown metric groups

API count

id before after diff
@kbn/core-saved-objects-server 494 496 +2

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@gsoldevila gsoldevila mentioned this pull request Apr 18, 2023
Merged
pgayvallet pushed a commit that referenced this pull request Apr 20, 2023
@gsoldevila
[dot-kibana-split] Address some PR remarks (#155116)
b1ce437 
Part of #154882

This PR addresses the following NITS and remarks:
- Rename ES archiver internal methods that deal with SO indices. Some ES
archiver internal methods' names must be renamed to match their new
behaviour.
- Rename `SavedObjectsIndexPatterns` to `ALL_SAVED_OBJECT_INDICES`.
gsoldevila added a commit that referenced this pull request Apr 25, 2023
@gsoldevila @pgayvallet
@cnasikas @kibanamachine @banderror
Split the .kibana saved objects index into multiple indices (#154888)
21351df 
## Description 

Fix #104081

This PR move some of the SO types from the `.kibana` index into the
following ones:
- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`

This split/reallocation will occur during the `8.8.0` Kibana upgrade
(*meaning: from any version older than `8.8.0` to any version greater or
equal to `8.8.0`*)

**This PR main changes are:**
- implement the changes required in the SO migration algorithm to
support this reallocation
- update the FTR tools (looking at you esArchiver) to support these new
indices
- update hardcoded references to `.kibana` and usage of the
`core.savedObjects.getKibanaIndex()` to use new APIs to target the
correct index/indices
- update FTR datasets, tests and utility accordingly 

## To reviewers

**Overall estimated risk of regressions: low**

But, still, please take the time to review changes in your code. The
parts of the production code that were the most impacted are the
telemetry collectors, as most of them were performing direct requests
against the `.kibana` index, so we had to adapt them. Most other
contributor-owned changes are in FTR tests and datasets.

If you think a type is misplaced (either we missed some types that
should be moved to a specific index, or some types were moved and
shouldn't have been) please tell us, and we'll fix the reallocation
either in this PR or in a follow-up.

## .Kibana split

The following new indices are introduced by this PR, with the following
SO types being moved to it. (any SO type not listed here will be staying
in its current index)

Note: The complete **_type => index_** breakdown is available in [this
spreadsheet](https://docs.google.com/spreadsheets/d/1b_MG_E_aBksZ4Vkd9cVayij1oBpdhvH4XC8NVlChiio/edit#gid=145920788).

#### `.kibana_alerting_cases`
- action
- action_task_params
- alert
- api_key_pending_invalidation
- cases
- cases-comments
- cases-configure
- cases-connector-mappings
- cases-telemetry
- cases-user-actions
- connector_token
- rules-settings
- maintenance-window

#### `.kibana_security_solution`
- csp-rule-template
- endpoint:user-artifact
- endpoint:user-artifact-manifest
- exception-list
- exception-list-agnostic
- osquery-manager-usage-metric
- osquery-pack
- osquery-pack-asset
- osquery-saved-query
- security-rule
- security-solution-signals-migration
- siem-detection-engine-rule-actions
- siem-ui-timeline
- siem-ui-timeline-note
- siem-ui-timeline-pinned-event

#### `.kibana_analytics`

- canvas-element
- canvas-workpad-template
- canvas-workpad
- dashboard
- graph-workspace
- index-pattern
- kql-telemetry
- lens
- lens-ui-telemetry
- map
- search
- search-session
- search-telemetry
- visualization

#### `.kibana_ingest`

- epm-packages
- epm-packages-assets
- fleet-fleet-server-host
- fleet-message-signing-keys
- fleet-preconfiguration-deletion-record
- fleet-proxy
- ingest_manager_settings
- ingest-agent-policies
- ingest-download-sources
- ingest-outputs
- ingest-package-policies

## Tasks / PRs

### Sub-PRs

**Implementation**
- 🟣 #154846
- 🟣 #154892
- 🟣 #154882
- 🟣 #154884
- 🟣 #155155

**Individual index split**
- 🟣 #154897
- 🟣 #155129
- 🟣 #155140
- 🟣 #155130

### Improvements / follow-ups 

- 👷🏼 Extract logic into
[runV2Migration](#154151 (comment))
@gsoldevila
- Make `getCurrentIndexTypesMap` resillient to intermittent failures
#154151 (comment)
- 🚧 Build a more structured
[MigratorSynchronizer](#154151 (comment))
- 🟣 #155035
- 🟣 #155116
- 🟣 #155366
## Reallocation tweaks

Tweaks to the reallocation can be done after the initial merge, as long
as it's done before the public release of 8.8

- `url` should get back to `.kibana` (see
[comment](#154888 (comment)))

## Release Note

For performance purposes, Kibana is now using more system indices to
store its internal data.

The following system indices will be created when upgrading to `8.8.0`:

- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`

---------

Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
nikitaindik pushed a commit to nikitaindik/kibana that referenced this pull request Apr 25, 2023
@gsoldevila @pgayvallet
@cnasikas @kibanamachine @banderror @nikitaindik
Split the .kibana saved objects index into multiple indices (elastic#…
e7d50f9 
…154888)

## Description 

Fix elastic#104081

This PR move some of the SO types from the `.kibana` index into the
following ones:
- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`

This split/reallocation will occur during the `8.8.0` Kibana upgrade
(*meaning: from any version older than `8.8.0` to any version greater or
equal to `8.8.0`*)

**This PR main changes are:**
- implement the changes required in the SO migration algorithm to
support this reallocation
- update the FTR tools (looking at you esArchiver) to support these new
indices
- update hardcoded references to `.kibana` and usage of the
`core.savedObjects.getKibanaIndex()` to use new APIs to target the
correct index/indices
- update FTR datasets, tests and utility accordingly 

## To reviewers

**Overall estimated risk of regressions: low**

But, still, please take the time to review changes in your code. The
parts of the production code that were the most impacted are the
telemetry collectors, as most of them were performing direct requests
against the `.kibana` index, so we had to adapt them. Most other
contributor-owned changes are in FTR tests and datasets.

If you think a type is misplaced (either we missed some types that
should be moved to a specific index, or some types were moved and
shouldn't have been) please tell us, and we'll fix the reallocation
either in this PR or in a follow-up.

## .Kibana split

The following new indices are introduced by this PR, with the following
SO types being moved to it. (any SO type not listed here will be staying
in its current index)

Note: The complete **_type => index_** breakdown is available in [this
spreadsheet](https://docs.google.com/spreadsheets/d/1b_MG_E_aBksZ4Vkd9cVayij1oBpdhvH4XC8NVlChiio/edit#gid=145920788).

#### `.kibana_alerting_cases`
- action
- action_task_params
- alert
- api_key_pending_invalidation
- cases
- cases-comments
- cases-configure
- cases-connector-mappings
- cases-telemetry
- cases-user-actions
- connector_token
- rules-settings
- maintenance-window

#### `.kibana_security_solution`
- csp-rule-template
- endpoint:user-artifact
- endpoint:user-artifact-manifest
- exception-list
- exception-list-agnostic
- osquery-manager-usage-metric
- osquery-pack
- osquery-pack-asset
- osquery-saved-query
- security-rule
- security-solution-signals-migration
- siem-detection-engine-rule-actions
- siem-ui-timeline
- siem-ui-timeline-note
- siem-ui-timeline-pinned-event

#### `.kibana_analytics`

- canvas-element
- canvas-workpad-template
- canvas-workpad
- dashboard
- graph-workspace
- index-pattern
- kql-telemetry
- lens
- lens-ui-telemetry
- map
- search
- search-session
- search-telemetry
- visualization

#### `.kibana_ingest`

- epm-packages
- epm-packages-assets
- fleet-fleet-server-host
- fleet-message-signing-keys
- fleet-preconfiguration-deletion-record
- fleet-proxy
- ingest_manager_settings
- ingest-agent-policies
- ingest-download-sources
- ingest-outputs
- ingest-package-policies

## Tasks / PRs

### Sub-PRs

**Implementation**
- 🟣 elastic#154846
- 🟣 elastic#154892
- 🟣 elastic#154882
- 🟣 elastic#154884
- 🟣 elastic#155155

**Individual index split**
- 🟣 elastic#154897
- 🟣 elastic#155129
- 🟣 elastic#155140
- 🟣 elastic#155130

### Improvements / follow-ups 

- 👷🏼 Extract logic into
[runV2Migration](elastic#154151 (comment))
@gsoldevila
- Make `getCurrentIndexTypesMap` resillient to intermittent failures
elastic#154151 (comment)
- 🚧 Build a more structured
[MigratorSynchronizer](elastic#154151 (comment))
- 🟣 elastic#155035
- 🟣 elastic#155116
- 🟣 elastic#155366
## Reallocation tweaks

Tweaks to the reallocation can be done after the initial merge, as long
as it's done before the public release of 8.8

- `url` should get back to `.kibana` (see
[comment](elastic#154888 (comment)))

## Release Note

For performance purposes, Kibana is now using more system indices to
store its internal data.

The following system indices will be created when upgrading to `8.8.0`:

- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`

---------

Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
@rudolf rudolf added the Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects label Jun 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rudolf rudolf rudolf left review comments

@pgayvallet pgayvallet pgayvallet approved these changes

@jbudz jbudz jbudz approved these changes

Assignees
No one assigned
Labels
backport:skip This commit does not require backporting enhancement New value added to drive a business result Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Team:Operations Team label for Operations Team technical debt Improvement of the software architecture and operational architecture v8.8.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@gsoldevila @elasticmachine @kibana-ci @rudolf @pgayvallet @jbudz