[Security Solution][Detections] Signals Migration API (#84721)

* WIP: basic reindexing works, lots of edge cases and TODOs to tackle

* Add note

* Add version metadata to signals documents

* WIP: Starting over from the ground up

* Removes obsolete endpoints/functions
* Adds endpoint for checking the migration status of signals indices
* Adds helper functions to represent the logical pieces of answering
  that question

* Fleshing out upgrade of signals

* triggers reindex for each index
* starts implementing followup endpoint to "finalize" after reindexing
  is finished

* Fleshing out more of the upgrade path

Still moving logic around a bunch.

* Pad the version number of our destination migration index

Instead of e.g. `.siem-signals-default-000001-r5`, this will generate
`.siem-signals-default-000001-r000005`.

This shouldn't matter much, but it may make it easier for users at a
glance to see the story of each index.

* Fleshing out more upgrade finalization

* Verifies that task matches the specified parameters
* Verifies that document counts are the same
* updates aliases
* finalization endpoint requires both source/dest indexes since we can't
  determine that from the task itself.

* Ensure that new signals are generated with an appropriate schema_version

* Apply migration cleanup policy to obsolete signals indexes

After upgrading a particular signals index, we're left with both the old
and new copies of the index. While the former is unlinked, it's still
taking up disk space; this ensures that it will eventually be deleted,
but gives users enough time to recover data if necessary.

This also ensures that, as with the normal signals ILM policy, it is
present during our normal sanity checks.

* Move more logic into component functions

* Fix type errors

* Refactor to make things a little more organized

* Moves migration-related routes under signals/ to match their routing
* Generalizes migration-agnostic helpers, moves them to appropriate
  folders (namely index/)
* Inlined getMigrationStatusInRange, a hyper-specific function with
  limited utility elsewhere

* Add some JSDoc comments around our new functions

This is as much to get my thoughts in order as it is for posterity.

Next: tests!

* Adds integration tests around migration status route

* Adds io-ts schema for route params
* Adds es_archiver data to represent an outdated signals index

* Adds API integration tests for our signals upgrade endpoint

* Adds io-ts schema for route params
* Adds second signals index archive, updates docs
* Adds test helper to wait for a given index to have documents
* Adds test helper to retrieve the relevant index name from a call to
  esArchive.load

* WIP: Fleshing out finalization tests

* Consolidate terminalogy around a migration

We're no longer making a distinction between an upgrade vs. an update
vs. a migration vs. a reindex: a migration is the concept that
encompasses this work. Both an index and individual documents can
require a migration, but both follow the same code path to migrate.

* Implement encoding of migration details

This will be a slightly better API: rather than having to pass all three
fields to finalize the migration, API users can instead send the token.

* Better transformation of errors thrown from the elasticsearch client

These often contain detailed information that we were previously
dropping. This will give better info on the migration finalization
endpoint, but should give more information across all detection_engine
endpoints in the case of an es client error.

* Finishing integration tests around finalization endpoint

This lead to a few changes in the responses from our different
endpoints; mainly, we pass both the migration token AND its constituent
parts to aid in debugging.

* Test an error case due to a reindexing failure

This would be really hard to reproduce with an integration test since
we'd need to generate a specific reindex failure. Much easier to stub
some ES calls to exercise that code in a unit test.

* Remove unnecessary version info from signals documents

We now record a single document-level version field. This represents the
version of the document's _source, which is generated by our rule
execution.

When either a mapping _or_ a transformation is added, this version will
be bumped such that new signals will contain the newest version, while
the index itself may still contain the old mappings.

The transformation pipeline will use the signal version to short-circuit
unnecessary transformations.

* Migrate an index relative to the ACTUAL template version

This handles the case where a user is attempting to migrate, but has not
yet rolled over to the newest template. Running rules may insert "new"
signals into an "old" index, but from the perspective of the app no
migration is necessary in that case.

If/when they roll over, the aforementioned index (and possibly older
ones) will be qualified as outdated, and can be migrated.

* Enrich our migration_status endpoint with an is_outdated qualification

This can be determined programatically, but for users manually
interpreting this response, the qualification will help.

* Update migration scripts

* More uniform version checking

* getIndexVersion always returns a number
* version comparisons use isOutdated

* Fix signal generation unit tests

We now generate a version field to indicate the version under which the
signal was created/migrated.

* Support reindex options to be sent to create_migration endpoint

Rather than having to perform a manual reindex, this should give API
users some control over the performance of their automated migration.

* Fix signal generation integration tests

These were failing on our new signal field.

* Add unit tests for getMigrationStatus

* Add a basic test for getSignalsIndicesInRange

Since this is ultimately just an aggregation query there's not much else
to test.

* Add unit test for the naming of our destination migration index

* Handle write indices in our migration logic

* Treat write indices as any other index in migration status endpoint
* Migration API rejects requests containing write indices
* Migration API rejects requests containing unknown/non-signals indices

* Add original hot phase to migration cleanup policy

Without this phase, ILM gets confused as it tries to move to the delete
phase and fails.

* Update old comment

The referenced field has changed.

* Delete task document as part of finalization

* Accurately report recoverable errors on create_signals_migration route

If we have a recoverable error: e.g. the destination index already
exists, or a specified index is a write index, we now report those
errors as part of the normal 200 response as these do not preclude other
specified indices from being migrated.

However, if non-signals indices are specified, we do continue to reject
the entire request, as that's indicative of misuse of the endpoint.

x-pack/plugins/security_solution/common/constants.ts

@@ -136,9 +136,13 @@ export const TIMELINE_PREPACKAGED_URL = `${TIMELINE_URL}/_prepackaged`;
  * Default signals index key for kibana.dev.yml
  */
 export const SIGNALS_INDEX_KEY = 'signalsIndex';
+
 export const DETECTION_ENGINE_SIGNALS_URL = `${DETECTION_ENGINE_URL}/signals`;
 export const DETECTION_ENGINE_SIGNALS_STATUS_URL = `${DETECTION_ENGINE_SIGNALS_URL}/status`;
 export const DETECTION_ENGINE_QUERY_SIGNALS_URL = `${DETECTION_ENGINE_SIGNALS_URL}/search`;
+export const DETECTION_ENGINE_SIGNALS_MIGRATION_URL = `${DETECTION_ENGINE_SIGNALS_URL}/migration`;
+export const DETECTION_ENGINE_SIGNALS_MIGRATION_STATUS_URL = `${DETECTION_ENGINE_SIGNALS_URL}/migration_status`;
+export const DETECTION_ENGINE_SIGNALS_FINALIZE_MIGRATION_URL = `${DETECTION_ENGINE_SIGNALS_URL}/finalize_migration`;
 
 /**
  * Common naming convention for an unauthenticated user

x-pack/plugins/security_solution/common/detection_engine/schemas/request/create_signals_migration_schema.mock.ts

@@ -0,0 +1,13 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import { CreateSignalsMigrationSchema } from './create_signals_migration_schema';
+
+export const getCreateSignalsMigrationSchemaMock = (
+  index: string = 'signals-index'
+): CreateSignalsMigrationSchema => ({
+  index: [index],
+});

x-pack/plugins/security_solution/common/detection_engine/schemas/request/create_signals_migration_schema.ts

@@ -0,0 +1,29 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import * as t from 'io-ts';
+
+import { index } from '../common/schemas';
+import { PositiveInteger, PositiveIntegerGreaterThanZero } from '../types';
+
+export const signalsReindexOptions = t.partial({
+  requests_per_second: t.number,
+  size: PositiveIntegerGreaterThanZero,
+  slices: PositiveInteger,
+});
+
+export type SignalsReindexOptions = t.TypeOf<typeof signalsReindexOptions>;
+
+export const createSignalsMigrationSchema = t.intersection([
+  t.exact(
+    t.type({
+      index,
+    })
+  ),
+  t.exact(signalsReindexOptions),
+]);
+
+export type CreateSignalsMigrationSchema = t.TypeOf<typeof createSignalsMigrationSchema>;

x-pack/plugins/security_solution/common/detection_engine/schemas/request/finalize_signals_migration_schema.mock.ts

@@ -0,0 +1,12 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import { FinalizeSignalsMigrationSchema } from './finalize_signals_migration_schema';
+
+export const getFinalizeSignalsMigrationSchemaMock = (): FinalizeSignalsMigrationSchema => ({
+  migration_token:
+    'eyJkZXN0aW5hdGlvbkluZGV4IjoiZGVzdGluYXRpb25JbmRleCIsInNvdXJjZUluZGV4Ijoic291cmNlSW5kZXgiLCJ0YXNrSWQiOiJteS10YXNrLWlkIn0=',
+});

x-pack/plugins/security_solution/common/detection_engine/schemas/request/finalize_signals_migration_schema.ts

@@ -0,0 +1,19 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import * as t from 'io-ts';
+
+import { NonEmptyString } from '../types';
+
+const migrationToken = NonEmptyString;
+
+export const finalizeSignalsMigrationSchema = t.exact(
+  t.type({
+    migration_token: migrationToken,
+  })
+);
+
+export type FinalizeSignalsMigrationSchema = t.TypeOf<typeof finalizeSignalsMigrationSchema>;

x-pack/plugins/security_solution/common/detection_engine/schemas/request/get_migration_status_schema.ts

@@ -0,0 +1,17 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import * as t from 'io-ts';
+
+import { from } from '../common/schemas';
+
+export const getMigrationStatusSchema = t.exact(
+  t.type({
+    from,
+  })
+);
+
+export type GetMigrationStatusSchema = t.TypeOf<typeof getMigrationStatusSchema>;

x-pack/plugins/security_solution/server/lib/detection_engine/index/get_index_aliases.ts

@@ -0,0 +1,49 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import { ElasticsearchClient } from 'src/core/server';
+
+interface AliasesResponse {
+  [indexName: string]: {
+    aliases: {
+      [aliasName: string]: {
+        is_write_index: boolean;
+      };
+    };
+  };
+}
+
+interface IndexAlias {
+  alias: string;
+  index: string;
+  isWriteIndex: boolean;
+}
+
+/**
+ * Retrieves all index aliases for a given alias name
+ *
+ * @param esClient An {@link ElasticsearchClient}
+ * @param alias alias name used to filter results
+ *
+ * @returns an array of {@link IndexAlias} objects
+ */
+export const getIndexAliases = async ({
+  esClient,
+  alias,
+}: {
+  esClient: ElasticsearchClient;
+  alias: string;
+}): Promise<IndexAlias[]> => {
+  const response = await esClient.indices.getAlias<AliasesResponse>({
+    name: alias,
+  });
+
+  return Object.keys(response.body).map((index) => ({
+    alias,
+    index,
+    isWriteIndex: response.body[index].aliases[alias]?.is_write_index === true,
+  }));
+};

x-pack/plugins/security_solution/server/lib/detection_engine/index/get_index_count.ts

@@ -0,0 +1,29 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import { ElasticsearchClient } from 'src/core/server';
+
+/**
+ * Retrieves the count of documents in a given index
+ *
+ * @param esClient An {@link ElasticsearchClient}
+ * @param index index whose documents will be counted
+ *
+ * @returns the document count
+ */
+export const getIndexCount = async ({
+  esClient,
+  index,
+}: {
+  esClient: ElasticsearchClient;
+  index: string;
+}): Promise<number> => {
+  const response = await esClient.count<{ count: number }>({
+    index,
+  });
+
+  return response.body.count;
+};

x-pack/plugins/security_solution/server/lib/detection_engine/migrations/create_signals_migration_index.test.ts

@@ -0,0 +1,25 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import { ElasticsearchClient } from 'src/core/server';
+import { elasticsearchServiceMock } from 'src/core/server/mocks';
+import { createSignalsMigrationIndex } from './create_signals_migration_index';
+
+describe('getMigrationStatus', () => {
+  let esClient: ElasticsearchClient;
+
+  beforeEach(() => {
+    esClient = elasticsearchServiceMock.createElasticsearchClient();
+  });
+
+  it('creates an index suffixed with the template version', async () => {
+    await createSignalsMigrationIndex({ esClient, index: 'my-signals-index', version: 4 });
+
+    expect(esClient.indices.create).toHaveBeenCalledWith(
+      expect.objectContaining({ index: 'my-signals-index-r000004' })
+    );
+  });
+});

x-pack/plugins/security_solution/server/lib/detection_engine/migrations/create_signals_migration_index.ts

@@ -0,0 +1,48 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import { ElasticsearchClient } from 'src/core/server';
+
+/**
+ * Creates the destination index to be used during the migration of a
+ * given signals index.
+ *
+ * The destination index's name is determined by adding a suffix of
+ * `-r${templateVersion}` to the source index name
+ *
+ * @param esClient An {@link ElasticsearchClient}
+ * @param index name of the source signals index
+ * @param version version of the current signals template/mappings
+ *
+ * @returns the name of the created index
+ */
+export const createSignalsMigrationIndex = async ({
+  esClient,
+  index,
+  version,
+}: {
+  esClient: ElasticsearchClient;
+  index: string;
+  version: number;
+}): Promise<string> => {
+  const paddedVersion = `${version}`.padStart(6, '0');
+  const destinationIndexName = `${index}-r${paddedVersion}`;
+
+  const response = await esClient.indices.create<{ index: string }>({
+    index: destinationIndexName,
+    body: {
+      settings: {
+        index: {
+          lifecycle: {
+            indexing_complete: true,
+          },
+        },
+      },
+    },
+  });
+
+  return response.body.index;
+};

x-pack/plugins/security_solution/server/lib/detection_engine/migrations/get_migration_status.mock.ts

@@ -0,0 +1,32 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import { IndexMappingsResponse, MigrationStatusSearchResponse } from './types';
+
+export const getMigrationStatusSearchResponseMock = (
+  indices: string[] = ['signals-index'],
+  signalVersions: number[] = [-1]
+): MigrationStatusSearchResponse => ({
+  aggregations: {
+    signals_indices: {
+      buckets: indices.map((index) => ({
+        key: index,
+        signal_versions: {
+          buckets: signalVersions.map((version) => ({
+            key: version,
+            doc_count: 4,
+          })),
+        },
+      })),
+    },
+  },
+});
+
+export const getIndexMappingsResponseMock = (
+  index: string = 'signals-index'
+): IndexMappingsResponse => ({
+  [index]: { mappings: { _meta: { version: -1 } } },
+});