Merge branch 'master' into drop-source

docs/developer/plugin-list.asciidoc

@@ -150,8 +150,8 @@ It also provides a stateful version of it on the start contract.
 Content is fetched from the remote (https://feeds.elastic.co and https://feeds-staging.elastic.co in dev mode) once a day, with periodic checks if the content needs to be refreshed. All newsfeed content is hosted remotely.
 
 
-|{kib-repo}blob/{branch}/src/plugins/presentation_util/README.md[presentationUtil]
-|Utilities and components used by the presentation-related plugins
+|{kib-repo}blob/{branch}/src/plugins/presentation_util/README.mdx[presentationUtil]
+|The Presentation Utility Plugin is a set of common, shared components and toolkits for solutions within the Presentation space, (e.g. Dashboards, Canvas).
 
 
 |{kib-repo}blob/{branch}/src/plugins/region_map/README.md[regionMap]

docs/development/plugins/data/public/kibana-plugin-plugins-data-public.searchsource.fetch.md

@@ -4,8 +4,12 @@
 
 ## SearchSource.fetch() method
 
-Fetch this source and reject the returned Promise on error
+> Warning: This API is now obsolete.
+> 
+> Use fetch$ instead
+> 
 
+Fetch this source and reject the returned Promise on error
 
 <b>Signature:</b>
 

docs/development/plugins/data/public/kibana-plugin-plugins-data-public.searchsource.fetch_.md

@@ -0,0 +1,24 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-plugins-data-public](./kibana-plugin-plugins-data-public.md) &gt; [SearchSource](./kibana-plugin-plugins-data-public.searchsource.md) &gt; [fetch$](./kibana-plugin-plugins-data-public.searchsource.fetch_.md)
+
+## SearchSource.fetch$() method
+
+Fetch this source from Elasticsearch, returning an observable over the response(s)
+
+<b>Signature:</b>
+
+```typescript
+fetch$(options?: ISearchOptions): import("rxjs").Observable<import("elasticsearch").SearchResponse<any>>;
+```
+
+## Parameters
+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  options | <code>ISearchOptions</code> |  |
+
+<b>Returns:</b>
+
+`import("rxjs").Observable<import("elasticsearch").SearchResponse<any>>`
+

docs/development/plugins/data/public/kibana-plugin-plugins-data-public.searchsource.md

@@ -33,6 +33,7 @@ export declare class SearchSource
 |  [createCopy()](./kibana-plugin-plugins-data-public.searchsource.createcopy.md) |  | creates a copy of this search source (without its children) |
 |  [destroy()](./kibana-plugin-plugins-data-public.searchsource.destroy.md) |  | Completely destroy the SearchSource.  {<!-- -->undefined<!-- -->} |
 |  [fetch(options)](./kibana-plugin-plugins-data-public.searchsource.fetch.md) |  | Fetch this source and reject the returned Promise on error |
+|  [fetch$(options)](./kibana-plugin-plugins-data-public.searchsource.fetch_.md) |  | Fetch this source from Elasticsearch, returning an observable over the response(s) |
 |  [getField(field, recurse)](./kibana-plugin-plugins-data-public.searchsource.getfield.md) |  | Gets a single field from the fields |
 |  [getFields()](./kibana-plugin-plugins-data-public.searchsource.getfields.md) |  | returns all search source fields |
 |  [getId()](./kibana-plugin-plugins-data-public.searchsource.getid.md) |  | returns search source id |

docs/setup/upgrade/upgrade-migrations.asciidoc

@@ -19,17 +19,16 @@ Saved objects are stored in two indices:
 * `.kibana_{kibana_version}_001`, or if the `kibana.index` configuration setting is set `.{kibana.index}_{kibana_version}_001`. E.g. for Kibana v7.12.0 `.kibana_7.12.0_001`.
 * `.kibana_task_manager_{kibana_version}_001`, or if the `xpack.tasks.index` configuration setting is set `.{xpack.tasks.index}_{kibana_version}_001` E.g. for Kibana v7.12.0 `.kibana_task_manager_7.12.0_001`.
 
-The index aliases `.kibana` and `.kibana_task_manager` will always point to the most up-to-date version indices.
+The index aliases `.kibana` and `.kibana_task_manager` will always point to
+the most up-to-date saved object indices.
 
 The first time a newer {kib} starts, it will first perform an upgrade migration before starting plugins or serving HTTP traffic. To prevent losing acknowledged writes old nodes should be shutdown before starting the upgrade. To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later will add a write block to the outdated index. Table 1 lists the saved objects indices used by previous versions of {kib}.
 
 .Saved object indices and aliases per {kib} version
 [options="header"]
-[cols="a,a,a"]
 |=======================
-|Upgrading from version | Outdated index (alias) | Upgraded index (alias)
-| 6.0.0 through 6.4.x    | `.kibana`     1.3+^.^| `.kibana_7.12.0_001`
-(`.kibana` alias)
+|Upgrading from version | Outdated index (alias)
+| 6.0.0 through 6.4.x    | `.kibana`     
 
 `.kibana_task_manager_7.12.0_001` (`.kibana_task_manager` alias)
 | 6.5.0 through 7.3.x    | `.kibana_N` (`.kibana` alias)

docs/user/alerting/alert-types.asciidoc

@@ -8,7 +8,7 @@ This section covers stack alerts. For domain-specific alert types, refer to the
 Users will need `all` access to the *Stack Alerts* feature to be able to create and edit any of the alerts listed below.
 See <<kibana-feature-privileges, feature privileges>> for more information on configuring roles that provide access to this feature. 
 
-Currently {kib} provides one stack alert: the <<alert-type-index-threshold>> type.
+Currently {kib} provides two stack alerts: <<alert-type-index-threshold>> and <<alert-type-es-query>>.
 
 [float]
 [[alert-type-index-threshold]]
@@ -112,6 +112,47 @@ You can interactively change the time window and observe the effect it has on th
 [role="screenshot"]
 image::images/alert-types-index-threshold-example-comparison.png[Comparing two time windows]
 
+[float]
+[[alert-type-es-query]]
+=== ES query
+
+The ES query alert type is designed to run a user-configured {es} query over indices, compare the number of matches to a configured threshold, and schedule
+actions to run when the threshold condition is met.
+
+[float]
+==== Creating the alert
+
+An ES query alert can be created from the *Create* button in the <<alert-management, alert management UI>>. Fill in the <<defining-alerts-general-details, general alert details>>, then select *ES query*.
+
+[role="screenshot"]
+image::images/alert-types-es-query-select.png[Choosing an ES query alert type]
+
+[float]
+==== Defining the conditions
 
+The ES query alert has 4 clauses that define the condition to detect.
 
+[role="screenshot"]
+image::images/alert-types-es-query-conditions.png[Four clauses define the condition to detect]
+
+Index:: This clause requires an *index or index pattern* and a *time field* that will be used for the *time window*.
+ES query:: This clause specifies the ES DSL query to execute. The number of documents that match this query will be evaulated against the threshold
+condition. Aggregations are not supported at this time. 
+Threshold:: This clause defines a threshold value and a comparison operator  (`is above`, `is above or equals`, `is below`, `is below or equals`, or `is between`). The number of documents that match the specified query is compared to this threshold.
+Time window:: This clause determines how far back to search for documents, using the *time field* set in the *index* clause. Generally this value should be set to a value higher than the *check every* value in the <<defining-alerts-general-details, general alert details>>, to avoid gaps in detection. 
+
+[float]
+==== Testing your query
+
+Use the *Test query* feature to verify that your query DSL is valid.
 
+When your query is valid:: Valid queries will be executed against the configured *index* using the configured *time window*. The number of documents that
+match the query will be displayed.
+
+[role="screenshot"]
+image::images/alert-types-es-query-valid.png[Test ES query returns number of matches when valid]
+
+When your query is invalid:: An error message is shown if the query is invalid.
+
+[role="screenshot"]
+image::images/alert-types-es-query-invalid.png[Test ES query shows error when invalid]

package.json

@@ -393,6 +393,7 @@
     "@storybook/addon-essentials": "^6.0.26",
     "@storybook/addon-knobs": "^6.0.26",
     "@storybook/addon-storyshots": "^6.0.26",
+    "@storybook/addon-docs": "^6.0.26",
     "@storybook/components": "^6.0.26",
     "@storybook/core": "^6.0.26",
     "@storybook/core-events": "^6.0.26",

packages/kbn-telemetry-tools/src/tools/tasks/check_matching_schemas_task.ts

@@ -23,7 +23,7 @@ export function checkMatchingSchemasTask({ roots }: TaskContext, throwOnDiff: bo
         root.esMappingDiffs = Object.keys(differences);
         if (root.esMappingDiffs.length && throwOnDiff) {
           throw Error(
-            `The following changes must be persisted in ${fullPath} file. Use '--fix' to update.\n${JSON.stringify(
+            `The following changes must be persisted in ${fullPath} file. Run 'node scripts/telemetry_check --fix' to update.\n${JSON.stringify(
               differences,
               null,
               2

src/dev/build/lib/version_info.test.ts → src/dev/build/lib/integration_tests/version_info.test.ts

@@ -6,10 +6,11 @@
  * Public License, v 1.
  */
 
-import pkg from '../../../../package.json';
-import { getVersionInfo } from './version_info';
+import { kibanaPackageJSON as pkg } from '@kbn/dev-utils';
 
-jest.mock('./get_build_number');
+import { getVersionInfo } from '../version_info';
+
+jest.mock('../get_build_number');
 
 describe('isRelease = true', () => {
   it('returns unchanged package.version, build sha, and build number', async () => {

src/dev/build/tasks/os_packages/docker_generator/templates/build_docker_sh.template.ts

@@ -19,7 +19,6 @@ function generator({
   ubiImageFlavor,
   architecture,
 }: TemplateContext) {
-  const fileArchitecture = architecture === 'aarch64' ? 'arm64' : 'amd64';
   return dedent(`
   #!/usr/bin/env bash
   #
@@ -56,9 +55,9 @@ function generator({
   retry_docker_pull ${baseOSImage}
 
   echo "Building: kibana${imageFlavor}${ubiImageFlavor}-docker"; \\
-  docker build -t ${imageTag}${imageFlavor}${ubiImageFlavor}:${version}-${fileArchitecture} -f Dockerfile . || exit 1;
+  docker build -t ${imageTag}${imageFlavor}${ubiImageFlavor}:${version} -f Dockerfile . || exit 1;
 
-  docker save ${imageTag}${imageFlavor}${ubiImageFlavor}:${version}-${fileArchitecture} | gzip -c > ${dockerTargetFilename}
+  docker save ${imageTag}${imageFlavor}${ubiImageFlavor}:${version} | gzip -c > ${dockerTargetFilename}
 
   exit 0
   `);

src/dev/storybook/aliases.ts

@@ -18,4 +18,5 @@ export const storybookAliases = {
   security_solution: 'x-pack/plugins/security_solution/.storybook',
   ui_actions_enhanced: 'x-pack/plugins/ui_actions_enhanced/.storybook',
   observability: 'x-pack/plugins/observability/.storybook',
+  presentation: 'src/plugins/presentation_util/storybook',
 };

src/plugins/data/common/search/search_source/mocks.ts

@@ -6,7 +6,7 @@
  * Public License, v 1.
  */
 
-import { BehaviorSubject } from 'rxjs';
+import { BehaviorSubject, of } from 'rxjs';
 import type { MockedKeys } from '@kbn/utility-types/jest';
 import { uiSettingsServiceMock } from '../../../../../core/public/mocks';
 
@@ -27,6 +27,7 @@ export const searchSourceInstanceMock: MockedKeys<ISearchSource> = {
   createChild: jest.fn().mockReturnThis(),
   setParent: jest.fn(),
   getParent: jest.fn().mockReturnThis(),
+  fetch$: jest.fn().mockReturnValue(of({})),
   fetch: jest.fn().mockResolvedValue({}),
   onRequestStart: jest.fn(),
   getSearchRequestBody: jest.fn(),

src/plugins/data/common/search/search_source/search_source.test.ts

@@ -51,7 +51,14 @@ describe('SearchSource', () => {
   let searchSource: SearchSource;
 
   beforeEach(() => {
-    mockSearchMethod = jest.fn().mockReturnValue(of({ rawResponse: '' }));
+    mockSearchMethod = jest
+      .fn()
+      .mockReturnValue(
+        of(
+          { rawResponse: { isPartial: true, isRunning: true } },
+          { rawResponse: { isPartial: false, isRunning: false } }
+        )
+      );
 
     searchSourceDependencies = {
       getConfig: jest.fn(),
@@ -564,6 +571,34 @@ describe('SearchSource', () => {
       await searchSource.fetch(options);
       expect(mockSearchMethod).toBeCalledTimes(1);
     });
+
+    test('should return partial results', (done) => {
+      searchSource = new SearchSource({ index: indexPattern }, searchSourceDependencies);
+      const options = {};
+
+      const next = jest.fn();
+      const complete = () => {
+        expect(next).toBeCalledTimes(2);
+        expect(next.mock.calls[0]).toMatchInlineSnapshot(`
+          Array [
+            Object {
+              "isPartial": true,
+              "isRunning": true,
+            },
+          ]
+        `);
+        expect(next.mock.calls[1]).toMatchInlineSnapshot(`
+          Array [
+            Object {
+              "isPartial": false,
+              "isRunning": false,
+            },
+          ]
+        `);
+        done();
+      };
+      searchSource.fetch$(options).subscribe({ next, complete });
+    });
   });
 
   describe('#serialize', () => {

src/plugins/data/common/search/search_source/search_source.ts

@@ -60,7 +60,8 @@
 
 import { setWith } from '@elastic/safer-lodash-set';
 import { uniqueId, keyBy, pick, difference, omit, isObject, isFunction } from 'lodash';
-import { map } from 'rxjs/operators';
+import { map, switchMap, tap } from 'rxjs/operators';
+import { defer, from } from 'rxjs';
 import { normalizeSortRequest } from './normalize_sort_request';
 import { fieldWildcardFilter } from '../../../../kibana_utils/common';
 import { IIndexPattern } from '../../index_patterns';
@@ -244,30 +245,35 @@ export class SearchSource {
   }
 
   /**
-   * Fetch this source and reject the returned Promise on error
-   *
-   * @async
+   * Fetch this source from Elasticsearch, returning an observable over the response(s)
+   * @param options
    */
-  async fetch(options: ISearchOptions = {}) {
+  fetch$(options: ISearchOptions = {}) {
     const { getConfig } = this.dependencies;
-    await this.requestIsStarting(options);
-
-    const searchRequest = await this.flatten();
-    this.history = [searchRequest];
-
-    let response;
-    if (getConfig(UI_SETTINGS.COURIER_BATCH_SEARCHES)) {
-      response = await this.legacyFetch(searchRequest, options);
-    } else {
-      response = await this.fetchSearch(searchRequest, options);
-    }
-
-    // TODO: Remove casting when https://github.com/elastic/elasticsearch-js/issues/1287 is resolved
-    if ((response as any).error) {
-      throw new RequestFailure(null, response);
-    }
+    return defer(() => this.requestIsStarting(options)).pipe(
+      switchMap(() => {
+        const searchRequest = this.flatten();
+        this.history = [searchRequest];
+
+        return getConfig(UI_SETTINGS.COURIER_BATCH_SEARCHES)
+          ? from(this.legacyFetch(searchRequest, options))
+          : this.fetchSearch$(searchRequest, options);
+      }),
+      tap((response) => {
+        // TODO: Remove casting when https://github.com/elastic/elasticsearch-js/issues/1287 is resolved
+        if ((response as any).error) {
+          throw new RequestFailure(null, response);
+        }
+      })
+    );
+  }
 
-    return response;
+  /**
+   * Fetch this source and reject the returned Promise on error
+   * @deprecated Use fetch$ instead
+   */
+  fetch(options: ISearchOptions = {}) {
+    return this.fetch$(options).toPromise();
   }
 
   /**
@@ -305,16 +311,16 @@ export class SearchSource {
    * Run a search using the search service
    * @return {Promise<SearchResponse<unknown>>}
    */
-  private fetchSearch(searchRequest: SearchRequest, options: ISearchOptions) {
+  private fetchSearch$(searchRequest: SearchRequest, options: ISearchOptions) {
     const { search, getConfig, onResponse } = this.dependencies;
 
     const params = getSearchParamsFromRequest(searchRequest, {
       getConfig,
     });
 
-    return search({ params, indexType: searchRequest.indexType }, options)
-      .pipe(map(({ rawResponse }) => onResponse(searchRequest, rawResponse)))
-      .toPromise();
+    return search({ params, indexType: searchRequest.indexType }, options).pipe(
+      map(({ rawResponse }) => onResponse(searchRequest, rawResponse))
+    );
   }
 
   /**

src/plugins/data/public/public.api.md

@@ -2360,6 +2360,8 @@ export class SearchSource {
     createChild(options?: {}): SearchSource;
     createCopy(): SearchSource;
     destroy(): void;
+    fetch$(options?: ISearchOptions): import("rxjs").Observable<import("elasticsearch").SearchResponse<any>>;
+    // @deprecated
     fetch(options?: ISearchOptions): Promise<import("elasticsearch").SearchResponse<any>>;
     getField<K extends keyof SearchSourceFields>(field: K, recurse?: boolean): SearchSourceFields[K];
     getFields(): {
@@ -2601,7 +2603,7 @@ export const UI_SETTINGS: {
 // src/plugins/data/common/index_patterns/index_patterns/index_pattern.ts:138:7 - (ae-forgotten-export) The symbol "FieldAttrSet" needs to be exported by the entry point index.d.ts
 // src/plugins/data/common/index_patterns/index_patterns/index_pattern.ts:169:7 - (ae-forgotten-export) The symbol "RuntimeField" needs to be exported by the entry point index.d.ts
 // src/plugins/data/common/search/aggs/types.ts:139:51 - (ae-forgotten-export) The symbol "AggTypesRegistryStart" needs to be exported by the entry point index.d.ts
-// src/plugins/data/common/search/search_source/search_source.ts:186:7 - (ae-forgotten-export) The symbol "SearchFieldValue" needs to be exported by the entry point index.d.ts
+// src/plugins/data/common/search/search_source/search_source.ts:187:7 - (ae-forgotten-export) The symbol "SearchFieldValue" needs to be exported by the entry point index.d.ts
 // src/plugins/data/public/field_formats/field_formats_service.ts:56:3 - (ae-forgotten-export) The symbol "FormatFactory" needs to be exported by the entry point index.d.ts
 // src/plugins/data/public/index.ts:55:23 - (ae-forgotten-export) The symbol "FILTERS" needs to be exported by the entry point index.d.ts
 // src/plugins/data/public/index.ts:55:23 - (ae-forgotten-export) The symbol "getDisplayValueFromFilter" needs to be exported by the entry point index.d.ts