Merge branch 'main' into kibana_root_context

dev_docs/key_concepts/security.mdx

@@ -0,0 +1,100 @@
+---
+id: kibDevDocsSecurityIntro
+slug: /kibana-dev-docs/key-concepts/security-intro
+title: Security
+description: Maintaining Kibana's security posture
+date: 2023-07-11
+tags: ['kibana', 'dev', 'contributor', 'security']
+---
+
+Security is everyone's responsibility. This is inclusive of design, product, and engineering. The purpose of this guide is to give a high-level overview of security constructs and expectations.
+
+This guide covers the following topics:
+
+* [API authorization](#api-authorization)
+* [The `kibana_system` user](#the-kibana_system-user)
+
+## API authorization
+Kibana API routes do not have any authorization checks applied by default. This means that your APIs are accessible to anyone with valid credentials, regardless of their permissions. This includes users with no roles assigned.
+This on its own is insufficient, and care must be taken to ensure that only authorized users can invoke your endpoints.
+
+### Adding API authorization
+Kibana leverages <DocLink id="kibDevDocsSavedObjectsIntro" text="Saved Objects" /> for a majority of its persistence. The Saved Objects Service performs its own authorization checks, so if your API route is primarily a CRUD interface to Saved Objects, then your authorization needs are already met.
+This is also true for derivatives of the Saved Objects Service, such as the Alerting and Cases services.
+
+If your endpoint is not a CRUD interface to Saved Objects, then your route should include `access` tags to ensure that only authorized users can invoke your endpoint. This is **especially** important if your route does any of the following:
+1. Performs non-insignificant processing, causing load on the Kibana server.
+2. Calls Elasticsearch using the internal `kibana_system` user.
+3. Calls a third-party service.
+4. Returns any non-public information to the caller, such as system configuration or state.
+
+More information on adding `access` tags to your routes can be found temporarily in the [legacy documentation](https://www.elastic.co/guide/en/kibana/current/development-security.html#development-plugin-feature-registration)
+
+### Why not add `access` tags to all routes by default?
+Each authorization check that we perform involves a round-trip to Elasticsearch, so they are not as cheap as we'd like. Therefore, we want to keep the number of authorization checks we perform within a single route to a minimum.
+Adding an `access` tag to routes that leverage the Saved Objects Service would be redundant in most cases, since the Saved Objects Service will be performing authorization anyway.
+
+
+## The `kibana_system` user
+
+The Kibana server authenticates to Elasticsearch using the `elastic/kibana` [service account](https://www.elastic.co/guide/en/elasticsearch/reference/current/service-accounts.html#service-accounts-explanation). This service account has privileges that are equivilent to the `kibana_system` reserved role, whose descriptor is managed in the Elasticsearch repository ([source link](https://github.com/elastic/elasticsearch/blob/430cde6909eae12e1a90ac2bff29b71cbf4af18b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java#L58)).
+The vast majority of features will not require changes to the `kibana_system` user privileges. Changes to these privileges must be carefully considered, as we do not want the `kibana_system` account to have access to user data.
+
+### Guiding principals
+
+Consider these guidelines when reviewing changes to the `kibana_system` role descriptor.
+
+#### 1. Kibana should only access system and hidden indices
+
+- System indices are managed entirely by the stack, and should never be accessed by end users.
+- Hidden indices are _typically_ managed by the stack, but _may_ be accessed by end users.
+- Data indices are those which an end user could conceivably create on their own. As a general rule, users can create indices of any pattern so long as it does not begin with a \``.`\` (dot). Users can also create hidden indices, so it is important that documentation exists for any stack-managed hidden indices to reduce the chance of conflict with user-managed indices.
+
+Therefore, Kibana should not have access to non-system indices which do not begin with a \``.`\` (dot).
+
+Kibana should also not have the ability to modify system/hidden indices for which it is not the owner.
+
+##### Examples
+| Index Type | Allowed Permissions | Examples |
+|-------|--------|-----
+| User-defined data index | none | `my-data`, `kibana-metrics` |
+| System index not owned by Kibana | `read` | `.security` |
+| System indices owned by Kibana | `all` | `.kibana*`, '.fleet*' |
+
+#### Decision tree
+<DocWhimsical id="kibana-system-privilege-decision-tree-VTTGaTtjs9SnpbRNSg2Ptp" title="Decision tree" />
+
+##### Exceptions for telemetry
+Exceptions to this rule have been made in the past to aid in telemetry collection. This is not something we want to support long-term, but there aren't viable alternatives at the moment without a significant redesign. 
+
+##### Exceptions for Fleet package lifecycle management
+Fleet maintains the lifecycle of certain packages. These packages need to be upgraded during stack upgrades, and therefore have to happen in an automated fashion. The `kibana_system` user has elevated privileges to a set of **data incides** to facilitate this.
+
+If the set of indices managed by Fleet changes, we should ensure that they update [the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html) to make note of index naming collisions.
+
+
+#### 2. Kibana should not have the ability to manage security constructs.
+
+This includes:
+- Users
+- Roles
+- Role Mappings
+
+### Rationale
+
+These guidelines exist for two primary reasons.
+
+#### Reduce impact of account compromise
+Compromised `kibana_system` credentials can severely impact an installation. We want to make sure that this doesn't become catastrophic. More privileges == more potential damage. We shouldn't add privileges unnecessarily. We should remove privileges as soon as they aren't needed anymore.
+
+Credentials can be compromised in a number of ways:
+1. Insecure storage (e.g. `kibana.yml`, a post-it note, etc.).
+2. Kibana server host compromise.
+3. Kibana server runtime compromise (e.g. RCE).
+
+#### Reduce impact of developer error
+Kibana allows engineers to call ES using different sets of credentials:
+1. `kibana_system` credentials. 
+2. End-user credentials.
+
+An authorization bypass could occur if an engineer accidentally uses the `kibana_system` credentials when they intended to use end-user credentials. See [Broken Access Control](https://owasp.org/Top10/A01_2021-Broken_Access_Control/).

docs/dev-tools/console/console.asciidoc

@@ -122,6 +122,34 @@ GET ${pathVariable}
 }
 ----------------------------------
 
+By default, variables in the body may be substituted as a boolean, number, array, or
+object by removing nearby quotes instead of a string with surrounding quotes. Triple
+quotes overwrite this default behavior and enforce simple replacement as a string.
+
+[source,js]
+----------------------------------
+GET /locations/_search
+{
+  "query": {
+    "bool": {
+      "must": {
+        "match": {
+          // ${shopName} shall be replaced as a string if the variable exists.
+          "shop.name": """${shopName}"""
+        }
+      },
+      "filter": {
+        "geo_distance": {
+          "distance": "12km",
+          // "${pinLocation}" may be substituted with an array such as [-70, 40].
+          "pin.location": "${pinLocation}"
+        }
+      }
+    }
+  }
+}
+----------------------------------
+
 [float]
 [[auto-formatting]]
 ==== Auto-formatting

docs/management/connectors/action-types/gen-ai.asciidoc

@@ -86,4 +86,12 @@ Body::      A JSON payload sent to the OpenAI API URL. For example:
 [[gen-ai-connector-networking-configuration]]
 === Connector networking configuration
 
-Use the <<action-settings, Action configuration settings>> to customize connector networking configurations, such as proxies, certificates, or TLS settings. You can set configurations that apply to all your connectors or use `xpack.actions.customHostSettings` to set per-host configurations.
+Use the <<action-settings, Action configuration settings>> to customize connector networking configurations, such as proxies, certificates, or TLS settings. You can set configurations that apply to all your connectors or use `xpack.actions.customHostSettings` to set per-host configurations.
+
+[float]
+[[gen-ai-connector-token-dashboard]]
+=== Token usage dashboard
+
+Once you've created a Generative AI connector, you can monitor its token usage using the *Generative AI Token Usage* dashboard. Select the connector in *{stack-manage-app}* > *{connectors-ui}* to view its details, then click the *View OpenAI Usage Dashboard for "_<Name>_" Connector* link to open the dashboard.
+
+NOTE: To view the dashboard, you need at least `read` and `view_index_metadata` privileges for the `.kibana-event-log-*` index and the `Read` feature privilege for {kib}. You can set up a role with these minimum privileges and assign it to non-admin users who need to view this dashboard.

nav-kibana-dev.docnav.json

@@ -78,6 +78,9 @@
         {
           "id": "kibBuildingBlocks"
         },
+        {
+          "id": "kibDevDocsSecurityIntro"
+        },
         {
           "id": "kibDevDocsSavedObjectsIntro",
           "label": "Saved objects"
@@ -177,6 +180,9 @@
         },
         {
           "id": "kibDevTutorialScreenshotting"
+        },
+        {
+          "id": "kibDevTutorialsServerlessProjectNavigation"
         }
       ]
     },
@@ -479,4 +485,4 @@
       ]
     }
   ]
-}
+}

package.json

@@ -104,7 +104,7 @@
     "@elastic/numeral": "^2.5.1",
     "@elastic/react-search-ui": "^1.20.2",
     "@elastic/react-search-ui-views": "^1.20.2",
-    "@elastic/request-crypto": "2.0.1",
+    "@elastic/request-crypto": "2.0.2",
     "@elastic/search-ui": "^1.20.2",
     "@elastic/search-ui-app-search-connector": "^1.20.2",
     "@elastic/search-ui-engines-connector": "^1.20.2",

packages/kbn-discover-utils/index.ts

@@ -7,6 +7,27 @@
  */
 
 export {
+  CONTEXT_DEFAULT_SIZE_SETTING,
+  CONTEXT_STEP_SETTING,
+  CONTEXT_TIE_BREAKER_FIELDS_SETTING,
+  DEFAULT_COLUMNS_SETTING,
+  DOC_HIDE_TIME_COLUMN_SETTING,
+  DOC_TABLE_LEGACY,
+  ENABLE_SQL,
+  FIELDS_LIMIT_SETTING,
+  HIDE_ANNOUNCEMENTS,
+  MAX_DOC_FIELDS_DISPLAYED,
+  MODIFY_COLUMNS_ON_SWITCH,
+  ROW_HEIGHT_OPTION,
+  SAMPLE_ROWS_PER_PAGE_SETTING,
+  SAMPLE_SIZE_SETTING,
+  SEARCH_EMBEDDABLE_TYPE,
+  SEARCH_FIELDS_FROM_SOURCE,
+  SEARCH_ON_PAGE_LOAD_SETTING,
+  SHOW_FIELD_STATISTICS,
+  SHOW_MULTIFIELDS,
+  SORT_DEFAULT_ORDER_SETTING,
+  TRUNCATE_MAX_HEIGHT,
   IgnoredReason,
   buildDataTableRecord,
   buildDataTableRecordList,

packages/kbn-discover-utils/package.json

@@ -2,5 +2,6 @@
   "name": "@kbn/discover-utils",
   "private": true,
   "version": "1.0.0",
-  "license": "SSPL-1.0 OR Elastic License 2.0"
+  "license": "SSPL-1.0 OR Elastic License 2.0",
+  "sideEffects": false
 }

packages/kbn-discover-utils/src/constants.ts

@@ -0,0 +1,29 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+export const CONTEXT_DEFAULT_SIZE_SETTING = 'context:defaultSize';
+export const CONTEXT_STEP_SETTING = 'context:step';
+export const CONTEXT_TIE_BREAKER_FIELDS_SETTING = 'context:tieBreakerFields';
+export const DEFAULT_COLUMNS_SETTING = 'defaultColumns';
+export const DOC_HIDE_TIME_COLUMN_SETTING = 'doc_table:hideTimeColumn';
+export const DOC_TABLE_LEGACY = 'doc_table:legacy';
+export const ENABLE_SQL = 'discover:enableSql';
+export const FIELDS_LIMIT_SETTING = 'fields:popularLimit';
+export const HIDE_ANNOUNCEMENTS = 'hideAnnouncements';
+export const MAX_DOC_FIELDS_DISPLAYED = 'discover:maxDocFieldsDisplayed';
+export const MODIFY_COLUMNS_ON_SWITCH = 'discover:modifyColumnsOnSwitch';
+export const ROW_HEIGHT_OPTION = 'discover:rowHeightOption';
+export const SAMPLE_ROWS_PER_PAGE_SETTING = 'discover:sampleRowsPerPage';
+export const SAMPLE_SIZE_SETTING = 'discover:sampleSize';
+export const SEARCH_EMBEDDABLE_TYPE = 'search';
+export const SEARCH_FIELDS_FROM_SOURCE = 'discover:searchFieldsFromSource';
+export const SEARCH_ON_PAGE_LOAD_SETTING = 'discover:searchOnPageLoad';
+export const SHOW_FIELD_STATISTICS = 'discover:showFieldStatistics';
+export const SHOW_MULTIFIELDS = 'discover:showMultiFields';
+export const SORT_DEFAULT_ORDER_SETTING = 'discover:sort:defaultOrder';
+export const TRUNCATE_MAX_HEIGHT = 'truncate:maxHeight';

packages/kbn-discover-utils/src/index.ts

@@ -6,5 +6,6 @@
  * Side Public License, v 1.
  */
 
+export * from './constants';
 export * from './hooks';
 export * from './utils';

packages/kbn-es/README.mdx

@@ -7,11 +7,13 @@ date: 2022-05-24
 tags: ['kibana', 'dev', 'contributor', 'operations', 'es']
 ---
 
-> A command line utility for running elasticsearch from snapshot, source, archive or even building snapshot artifacts. 
+> A command line utility for running elasticsearch from snapshot, source, archive, docker, serverless or even building snapshot artifacts. 
 
 ## Getting started
 If running elasticsearch from source, elasticsearch needs to be cloned to a sibling directory of Kibana.
 
+If running elasticsearch serverless or a docker container, docker is required to be installed locally. Installation instructions can be found [here](https://www.docker.com/).
+
 To run, go to the Kibana root and run `node scripts/es --help` to get the latest command line options.
 
 The script attempts to preserve the existing interfaces used by Elasticsearch CLI. This includes passing through options with the `-E` argument and the `ES_JAVA_OPTS` environment variable for Java options.
@@ -28,6 +30,16 @@ Run from source with a configured data directory
 node scripts/es source --Epath.data=/home/me/es_data
 ```
 
+Run serverless with a specific image tag
+```
+node scripts/es serverless --tag git-fec36430fba2-x86_64
+```
+
+Run an official Docker release
+```
+node scripts/es docker --tag 8.8.2
+```
+
 ## API
 
 ### run

packages/kbn-es/src/cli_commands/docker.ts

@@ -0,0 +1,67 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import dedent from 'dedent';
+import getopts from 'getopts';
+import { ToolingLog } from '@kbn/tooling-log';
+import { getTimeReporter } from '@kbn/ci-stats-reporter';
+
+import { Cluster } from '../cluster';
+import { DOCKER_IMG, DOCKER_REPO, DOCKER_TAG } from '../utils';
+import { Command } from './types';
+
+export const docker: Command = {
+  description: 'Run an Elasticsearch Docker image',
+  usage: 'es docker [<args>]',
+  help: (defaults: Record<string, any> = {}) => {
+    const { password } = defaults;
+
+    return dedent`
+    Options:
+
+      --tag               Image tag of ES to run from ${DOCKER_REPO} [default: ${DOCKER_TAG}]
+      --image             Full path to image of ES to run, has precedence over tag. [default: ${DOCKER_IMG}]
+      --password          Sets password for elastic user [default: ${password}]
+      -E                  Additional key=value settings to pass to Elasticsearch
+      -D                  Override Docker command
+
+    Examples:
+
+      es docker --tag master-SNAPSHOT-amd64
+      es docker --image docker.elastic.co/repo:tag
+      es docker -D 'start es01' 
+    `;
+  },
+  run: async (defaults = {}) => {
+    const runStartTime = Date.now();
+    const log = new ToolingLog({
+      level: 'info',
+      writeTo: process.stdout,
+    });
+    const reportTime = getTimeReporter(log, 'scripts/es docker');
+
+    const argv = process.argv.slice(2);
+    const options = getopts(argv, {
+      alias: {
+        esArgs: 'E',
+        dockerCmd: 'D',
+      },
+
+      string: ['tag', 'image', 'D'],
+
+      default: defaults,
+    });
+
+    const cluster = new Cluster();
+    await cluster.runDocker({
+      reportTime,
+      startTime: runStartTime,
+      ...options,
+    });
+  },
+};

packages/kbn-es/src/cli_commands/index.ts

@@ -10,10 +10,14 @@ import { snapshot } from './snapshot';
 import { source } from './source';
 import { archive } from './archive';
 import { buildSnapshots } from './build_snapshots';
+import { docker } from './docker';
+import { serverless } from './serverless';
 
 export const commands = {
   snapshot,
   source,
   archive,
   build_snapshots: buildSnapshots,
+  docker,
+  serverless,
 };