Merge 4c54d88 into 4d6c4a8

ydb/docs/build.sh

@@ -36,10 +36,10 @@ echo "Output directory: $DIR"
 
 if ! yfm -i . -o $DIR --allowHTML  --apply-presets; then
   echo
-  echo "================================"
-  echo "YFM build completed with ERRORS!"
-  echo "================================"
-  echo "It may be necessary to use the latest version of npm. Run the command `nvm use v22.9.0` to update it."
+  echo '================================'
+  echo 'YFM build completed with ERRORS!'
+  echo '================================'
+  echo 'It may be necessary to use the latest version of npm. Run the command `nvm use v22.9.0` to update it.'
   exit 1
 fi
 

ydb/docs/en/core/devops/manual/node-authorization.md

@@ -0,0 +1,92 @@
+# Database node authentication and authorization
+
+Node authentication in the {{ ydb-short-name }} cluster ensures that database nodes are authenticated when making service requests to other nodes via the gRPC protocol. Node authorization ensures that the privileges required by the service requests are checked and granted during request processing. These service calls include database node registration within the cluster and access to [dynamic configuration](../../maintenance/manual/dynamic-config.md) information. The use of node authorization is recommended for all {{ ydb-short-name }} clusters, as it helps prevent unauthorized access to data by adding nodes controlled by an attacker to the cluster.
+
+Database node authentication and authorization are performed in the following order:
+
+1. The database node being started opens a gRPC connection to one of the cluster storage nodes specified in the `--node-broker` command-line option. The connection uses the TLS protocol, and the certificate of the running node is used as the client certificate for the connection.
+2. The storage node and the database node perform mutual authentication checks using the TLS protocol: the certificate trust chain is checked, and the hostname is matched against the value of the "Subject Name" field of the certificate.
+3. The storage node checks the "Subject" field of the certificate for compliance with the requirements set in the static configuration.
+4. If the above checks are successful, the connection from the database node is considered authenticated, and it is assigned a special [SID](../../concepts/glossary.md#access-sid) determined by the settings.
+5. The database node uses the established gRPC connection to register with the cluster and obtain the dynamic configuration through the corresponding service requests.
+6. The storage node checks whether the SID assigned to the connection is in the list of acceptable ones. If this check is successful, the storage node performs the requested action.
+
+Below are the steps required to enable the node authentication and authorization feature.
+
+## Configuration prerequisites
+
+1. The deployed {{ ydb-short-name }} cluster must have [gRPC traffic encryption](../../reference/configuration/tls.md#grpc) configured to use the TLS protocol.
+1. When preparing node certificates for a cluster where you plan to use the node authorization feature, uniform rules must be used for populating the "Subject" field of the certificates. This allows the identification of certificates issued for the cluster nodes. For more information, see the [certificate verification rules documentation](../../reference/configuration/node-authentication.md).
+
+    {% note info %}
+
+    The proposed [example script](https://github.com/ydb-platform/ydb/blob/main/ydb/deploy/tls_cert_gen/) generates self-signed certificates for {{ ydb-short-name }} nodes and ensures that the "Subject" field is populated with the value `O=YDB` for all node certificates. The configuration examples provided below are prepared for certificates with this specific "Subject" field configuration, but feel free to use your real organization name instead.
+
+    {% endnote %}
+
+1. The command-line parameters for [starting database nodes](../../devops/manual/initial-deployment.md#start-dynnode) must include options that specify the paths to the trusted CA certificate, the node certificate, and the node key files. The required additional command-line options are shown in the table below.
+
+    | **Command-line option** | **Description** |
+    |-------------------------|-----------------|
+    | `--grpc-ca`             | Path to the trusted certification authority file `ca.crt` |
+    | `--grpc-cert`           | Path to the node certificate file `node.crt` |
+    | `--grpc-key`            | Path to the node secret key file `node.key` |
+
+    Below is an example of the complete command to start the database node, including the extra options for gRPC TLS key and certificate files:
+
+    ```bash
+    /opt/ydb/bin/ydbd server --yaml-config  /opt/ydb/cfg/config.yaml --tenant /Root/testdb \
+        --grpcs-port 2136 --grpc-ca /opt/ydb/certs/ca.crt \
+        --grpc-cert /opt/ydb/certs/node.crt --grpc-key /opt/ydb/certs/node.key \
+        --ic-port 19002 --ca /opt/ydb/certs/ca.crt \
+        --mon-port 8766 --mon-cert /opt/ydb/certs/web.pem \
+        --node-broker grpcs://<ydb1>:2135 \
+        --node-broker grpcs://<ydb2>:2135 \
+        --node-broker grpcs://<ydb3>:2135
+    ```
+
+## Enabling database node authentication and authorization
+
+To enable mandatory database node authorization, add the following configuration blocks to the [static cluster configuration](../../reference/configuration/index.md) file:
+
+1. At the root level, add the `client_certificate_authorization` block to define the requirements for the "Subject" field of trusted node certificates. For example:
+
+    ```yaml
+    client_certificate_authorization:
+      request_client_certificate: true
+      client_certificate_definitions:
+        - member_groups: ["registerNode@cert"]
+          subject_terms:
+          - short_name: "O"
+            values: ["YDB"]
+    ```
+
+    Add other certificate validation settings [as defined in the documentation](../../reference/configuration/node-authentication.md), if required.
+
+    If the certificate is successfully verified and the components of the "Subject" field comply with the requirements defined in the `subject_terms` sub-block, the connection will be assigned the access subjects listed in the `member_groups` parameter. To distinguish these subjects from other user groups and accounts, their names typically have the `@cert` suffix.
+
+1. Add the `register_dynamic_node_allowed_sids` element to the cluster authentication settings `security_config` block, and list the subjects permitted for database node registration. For internal technical reasons, the list must include the `root@builtin` element. Example:
+
+    ```yaml
+    domains_config:
+        ...
+        security_config:
+            enforce_user_token_requirement: true
+            monitoring_allowed_sids:
+            ...
+            administration_allowed_sids:
+            ...
+            viewer_allowed_sids:
+            ...
+            register_dynamic_node_allowed_sids:
+            - "root@builtin" # required for internal technical reasons
+            - "registerNode@cert"
+    ```
+
+    For more detailed information on configuring cluster authentication parameters, see the [relevant documentation section](../../reference/configuration/index.md#security-access-levels).
+
+1. Deploy the static configuration files on all cluster nodes either manually, or [using the Ansible playbook action](../ansible/update-config.md).
+
+1. Perform the rolling restart of storage nodes by [using ydbops](../../reference/ydbops/scenarios.md) or [Ansible playbook action](../ansible/restart.md).
+
+1. Perform the rolling restart of database nodes through ydbops or Ansible playbooks.

ydb/docs/en/core/maintenance/manual/dynamic-config.md

@@ -1,4 +1,4 @@
-## Dynamic cluster configuration
+# Dynamic cluster configuration
 
 Dynamic configuration allows running dynamic [nodes](../../concepts/cluster/common_scheme_ydb#nodes) by configuring them centrally without manually distributing files across the nodes. {{ ydb-short-name }} acts as a configuration management system, providing tools for reliable storage, versioning, and delivery of configurations, as well as a [DSL (Domain Specific Language)](./dynamic-config-selectors.md) for overriding parts of the configuration for specific groups of nodes. The configuration is a YAML document and is an extended version of the static configuration:
 
@@ -8,83 +8,151 @@ Dynamic configuration allows running dynamic [nodes](../../concepts/cluster/comm
 
 This configuration is uploaded to the cluster, where it is reliably stored and delivered to each dynamic node upon startup. [Certain settings](#dynamic-kinds) are updated on the fly without restarting nodes. Using dynamic configuration, you can centrally solve the following tasks:
 
-* Switch logging settings for components for the entire cluster or specific groups of nodes;
-* Enable experimental features (feature flags) on specific databases;
-* Change actor system settings on individual nodes or groups of nodes;
-* And more.
+* Change logging levels for all or specific components across the entire cluster or for specific groups of nodes.
+* Enable experimental features (feature flags) on specific databases.
+* Change actor system settings on individual nodes or groups of nodes.
 
-### Configuration examples {#example}
+## Preparing to use the dynamic configuration {#preparation}
+
+The following tasks should be performed before using the dynamic configuration in the cluster:
+
+1. Enable [database node authentication and authorization](../../devops/manual/node-authorization.md).
+
+1. Export the current settings from the [CMS](../../concepts/glossary.md#cms) in YAML format using the following command if [CMS-based configuration management](cms.md) has been used in the cluster:
+
+    ```bash
+    ./ydbd -s grpcs://<node1.ydb.tech>:2135 --ca-file ca.crt --token-file ydbd-token \
+         admin console configs dump-yaml > dynconfig.yaml
+    ```
+
+    Before running the command shown above, obtain the authentication token using the `ydb auth get-token` command, as detailed in the [cluster initial deployment procedure](../../devops/manual/initial-deployment.md#initialize-cluster).
+
+1. Prepare the initial dynamic configuration file:
+
+   * If there are non-empty CMS settings exported in the previous step, adjust the YAML file with the exported CMS settings:
+      * Add the `metadata` section based on the [configuration example](#example).
+      * Add the `yaml_config_enabled: true` parameter to the `config` section.
+   * If there are no previous CMS-based settings, use the [minimal configuration example](#example).
+   * For clusters using TLS encryption for [actor system interconnect](../../concepts/glossary.md#actor-system-interconnect), add the [interconnect TLS settings](../../reference/configuration/tls.md#interconnect) to the `config` section.
+
+1. Apply the dynamic configuration settings file to the cluster:
+
+    ```bash
+    # Apply the dynconfig.yaml on the cluster
+    {{ ydb-cli }} admin config replace -f dynconfig.yaml
+    ```
+
+{% note info %}
+
+The legacy configuration management via CMS will become unavailable after enabling dynamic configuration support on the {{ ydb-short-name }} cluster.
+
+{% endnote %}
+
+## Configuration examples {#example}
 
 Example of a minimal dynamic configuration for a single-datacenter cluster:
 
 ```yaml
 # Configuration metadata.
 # This field is managed by the server.
 metadata:
- # Cluster name from the cluster_uuid parameter set during cluster installation, or "", if the parameter is not set.
- cluster: unknown
- # Configuration file identifier, always increments by 1 starting from 1.
- # Automatically increases when a new configuration is uploaded to the server.
- version: 1
+  # Cluster name from the cluster_uuid parameter set during cluster installation, or "", if the parameter is not set.
+  cluster: ""
+  # Configuration file identifier, always increments by 1 starting from 0.
+  # Automatically increases when a new configuration is uploaded to the server.
+  version: 0
 # Main cluster configuration. All values here are applied by default unless overridden by selectors.
 # Content is similar to the static cluster configuration.
 config:
-# It must always be set to true when using YAML configuration.
- yaml_config_enabled: true
-# Actor system configuration, as by default, this section is used only by dynamic nodes.
-# Configuration is set specifically for them.
- actor_system_config:
-# Automatic configuration selection for the node based on type and available cores.
- use_auto_config: true
-# HYBRID || COMPUTE || STORAGE — node type.
- node_type: COMPUTE
-# Number of cores.
- cpu_count: 4
-# Root domain configuration of the {{ ydb-short-name }} cluster.
- domains_config:
- domain:
- - name: Root
- storage_pool_types:
- - kind: ssd
- pool_config:
- box_id: 1
- erasure_species: none
- kind: ssd
- pdisk_filter:
- - property:
- - type: SSD
- vdisk_kind: Default
- explicit_mediators: [72075186232426497, 72075186232426498, 72075186232426499]
- explicit_coordinators: [72075186232360961, 72075186232360962, 72075186232360963]
- explicit_allocators: [72075186232492033, 72075186232492034, 72075186232492035]
- state_storage:
- - ssid: 1
- ring:
- nto_select: 5
- node: [1, 2, 3, 4, 5, 6, 7, 8]
- hive_config:
- - hive_uid: 1
- hive: 72057594037968897
-# Channel profile configuration for tablets.
- channel_profile_config:
- profile:
- - profile_id: 0
- channel:
- - &default_channel
- erasure_species: block-4-2
- pdisk_category: 1
- vdisk_category: Default
- - *default_channel
- - *default_channel
+  # It must always be set to true when using YAML configuration.
+  yaml_config_enabled: true
+  # Actor system configuration, as by default, this section is used only by dynamic nodes.
+  # Configuration is set specifically for them.
+  actor_system_config:
+    # Automatic configuration selection for the node based on type and available cores.
+    use_auto_config: true
+    # HYBRID || COMPUTE || STORAGE — node type.
+    node_type: COMPUTE
+    # Number of cores.
+    cpu_count: 14
 allowed_labels: {}
 selector_config: []
 ```
 
 Detailed configuration parameters are described on the [{#T}](../../reference/configuration/index.md) page.
 
-By default, the cluster configuration is empty and has version 1. When applying a new configuration, the uploaded configuration's version is compared and automatically incremented by one.
+By default, the cluster configuration is assigned version 1. When applying a new configuration, the system compares the uploaded configuration's version with the value specified in the YAML file. If the versions match, the current version number is automatically incremented by one.
+
+Below is a more comprehensive example of a dynamic configuration that defines typical global parameters as well as parameters specific to a particular database:
+
+```yaml
+---
+metadata:
+  kind: MainConfig
+  cluster: ""
+  version: 1
+config:
+  yaml_config_enabled: true
+  table_profiles_config:
+    table_profiles:
+    - name: default
+      compaction_policy: default
+      execution_policy: default
+      partitioning_policy: default
+      storage_policy: default
+      replication_policy: default
+      caching_policy: default
+    compaction_policies:
+    - name: default
+    execution_policies:
+    - name: default
+    partitioning_policies:
+    - name: default
+      auto_split: true
+      auto_merge: true
+      size_to_split: 2147483648
+    storage_policies:
+    - name: default
+      column_families:
+      - storage_config:
+          sys_log:
+            preferred_pool_kind: ssd
+          log:
+            preferred_pool_kind: ssd
+          data:
+            preferred_pool_kind: ssd
+    replication_policies:
+    - name: default
+    caching_policies:
+    - name: default
+  interconnect_config:
+    encryption_mode: REQUIRED
+    path_to_certificate_file: "/opt/ydb/certs/node.crt"
+    path_to_private_key_file: "/opt/ydb/certs/node.key"
+    path_to_ca_file: "/opt/ydb/certs/ca.crt"
+allowed_labels:
+  node_id:
+    type: string
+  host:
+    type: string
+  tenant:
+    type: string
+selector_config:
+- description: Custom settings for testdb
+  selector:
+    tenant: /cluster1/testdb
+  config:
+    shared_cache_config:
+      memory_limit: 34359738368
+    feature_flags: !inherit
+      enable_views: true
+    actor_system_config:
+      use_auto_config: true
+      node_type: COMPUTE
+      cpu_count: 14
+```
 
-### Updating dynamic configuration
+## Updating the dynamic configuration
 
 ```bash
 # Fetch the cluster configuration
@@ -98,17 +166,17 @@ vim dynconfig.yaml
 Additional configuration options are described on the [selectors](./dynamic-config-selectors.md) and [temporary configuration](./dynamic-config-volatile-config.md) pages.
 All commands for working with configuration are described in the [{#T}](../../reference/ydb-cli/configs.md) section.
 
-### Operation mechanism
+## Operation mechanism
 
-#### Configuration update from the administrator's perspective
+### Configuration update from the administrator's perspective
 
 1. The configuration file is uploaded by the user using a [grpc call](https://github.com/ydb-platform/ydb/blob/5251c9ace0a7617c25d50f1aa4d0f13e3d56f985/ydb/public/api/grpc/draft/ydb_dynamic_config_v1.proto#L22) or [{{ ydb-short-name }} CLI](../../reference/ydb-cli/index.md) to the cluster.
 2. The file is checked for validity, basic constraints, version correctness, cluster name correctness, and the correctness of the configurations obtained after DSL transformation are verified.
 3. The configuration version in the file is incremented by one.
 4. The file is reliably stored in the cluster using the Console [tablet](../../concepts/glossary.md#tablet).
 5. File updates are distributed across the cluster nodes.
 
-#### Configuration update from the cluster node's perspective
+### Configuration update from the cluster node's perspective
 
 1. Each node requests the entire configuration at startup.
 2. Upon receiving the configuration, the node [generates the final configuration](./dynamic-config-selectors.md#selectors-resolve) for its set of [labels](./dynamic-config-selectors.md#selectors-intro).
@@ -118,11 +186,11 @@ All commands for working with configuration are described in the [{#T}](../../re
 
 Steps 1 and 2 are performed only for dynamic cluster nodes.
 
-#### Configuration versioning
+### Configuration versioning
 
 This mechanism prevents concurrent configuration modifications and makes updates idempotent. When a modification request is received, the server compares the version of the received modification with the stored one. If the version is one less, the configurations are compared: if they are identical, it means the user is attempting to upload the configuration again, the user receives OK, and the cluster configuration is not updated. If the version matches the current one on the cluster, the configuration is replaced with the new one, and the version field is incremented by one. In all other cases, the user receives an error.
 
-### Dynamically updated settings {#dynamic-kinds}
+## Dynamically updated settings {#dynamic-kinds}
 
 Some system settings are updated without restarting nodes. To change them, upload a new configuration and wait for it to propagate across the cluster.
 
@@ -138,7 +206,7 @@ List of dynamically updated settings:
 
 The list may be expanded in the future.
 
-### Limitations
+## Limitations
 
 * Using more than 30 different [labels](./dynamic-config-selectors.md) in [selectors](./dynamic-config-selectors.md) can lead to validation delays of several seconds, as {{ ydb-short-name }} needs to check the validity of each possible final configuration. The number of values for a single label has much less impact.
-* Using large files (more than 500KiB for a cluster with 1000 nodes) can lead to increased network traffic in the cluster when updating the configuration. The traffic volume is directly proportional to the number of nodes and the configuration size.
+* Using large files (more than 500KiB for a cluster with 1000 nodes) can lead to increased network traffic in the cluster when updating the configuration. The traffic volume is directly proportional to the number of nodes and the configuration size.