[META] Securing the Logstash HTTP API

[yaauie]

The Logstash HTTP API is currently unsecured. In order to prevent malicious actors on the network from abusing access to the API, we have historically taken two approaches:

• bind the HTTP API to the local loopback interface by default, AND
• avoid adding features to the API that could be abused.

Because it is not always practical for the Logstash API to be bound to the host's loopback interface, and because we would like to add a number of features that have the potential for abuse if left unguarded, we need to secure the HTTP API and ensure that only verifiably known, authorized users have access.

In doing so, we need to support a number of practical scenarios:

• Unsecured HTTP (status quo, bound to loopback interface by default);
• SSL-only Client Authentication; AND
• SLL + HTTP Authentication.

When configured in a way that allows us to establish trust of a client, we can safely bind to any interface and should do so by default.

Securing the connection (SSL/TLS)

Adding TLS/SSL allows Logstash to establish privacy with the client, which allows the client to safely send credentials if they are required. In some configurations, TLS allows Logstash to establish a baseline trust of the client. TLS is therefore either the means by which trust of the client is established, or a pre-requisite for doing so.

We will add a number of settings that mirror the intent behind similar Elasticsearch xpack.security.http.ssl.* settings, grouped under Logstash's api.ssl.*:

• api.ssl.enabled: boolean, default to false for now. When false, Logstash should warn about other SSL-related settings being provided.
• api.ssl.keystore.path: the path to a readable X509 Keystore, required when api.ssl.enabled: true.
• api.ssl.keystore.password: optional password to be used to access the provided keystore.
• api.ssl.client_authentication: one of none, optional, or required, mirroring xpack.security.http.ssl.client_authentication and mapping to Puma's none, peer, and force_peer respectively.
• api.ssl.supported_protocols: accepts an unordered list of supported TLS protocols (implementation tricky: underlying PUMA only allows us to explicitly disable TLSv1 and TLSv1.1, so we may need to require that the provided list include TLSv1.2 etc until such opt-out support is added to Puma; if we can't implement in this way, we should add one or more distinct api.ssl.* settings that do not overlap api.ssl.supported_protocols)

Establishing trust of client using SSL + HTTP Authentication

When configured with SSL enabled in any client_authentication mode, we have mutual proof-of-privacy with the client, and can use HTTP Authentication to establish authorization.

We will add a new option api.auth.type, whose valid values are either none or basic, so that we can add additional HTTP Authorization schemes or backends in the future.

To configure basic auth on the API, a user will add api.auth.basic.username and api.auth.basic.password directly in their logstash.yml configuration. Users who wish to store their credentials in the Logstash keystore will need to reference a value stored there using variable expansion.

• Failure to specify both values when api.auth.type: basic is considered a configuration error.
• Setting either value when api.auth.type is not basic is considered a configuration error.
• Setting api.auth.type: basic without auth.ssl.enabled will produce a warning and will not qualify the configuration to default to any-interface binding.
• Setting api.ssl.enabled: true and api.auth.type: basic qualifies the API for defaulting to bind to any interface (0.0.0.0).

Establishing trust of client using SSL-only client authentication

In some organizations, SSL-only client authentication (api.ssl.enabled: true AND api.ssl.client_authentication: required) is sufficient to establish trust of the client.

• Setting api.ssl.enabled: true and api.ssl.client_authentication: required qualifies the API for defaulting to bind to any interface (0.0.0.0).

API Settings Mapping Overhaul

Logstash currently has its API-related settings nested under http, even when some of those settings don't map to HTTP-related things. We are adding a number of API-related settings here that don't directly apply to http either, and adding them there or as top-level ssl is liable to add confusion as user-configured pipelines also have HTTP- and SSL-related settings that would be unaffected by these settings here.

As a part of securing the HTTP API, we therefore will be re-naming a number of settings to be consistently grouped under a consistent top-level api grouping.

• api.enabled (previously: http.enabled)
• api.http.host (previously: http.host)
• api.http.port (previously: http.port)
• api.environment (previously: http.environment)

Notable behaviours:

• when both a canonical setting and its deprecated counterpart are used, it is a configuration error.
• when a deprecated-and-renamed setting is used, a deprecation warning is emitted to the deprecation logger and the canonical setting is set.
• we are too close to 8.0 to consider removing these newly-deprecated http.* settings; they will be sticking with us for a while to allow users to make the switch in their own time.

[yaauie]

Status Update (7.16/8.0):

Logstash 7.16 and 8.0 are both shipping with a subset of the behaviour defined in this meta-issue:

• API-related settings regrouped under the api.* namespace, as specified in this meta-issue; old options are still supported into 8.0 and beyond, but a deprecation warning will be used to encourage migrating configs to the newly-namespaced setting names
• TLS/SSL is implemented (api.ssl.enabled: true) which requires the path to a P12- or JKS-formatted keystore (api.ssl.keystore.path) and associated credentials (api.ssl.keystore.password))
• Basic Auth is implemented (api.auth.type: basic, which requires api.auth.basic.username and api.auth.basic.password)
• When configured with both api.ssl.enabled: true and api.auth.type: basic, and without explicit instructions about what host to bind to (api.http.port or its deprecated http.port counterpart), it will bind to 0.0.0.0 (all IPv4 interfaces).

The remainder of this specification is NOT yet implemented:

• MTLS, or any behaviour that relies on api.ssl.client_authentication
• SSL protocol selection (api.ssl.supported_protocols)