diff --git a/CHANGELOG-0.5.md b/CHANGELOG-0.5.md index d538d446ba..72e33e5579 100644 --- a/CHANGELOG-0.5.md +++ b/CHANGELOG-0.5.md @@ -4,7 +4,9 @@ ### Added -- [#854](https://github.com/epiphany-platform/epiphany/issues/854) - PGBouncer implementation +- PostgreSQL: PGBouncer implementation [#854](https://github.com/epiphany-platform/epiphany/issues/854) +- PostgreSQL: pgAudit extension for audit logging [#905](https://github.com/epiphany-platform/epiphany/pull/905) +- PostgreSQL: Send logs to Elasticsearch ## [0.5.1] 2020-01-23 diff --git a/docs/design-docs/postgresql-auditing/postgresql-auditing.md b/docs/design-docs/postgresql-auditing/postgresql-auditing.md deleted file mode 100644 index e48dd2359b..0000000000 --- a/docs/design-docs/postgresql-auditing/postgresql-auditing.md +++ /dev/null @@ -1,40 +0,0 @@ -# Epiphany PostgreSQL auditing design document - -Affected version: 0.5.x - -## Goals - -Provide functionality to perform auditing of operations performed on PostgreSQL. - -## Use cases - -For SOX and other regulations compliance platform should provide auditing function for PostgreSQL database. -This should be set via Epiphany automation in Epiphany configuration yaml. - -## Example use - -In configuration for PostgreSQL we can add additional parameters, that could configure additional properties of PostgreSQL. -Config similar to proposed below can be used to configure auditing with using pgaudit. - -```yaml -kind: configuration/postgresql -title: "Postgresql" -name: default -specification: - ... - additional_config: - shared_preload_libraries: 'pgaudit' - pgaudit.log: 'all, -misc' - log_connections: on - log_disconnections: on - log_line_prefix: '<%m:%r:%u@%d:[%p]:> ' - log_statement: 'none' - ... -``` - - -## Design proposal - -Add to configuration for PostgreSQL additional parameters, that would enable auditing and install additional modules to enhance PostgreSQL -configuration. In case of auditing this would install pgaudit, which can be configured from Epiphany configuration yaml level. For Red Hat -this would require to replace PostgreSQL from Software Collections with version from PostgreSQL repository. diff --git a/docs/design-docs/database/connection-pooling.md b/docs/design-docs/postgresql/connection-pooling.md similarity index 100% rename from docs/design-docs/database/connection-pooling.md rename to docs/design-docs/postgresql/connection-pooling.md diff --git a/docs/design-docs/database/ha_cluster.PNG b/docs/design-docs/postgresql/ha_cluster.PNG similarity index 100% rename from docs/design-docs/database/ha_cluster.PNG rename to docs/design-docs/postgresql/ha_cluster.PNG diff --git a/docs/design-docs/postgresql/postgresql-auditing.md b/docs/design-docs/postgresql/postgresql-auditing.md new file mode 100644 index 0000000000..8d129ed894 --- /dev/null +++ b/docs/design-docs/postgresql/postgresql-auditing.md @@ -0,0 +1,43 @@ +# Epiphany PostgreSQL auditing design document + +Affected version: 0.5.x + +## Goals + +Provide functionality to perform auditing of operations performed on PostgreSQL. + +## Use cases + +For SOX and other regulations compliance platform should provide auditing function for PostgreSQL database. +This should be set via Epiphany automation in Epiphany configuration yaml. + +## Example use + +In configuration for PostgreSQL we can add additional parameters, that could configure additional properties of PostgreSQL. +Config similar to proposed below can be used to configure auditing with using pgaudit. + +```yaml +kind: configuration/postgresql +title: PostgreSQL +name: default +specification: + ... + extensions: + pgaudit: + enabled: false + shared_preload_libraries: + - pgaudit + config_file_parameters: + pgaudit.log: 'all, -misc' + log_connections: 'on' + log_disconnections: 'on' + log_line_prefix: "'%m [%p] %q%u@%d,host=%h '" + log_statement: 'none' + ... +``` + +## Design proposal + +Add to PostgreSQL configuration additional settings, that would install and configure pgaudit extension. +For RHEL we use PostgreSQL installed from Software Collections repository, which doesn't provide pgaudit package for PostgreSQL +versions older than 12. For this reason, on RHEL pgaudit will be installed from PostgreSQL repository. diff --git a/docs/home/COMPONENTS.md b/docs/home/COMPONENTS.md index c7b66a0b9a..514e63058c 100644 --- a/docs/home/COMPONENTS.md +++ b/docs/home/COMPONENTS.md @@ -25,4 +25,5 @@ Note that versions are default versions and can be changed in certain cases thro | jmx_exporter | 0.12.0 | https://github.com/prometheus/jmx_exporter | [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) | | PostgresSQL | 10 | https://www.postgresql.org/ | https://opensource.org/licenses/postgresql | | HAProxy | 1.8 | https://www.haproxy.org/ | [GNU General Public License 2.0](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) | -| PGBouncer | 1.10.0 | https://github.com/pgbouncer/pgbouncer | [ISC License](https://opensource.org/licenses/isc) | +| PGAudit | 1.2.0 | https://github.com/pgaudit/pgaudit | [PostgreSQL license](http://www.postgresql.org/about/licence/) | +| PGBouncer | 1.10.0 | https://github.com/pgbouncer/pgbouncer | [ISC License](https://opensource.org/licenses/isc) | diff --git a/docs/home/HOWTO.md b/docs/home/HOWTO.md index b40cd44178..f35645152d 100644 --- a/docs/home/HOWTO.md +++ b/docs/home/HOWTO.md @@ -59,7 +59,8 @@ - [Databases](./howto/DATABASES.md) - [How to configure PostgreSQL](./howto/DATABASES.md#how-to-configure-postgresql) - [How to configure PostgreSQL replication](./howto/DATABASES.md#how-to-configure-postgresql-replication) - - [How to set up Postgresql connection pooling](./howto/DATABASES.md#how-to-set-up-postgresql-connection-pooling) + - [How to set up PostgreSQL connection pooling](./howto/DATABASES.md#how-to-set-up-postgresql-connection-pooling) + - [How to set up PostgreSQL audit logging](./howto/DATABASES.md#how-to-set-up-postgresql-audit-logging) - [How to start working with OpenDistro for Elasticsearch](./howto/DATABASES.md#how-to-start-working-with-opendistro-for-elasticsearch) - [How to start working with Apache Ignite Stateful setup](./howto/DATABASES.md#how-to-start-working-with-apache-ignite-stateful-setup) - [How to start working with Apache Ignite Stateless setup](./howto/DATABASES.md#how-to-start-working-with-apache-ignite-stateless-setup) diff --git a/docs/home/howto/DATABASES.md b/docs/home/howto/DATABASES.md index a3c4f5c50b..3aced0f7d2 100644 --- a/docs/home/howto/DATABASES.md +++ b/docs/home/howto/DATABASES.md @@ -1,22 +1,22 @@ ## How to configure PostgreSQL -To configure PostgreSQL login to server using ssh and switch to postgres user with command: +To configure PostgreSQL, login to server using ssh and switch to `postgres` user with command: ```bash sudo -u postgres -i ``` -And then configure database server using psql according to your needs and -PostgreSQL documentation, to which link you can find at [here](https://www.postgresql.org/docs/) +Then configure database server using psql according to your needs and +[PostgreSQL documentation](https://www.postgresql.org/docs/). ## How to configure PostgreSQL replication -In order to configure PostgreSQL replication add to your data.yaml a block similar to the one below to core section: +In order to configure PostgreSQL replication, add to your data.yaml a block similar to the one below to core section: ```yaml kind: configuration/postgresql name: default -title: Postgresql +title: PostgreSQL version: 0.4.1 provider: aws specification: @@ -30,18 +30,72 @@ specification: If `enabled` is set to `yes` in `replication`, then Epiphany will automatically create cluster of master and slave server with replication user with name and password specified in data.yaml. -## How to set up Postgresql connection pooling +## How to set up PostgreSQL connection pooling -Postgresql connection pooling in Epiphany is served by PGBouncer application. This might be added as a feature if needed. Simplest configuration runs PGBouncer on Postgresql master node. This needs to be enabled in configuration yaml file: +PostgreSQL connection pooling in Epiphany is served by PGBouncer application. This might be added as a feature if needed. +Simplest configuration runs PGBouncer on PostgreSQL master node. This needs to be enabled in configuration yaml file: ```yaml +kind: configuration/postgresql +... specification: additional_components: pgbouncer: enabled: yes + ... ``` PGBouncer listens on standard port 6432. Basic configuration is just template, with very limited access to database. This is because security reasons. [Configuration needs to be tailored according component documentation and stick to security rules and best practices](http://www.pgbouncer.org/). +## How to set up PostgreSQL audit logging + +Audit logging of database activities is available through the PostgreSQL Audit Extension: [pgAudit](https://github.com/pgaudit/pgaudit/blob/REL_10_STABLE/README.md). +It provides session and/or object audit logging via the standard PostgreSQL log. + +pgAudit may generate a large volume of logging, which has an impact on performance and log storage. +For this reason, pgAudit is not enabled by default. + +To install and configure pgAudit, add to your configuration yaml file a doc similar to the following: + +```yaml +kind: configuration/postgresql +title: PostgreSQL +name: default +provider: aws +version: 0.5.2 +specification: + extensions: + pgaudit: + enabled: yes + config_file_parameters: + ## postgresql standard + log_connections: 'off' + log_disconnections: 'off' + log_statement: 'none' + log_line_prefix: "'%m [%p] %q%u@%d,host=%h '" + ## pgaudit specific, see https://github.com/pgaudit/pgaudit/blob/REL_10_STABLE/README.md#settings + pgaudit.log: "'write, function, role, ddl' # 'misc_set' is not supported for PG 10" + pgaudit.log_catalog: 'off # to reduce overhead of logging' + # the following first 2 parameters are set to values that make it easier to access audit log per table + # change their values to the opposite if you need to reduce overhead of logging + pgaudit.log_relation: 'on # separate log entry for each relation' + pgaudit.log_statement_once: 'off' + pgaudit.log_parameter: 'on' +``` + +If `specification.extensions.pgaudit.enabled` is set to `yes`, Epiphany will install pgAudit package +and add pgaudit extension to be loaded in [shared_preload_libraries](http://www.postgresql.org/docs/10/static/runtime-config-client.html#GUC-SHARED-PRELOAD-LIBRARIES). +Settings defined in `config_file_parameters` section are populated to Epiphany managed PostgreSQL configuration +file. Using this section, you can set any additional parameter if needed (e.g. `pgaudit.role`). + +To configure pgAudit according to your needs, see [pgAudit documentation](https://github.com/pgaudit/pgaudit/blob/REL_10_STABLE/README.md#settings). + +Once Epiphany installation is complete, there is one manual action (per each database). Connect to your database +using a client (like psql) and load pgaudit extension into current database by running command: + +```sql +CREATE EXTENSION pgaudit; +``` + ## How to start working with OpenDistro for Elasticsearch OpenDistro for Elasticsearch is [an Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise security, alerting, SQL](https://opendistro.github.io/for-elasticsearch/). In order to start working with OpenDistro change machines count to value greater than 0 in your cluster configuration: