Update docs

hitachienergy · Feb 10, 2020 · 0618a53 · 0618a53
1 parent 5e72a88
commit 0618a53
Show file tree

Hide file tree

Showing 8 changed files with 111 additions and 50 deletions.
diff --git a/CHANGELOG-0.5.md b/CHANGELOG-0.5.md
@@ -4,7 +4,9 @@
 
 ### Added
 
-- [#854](https://github.com/epiphany-platform/epiphany/issues/854) - PGBouncer implementation
+- PostgreSQL: PGBouncer implementation [#854](https://github.com/epiphany-platform/epiphany/issues/854)
+- PostgreSQL: pgAudit extension for audit logging [#905](https://github.com/epiphany-platform/epiphany/pull/905)
+- PostgreSQL: Send logs to Elasticsearch
 
 ## [0.5.1] 2020-01-23
 

diff --git a/docs/design-docs/postgresql-auditing/postgresql-auditing.md b/docs/design-docs/postgresql-auditing/postgresql-auditing.md
diff --git a/...esign-docs/database/connection-pooling.md → ...ign-docs/postgresql/connection-pooling.md b/...esign-docs/database/connection-pooling.md → ...ign-docs/postgresql/connection-pooling.md
diff --git a/docs/design-docs/database/ha_cluster.PNG → docs/design-docs/postgresql/ha_cluster.PNG b/docs/design-docs/database/ha_cluster.PNG → docs/design-docs/postgresql/ha_cluster.PNG
diff --git a/docs/design-docs/postgresql/postgresql-auditing.md b/docs/design-docs/postgresql/postgresql-auditing.md
@@ -0,0 +1,43 @@
+# Epiphany PostgreSQL auditing design document
+
+Affected version: 0.5.x
+
+## Goals
+
+Provide functionality to perform auditing of operations performed on PostgreSQL.
+
+## Use cases
+
+For SOX and other regulations compliance platform should provide auditing function for PostgreSQL database.
+This should be set via Epiphany automation in Epiphany configuration yaml.
+
+## Example use
+
+In configuration for PostgreSQL we can add additional parameters, that could configure additional properties of PostgreSQL.
+Config similar to proposed below can be used to configure auditing with using pgaudit.
+
+```yaml
+kind: configuration/postgresql
+title: PostgreSQL
+name: default
+specification:
+  ...
+  extensions:
+    pgaudit:
+      enabled: false
+      shared_preload_libraries:
+        - pgaudit
+      config_file_parameters:
+        pgaudit.log: 'all, -misc'
+        log_connections: 'on'
+        log_disconnections: 'on'
+        log_line_prefix: "'%m [%p] %q%u@%d,host=%h '"
+        log_statement: 'none'
+  ...
+```
+
+## Design proposal
+
+Add to PostgreSQL configuration additional settings, that would install and configure pgaudit extension.
+For RHEL we use PostgreSQL installed from Software Collections repository, which doesn't provide pgaudit package for PostgreSQL
+versions older than 12. For this reason, on RHEL pgaudit will be installed from PostgreSQL repository.
diff --git a/docs/home/COMPONENTS.md b/docs/home/COMPONENTS.md
@@ -25,4 +25,5 @@ Note that versions are default versions and can be changed in certain cases thro
 | jmx_exporter          | 0.12.0  | https://github.com/prometheus/jmx_exporter            | [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) |
 | PostgresSQL           | 10      | https://www.postgresql.org/                           | https://opensource.org/licenses/postgresql                        |
 | HAProxy               | 1.8     | https://www.haproxy.org/                              | [GNU General Public License 2.0](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) |
-| PGBouncer             | 1.10.0  | https://github.com/pgbouncer/pgbouncer                | [ISC License](https://opensource.org/licenses/isc)             |
+| PGAudit               | 1.2.0   | https://github.com/pgaudit/pgaudit                    | [PostgreSQL license](http://www.postgresql.org/about/licence/)    |
+| PGBouncer             | 1.10.0  | https://github.com/pgbouncer/pgbouncer                | [ISC License](https://opensource.org/licenses/isc)                |
diff --git a/docs/home/HOWTO.md b/docs/home/HOWTO.md
@@ -59,7 +59,8 @@
 - [Databases](./howto/DATABASES.md)
   - [How to configure PostgreSQL](./howto/DATABASES.md#how-to-configure-postgresql)
   - [How to configure PostgreSQL replication](./howto/DATABASES.md#how-to-configure-postgresql-replication)
-  - [How to set up Postgresql connection pooling](./howto/DATABASES.md#how-to-set-up-postgresql-connection-pooling)
+  - [How to set up PostgreSQL connection pooling](./howto/DATABASES.md#how-to-set-up-postgresql-connection-pooling)
+  - [How to set up PostgreSQL audit logging](./howto/DATABASES.md#how-to-set-up-postgresql-audit-logging)
   - [How to start working with OpenDistro for Elasticsearch](./howto/DATABASES.md#how-to-start-working-with-opendistro-for-elasticsearch)
   - [How to start working with Apache Ignite Stateful setup](./howto/DATABASES.md#how-to-start-working-with-apache-ignite-stateful-setup)
   - [How to start working with Apache Ignite Stateless setup](./howto/DATABASES.md#how-to-start-working-with-apache-ignite-stateless-setup)

diff --git a/docs/home/howto/DATABASES.md b/docs/home/howto/DATABASES.md
@@ -1,22 +1,22 @@
 ## How to configure PostgreSQL
 
-To configure PostgreSQL login to server using ssh and switch to postgres user with command:
+To configure PostgreSQL, login to server using ssh and switch to `postgres` user with command:
 
 ```bash
 sudo -u postgres -i
 ```
 
-And then configure database server using psql according to your needs and
-PostgreSQL documentation, to which link you can find at [here](https://www.postgresql.org/docs/)
+Then configure database server using psql according to your needs and
+[PostgreSQL documentation](https://www.postgresql.org/docs/).
 
 ## How to configure PostgreSQL replication
 
-In order to configure PostgreSQL replication add to your data.yaml a block similar to the one below to core section:
+In order to configure PostgreSQL replication, add to your data.yaml a block similar to the one below to core section:
 
 ```yaml
 kind: configuration/postgresql
 name: default
-title: Postgresql
+title: PostgreSQL
 version: 0.4.1
 provider: aws
 specification:
@@ -30,18 +30,72 @@ specification:
 If `enabled` is set to `yes` in `replication`, then Epiphany will automatically create cluster of master and slave server
 with replication user with name and password specified in data.yaml.
 
-## How to set up Postgresql connection pooling
+## How to set up PostgreSQL connection pooling
 
-Postgresql connection pooling in Epiphany is served by PGBouncer application. This might be added as a feature if needed. Simplest configuration runs PGBouncer on Postgresql master node. This needs to be enabled in configuration yaml file:
+PostgreSQL connection pooling in Epiphany is served by PGBouncer application. This might be added as a feature if needed.
+Simplest configuration runs PGBouncer on PostgreSQL master node. This needs to be enabled in configuration yaml file:
 
 ```yaml
+kind: configuration/postgresql
+...
 specification:
   additional_components:
     pgbouncer:
       enabled: yes
+  ...
 ```
 PGBouncer listens on standard port 6432. Basic configuration is just template, with very limited access to database. This is because security reasons. [Configuration needs to be tailored according component documentation and stick to security rules and best practices](http://www.pgbouncer.org/).
 
+## How to set up PostgreSQL audit logging
+
+Audit logging of database activities is available through the PostgreSQL Audit Extension: [pgAudit](https://github.com/pgaudit/pgaudit/blob/REL_10_STABLE/README.md).
+It provides session and/or object audit logging via the standard PostgreSQL log.
+
+pgAudit may generate a large volume of logging, which has an impact on performance and log storage.
+For this reason, pgAudit is not enabled by default.
+
+To install and configure pgAudit, add to your configuration yaml file a doc similar to the following:
+
+```yaml
+kind: configuration/postgresql
+title: PostgreSQL
+name: default
+provider: aws
+version: 0.5.2
+specification:
+  extensions:
+    pgaudit:
+      enabled: yes
+      config_file_parameters:
+        ## postgresql standard
+        log_connections: 'off'
+        log_disconnections: 'off'
+        log_statement: 'none'
+        log_line_prefix: "'%m [%p] %q%u@%d,host=%h '"
+        ## pgaudit specific, see https://github.com/pgaudit/pgaudit/blob/REL_10_STABLE/README.md#settings
+        pgaudit.log: "'write, function, role, ddl' # 'misc_set' is not supported for PG 10"
+        pgaudit.log_catalog: 'off # to reduce overhead of logging'
+        # the following first 2 parameters are set to values that make it easier to access audit log per table
+        # change their values to the opposite if you need to reduce overhead of logging
+        pgaudit.log_relation: 'on # separate log entry for each relation'
+        pgaudit.log_statement_once: 'off'
+        pgaudit.log_parameter: 'on'
+```
+
+If `specification.extensions.pgaudit.enabled` is set to `yes`, Epiphany will install pgAudit package
+and add pgaudit extension to be loaded in [shared_preload_libraries](http://www.postgresql.org/docs/10/static/runtime-config-client.html#GUC-SHARED-PRELOAD-LIBRARIES).
+Settings defined in `config_file_parameters` section are populated to Epiphany managed PostgreSQL configuration
+file. Using this section, you can set any additional parameter if needed (e.g. `pgaudit.role`).
+
+To configure pgAudit according to your needs, see [pgAudit documentation](https://github.com/pgaudit/pgaudit/blob/REL_10_STABLE/README.md#settings).
+
+Once Epiphany installation is complete, there is one manual action (per each database). Connect to your database
+using a client (like psql) and load pgaudit extension into current database by running command:
+
+```sql
+CREATE EXTENSION pgaudit;
+```
+
 ## How to start working with OpenDistro for Elasticsearch
 
 OpenDistro for Elasticsearch is [an Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise security, alerting, SQL](https://opendistro.github.io/for-elasticsearch/). In order to start working with OpenDistro change machines count to value greater than 0 in your cluster configuration: