Merge branch 'feature/crt_passphrase_check' of github.com:NETWAYS/ans…

…ible-collection-elasticstack into feature/crt_passphrase_check

.github/workflows/test_full_stack.yml

@@ -31,7 +31,7 @@ jobs:
 
     strategy:
       fail-fast: false
-      max-parallel: 4
+      max-parallel: 2
       matrix:
         distro:
           - rockylinux8

.github/workflows/test_roles_pr.yml

@@ -30,6 +30,7 @@ on:
       - 'roles/**'
       - '.github/workflows/test_roles_pr.yml'
       - 'molecule/elasticstack_default/**'
+  merge_group:
 
 jobs:
   lint_full:
@@ -47,11 +48,11 @@ jobs:
 
     strategy:
       fail-fast: false
-      max-parallel: 4
+      max-parallel: 2
       matrix:
         distro:
-          - ubuntu2204
           - rockylinux8
+          - ubuntu2204
         scenario:
           - elasticstack_default
         release:

README.md

@@ -35,31 +35,34 @@ collections:
 
 You will need the following Ansible collections installed
 
-* community.general (probably already present)
+* `community.general` (probably already present)
+
+You will need these packages / libraries installed. Some very basic packages like `openssl` get handled by the collection if needed. The following list contains packages and libraries which only apply to special cases or need for you to decide on the installation method.
+
+* `passlib` Python library if you do not disable password hashing for logstash user and you want to use logstash role from this collection. It should be installed with pip on the Ansible controller.
 
 You may want the following Ansible roles installed. There other ways to achieve what they are doing but using them is easy and convenient.
 
-* geerlingguy.redis
-* openssl if you want to use Elastic Security
+* `geerlingguy.redis` if you want to use logstash role
 
 ### Supported systems
 
 We test the collection on the following Linux distributions. Each one with Elastic Stack 7 and 8.
 
+* Rocky Linux 9
 * Rocky Linux 8
-* Ubuntu 20.04 LTS
 * Ubuntu 22.04 LTS
+* Ubuntu 20.04 LTS
 * Debian 11
+* Debian 10
+* CentOS 8
 
 We know from personal experience, that the collections work in following combinations. Missing tests mostly come from incompatibilties between the distribution and our testing environment, not from problems with the collection itself.
 
 * CentOS 7 - Elastic Stack 7
 
 ### Known Issues
 
-There are known issues with the following Linux distributions.
-
-* Rocky Linux 9: The GnuPG key used by Elastic seems to be incompatible with this version of Rocky.
 
 ## Usage
 
@@ -69,6 +72,8 @@ Make sure all hosts that should be configured are part of your playbook. (See be
 
 You will want to have reliable DNS resolution or enter all hosts of the stack into your systems hosts files.
 
+The variable `elasticstack_no_log` can be set to `false` if you want to see the output of all tasks. It defaults to `true` because some tasks could reveal passwords in production.
+
 ### Versioning
 
 *elasticstack_version*: Version number of tools to install. Only set if you don't want the latest. (default: none).

docs/role-beats.md

@@ -10,7 +10,6 @@ Requirements
 
 You need to have the beats you want to install available in your software repositories. We provide a [role](./role-repos.md) for just that but if you have other ways of managing software, just make sure it's available. Alternatively you can install the Beats yourself.
 
-* `cryptography` >= 2.5 
 * `community.crypto` collection: ansible-galaxy collection install community.crypto
 
 Role Variables
@@ -87,7 +86,6 @@ beats_filebeat_journald_inputs:
 * *beats_loglevel*: Level of logging (for all beats) (Default: `info`)
 * *beats_logpath*: If logging to file, where to put logfiles (Default: `/var/log/beats`)
 * *beats_fields*: Fields that are added to every input in the configuration
-* *beats_manage_unzip*: Install `unzip` via package manager (Default: `true`)
 
 The following variables only apply if you use this role together with our other Elastic Stack roles.
 

docs/role-elasticsearch.md

@@ -9,11 +9,6 @@ If you use the role to set up security you, can use its CA to create certificate
 
 Please note that setting `elasticsearch_bootstrap_pw` as variable will only take effect when initialising Elasticsearch. Changes after starting elasticsearch for the first time will not change the bootstrap password for the instance and will lead to breaking tests.
 
-Requirements
-------------
-
-* `cryptography` >= 2.5
-
 Role Variables
 --------------
 

docs/role-kibana.md

@@ -5,11 +5,6 @@ Ansible Role: Kibana
 
 This roles installs and configures Kibana.
 
-Requirements
-------------
-
-* `cryptography` >= 2.5
-
 Role Variables
 --------------
 
@@ -18,6 +13,7 @@ Role Variables
 * *kibana_tls*: Whether to offer `https` for clients or not (default: `false`)
 * *kibana_tls_cert*: Path to the certificate Kibana should show to its clients (default: `/etc/kibana/certs/cert.pem`)
 * *kibana_tls_key*: Path to the key Kibana should use when communicating with clients (default: `/etc/kibana/certs/key.pem`)
+* *kibana_extra_config*: You can add arbitraty configuration options with this option. Just start it with `|-` and indent the following lines. So you can add as many lines and options to `kibana.yml` as you like. (default: none)
 
 * *kibana_security*: Activate TLS and authentication when connecting to Elasticsearch. **Note**: Only works when `elasticstack_full_stack` is enabled. (default: `true`)
 

docs/role-logstash.md

@@ -19,7 +19,10 @@ Requirements
 ------------
 
 * `community.general` collection
-* `cryptography` >= 2.5
+
+You will need these packages / libraries installed. Some very basic packages like `openssl` get handled by the collection if needed. The following list contains packages and libraries which only apply to special cases or need for you to decide on the installation method.
+
+* `passlib` Python library if you do not disable password hashing for logstash user. It should be installed with pip on the Ansible controller.
 
 You need to have the Elastic Repos configured on your system. You can use our [role](./role-repos.md)
 
@@ -67,6 +70,9 @@ Aside from `logstash.yml` we can manage Logstashs pipelines.
 * *logstash_security*: Enable X-Security (No default set, but will be activated when in full stack mode)
 * *logstash_user*: Name of the user to connect to Elasticsearch (Default: `logstash_writer`)
 * *logstash_password_hash*: Generate and use a hash from your `logstash_password` (default: `true`)
+* *logstash_password_hash_algorithm*: Password hashing algorithms. Value must be same as `xpack.security.authc.password_hashing.algorithm` (default: `bcrypt`)
+* *logstash_password_salt_length*: base64 encoded Salt character lenght. This value must be integer and must be compatible to the selected password hashing algorithms (default: `22`)
+* *logstash_password_hash_salt_seed*: A seed to generate random but idempotent salt on the elasticstack ca host. The salt will be used to create idempotent logstash hashed user password (default: `SeedChangeMe`)
 * *logstash_password*: Password of Elasticsearch user. It must be at least 6 characters long (default: `password`)
 * *logstash_user_indices*: Indices the user has access to (default: `'"ecs-logstash*", "logstash*", "logs*"'`)
 * *logstash_reset_writer_role*: Reset user and role with every run: (default: `true`)

molecule/beats_default/converge.yml

@@ -12,6 +12,7 @@
     elasticsearch_jna_workaround: true
     elasticsearch_disable_systemcallfilterchecks: true
     elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
+    elasticstack_no_log: false
   tasks:
     - name: Include Elastics repos role
       ansible.builtin.include_role:

molecule/beats_peculiar/converge.yml

@@ -21,6 +21,7 @@
     elasticsearch_jna_workaround: true
     elasticsearch_disable_systemcallfilterchecks: true
     elasticstack_full_stack: false
+    elasticstack_no_log: false
     beats_filebeat_mysql_slowlog_input: true
     beats_auditbeat: true
     beats_auditbeat_output: logstash

molecule/elasticsearch_cluster-oss/converge.yml

@@ -11,6 +11,7 @@
     elasticsearch_disable_systemcallfilterchecks: true
     elasticstack_release: 7
     elasticsearch_heap: "1"
+    elasticstack_no_log: false
   tasks:
     - name: Include Elastics repos role
       ansible.builtin.include_role:

molecule/elasticsearch_cluster-oss/molecule.yml

@@ -1,6 +1,8 @@
 ---
 dependency:
   name: galaxy
+  options:
+    requirements-file: requirements.yml
 driver:
   name: docker
 platforms:

molecule/elasticsearch_default/molecule.yml

@@ -1,6 +1,8 @@
 ---
 dependency:
   name: galaxy
+  options:
+    requirements-file: requirements.yml
 driver:
   name: docker
 platforms:

molecule/elasticsearch_no-security/converge.yml

@@ -12,6 +12,7 @@
     elasticsearch_disable_systemcallfilterchecks: true
     elasticsearch_heap: "1"
     elasticstack_release: 7
+    elasticstack_no_log: false
   tasks:
     - name: Include Elastics repos role
       ansible.builtin.include_role:

molecule/elasticsearch_no-security/molecule.yml

@@ -1,6 +1,8 @@
 ---
 dependency:
   name: galaxy
+  options:
+    requirements-file: requirements.yml
 driver:
   name: docker
 platforms:

molecule/elasticsearch_roles_calculation/converge.yml

@@ -14,6 +14,7 @@
       - data
     elasticsearch_heap: 1
     elasticsearch_check_calculation: true
+    elasticstack_no_log: false
   tasks:
     - name: Include Elastics repos role
       ansible.builtin.include_role:

molecule/elasticsearch_roles_calculation/molecule.yml

@@ -1,6 +1,8 @@
 ---
 dependency:
   name: galaxy
+  options:
+    requirements-file: requirements.yml
 driver:
   name: docker
 platforms:

molecule/elasticstack_default/converge.yml

@@ -13,31 +13,23 @@
     elasticsearch_jna_workaround: true
     elasticsearch_disable_systemcallfilterchecks: true
     elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
-    elasticsearch_heap: "2"
+    elasticsearch_heap: "1"
     elasticstack_full_stack: true
+    elasticstack_no_log: false
     logstash_pipeline_unsafe_shutdown: true
-    logstash_password_hash: false
     beats_filebeat_syslog_udp: true
     beats_filebeat_syslog_tcp: true
     beats_filebeat_modules:
       - system
     beats_fields:
       - "testbed: molecule"
+    kibana_extra_config: |-
+      ops.interval: 5000
   tasks:
     - name: Enable Elastic installation on RHEL 9
       ansible.builtin.set_fact:
         elasticstack_rpm_workaround: true
       when: ansible_os_family == 'RedHat' and ansible_distribution_major_version >= "9"
-    - name: Update apt cache.
-      ansible.builtin.apt:
-        update_cache: yes
-        cache_valid_time: 600
-      changed_when: false
-      when: ansible_os_family == 'Debian'
-    - name: Install dependencies
-      ansible.builtin.package:
-        name:
-          - curl
     - name: Include Redis
       ansible.builtin.include_role:
         name: geerlingguy.redis
@@ -50,12 +42,20 @@
     - name: Include logstash
       ansible.builtin.include_role:
         name: logstash
+    - name: Include kibana
+      ansible.builtin.include_role:
+        name: kibana
     - name: Include Beats
       ansible.builtin.include_role:
         name: beats
     - name: Install rsyslog
       ansible.builtin.package:
         name: rsyslog
+    - name: Remove cache # noqa: risky-shell-pipe
+      ansible.builtin.shell: >
+        if test -n "$(ps -p $$ | grep bash)"; then set -o pipefail; fi;
+        rm -rf /var/cache/*
+      changed_when: false
     - name: Configure rsyslog
       ansible.builtin.lineinfile:
         line: "*.* @@localhost:514"
@@ -64,9 +64,3 @@
       ansible.builtin.service:
         name: rsyslog
         state: started
-    - name: Include kibana
-      ansible.builtin.include_role:
-        name: kibana
-    - name: Include Beats
-      ansible.builtin.include_role:
-        name: beats

molecule/elasticstack_default/molecule.yml

@@ -1,10 +1,12 @@
 ---
 dependency:
   name: galaxy
+  options:
+    requirements-file: requirements.yml
 driver:
   name: docker
 platforms:
-  - name: elasticstack-cluster1
+  - name: "elasticstack${ELASTIC_RELEASE}-cluster1-${MOLECULE_DISTRO}"
     groups:
       - beats
       - logstash
@@ -17,7 +19,7 @@ platforms:
     cgroupns_mode: host
     privileged: true
     pre_build_image: true
-  - name: elasticstack-cluster2
+  - name: "elasticstack${ELASTIC_RELEASE}-cluster2-${MOLECULE_DISTRO}"
     groups:
       - beats
       - logstash

molecule/kibana_default/converge.yml

@@ -8,6 +8,7 @@
   vars:
     elasticstack_full_stack: false
     elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
+    elasticstack_no_log: false
   collections:
     - netways.elasticstack
   tasks:

molecule/logstash_full_stack-oss/converge.yml

@@ -23,6 +23,7 @@
     beats_filebeat_syslog_tcp: true
     logstash_beats_tls: false
     elasticstack_release: 7
+    elasticstack_no_log: false
   tasks:
     - name: "Include Elastics repos role"
       ansible.builtin.include_role:

molecule/logstash_full_stack-oss/molecule.yml

@@ -1,6 +1,8 @@
 ---
 dependency:
   name: galaxy
+  options:
+    requirements-file: requirements.yml
 driver:
   name: docker
 platforms:

molecule/logstash_pipelines/converge.yml

@@ -32,6 +32,7 @@
     logstash_pipeline_unsafe_shutdown: true
     elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
     elasticstack_full_stack: false
+    elasticstack_no_log: false
   tasks:
     - name: "Include Elastics repos role"
       ansible.builtin.include_role:

molecule/logstash_pipelines/molecule.yml

@@ -1,6 +1,8 @@
 ---
 dependency:
   name: galaxy
+  options:
+    requirements-file: requirements.yml
 driver:
   name: docker
 platforms:

molecule/logstash_specific_version/converge.yml

@@ -15,6 +15,7 @@
     logstash_pipeline_unsafe_shutdown: true
     elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
     elasticstack_full_stack: false
+    elasticstack_no_log: false
   tasks:
 
     - name: Set Filebeat version for 7.x

molecule/logstash_specific_version/molecule.yml

@@ -1,6 +1,8 @@
 ---
 dependency:
   name: galaxy
+  options:
+    requirements-file: requirements.yml
 driver:
   name: docker
 platforms:

molecule/repos_default/converge.yml

@@ -7,6 +7,7 @@
     elasticstack_rpm_workaround: true
     elasticstack_full_stack: false
     elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
+    elasticstack_no_log: false
   tasks:
     - name: Include Elastic Repos
       ansible.builtin.include_role:

molecule/repos_default/molecule.yml

@@ -1,6 +1,8 @@
 ---
 dependency:
   name: galaxy
+  options:
+    requirements-file: requirements.yml
 driver:
   name: docker
 platforms: