Skip to content

Commit

Permalink
Add Kafka log4j template configuration
Browse files Browse the repository at this point in the history
The Kafka log4j configuration is supplied as a template that can be used to update this configuration.

This provides the means to change logging levels, and other use cases such as changing the logging appender used to be able to roll over logs based on file size.

Additional changes:

* Update to Kafka 2.8.1
* Remove TravisCI integration and tests, this has been superseded by Molecule
  • Loading branch information
fidanf authored Oct 30, 2021
1 parent 3027475 commit 92c7187
Show file tree
Hide file tree
Showing 18 changed files with 208 additions and 312 deletions.
1 change: 0 additions & 1 deletion .ansible-lint
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,3 @@ exclude_paths:

skip_list:
- '106' # Role name does not match ``^[a-z][a-z0-9_]+$`` pattern

28 changes: 0 additions & 28 deletions .github/workflows/ansible-lint.yaml

This file was deleted.

115 changes: 0 additions & 115 deletions .travis.yml

This file was deleted.

7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
[![Build Status]](https://travis-ci.org/sleighzy/ansible-kafka)
![Lint Code Base] ![Ansible Lint] ![Molecule]

Ansible role to install and configure [Apache Kafka] 2.8.0
Ansible role to install and configure [Apache Kafka] 2.8.1

[Apache Kafka] is a distributed event streaming platform using publish-subscribe
topics. Applications and streaming components can produce and consume messages
Expand Down Expand Up @@ -45,7 +45,7 @@ See <https://github.com/ansible/ansible/issues/71528> for more information.
| Variable | Default |
| ---------------------------------------------- | ------------------------------------- |
| kafka_download_base_url | <http://www-eu.apache.org/dist/kafka> |
| kafka_version | 2.8.0 |
| kafka_version | 2.8.1 |
| kafka_scala_version | 2.13 |
| kafka_create_user_group | true |
| kafka_user | kafka |
Expand Down Expand Up @@ -85,6 +85,9 @@ See <https://github.com/ansible/ansible/issues/71528> for more information.
| kafka_bootstrap_servers | localhost:9092 |
| kafka_consumer_group_id | kafka-consumer-group |

See [log4j.yml](./defaults/main/002-log4j.yml) for detailled
log4j-related available variables.

## Starting and Stopping Kafka services using systemd

- The Kafka service can be started via: `systemctl start kafka`
Expand Down
2 changes: 1 addition & 1 deletion defaults/main.yaml → defaults/main/001-kafka.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# The Apache Kafka version to be downloaded and installed
# kafka_download_base_url should be set to https://archive.apache.org/dist/kafka/ for older versions than the current
kafka_download_base_url: http://www-eu.apache.org/dist/kafka
kafka_version: 2.8.0
kafka_version: 2.8.1
kafka_scala_version: 2.13

# The kafka user and group to create files/dirs with and for running the kafka service
Expand Down
74 changes: 74 additions & 0 deletions defaults/main/002-log4j.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
---
log4j_rootlogger: "INFO, stdout, kafkaAppender"

log4j_appender_stdout: "org.apache.log4j.ConsoleAppender"
log4j_appender_stdout_layout: "org.apache.log4j.PatternLayout"
log4j_appender_stdout_layout_conversionpattern: "[%d] %p %m (%c)%n"

log4j_appender_kafkaappender: "org.apache.log4j.DailyRollingFileAppender"
log4j_appender_kafkaappender_datepattern: "'.'yyyy-MM-dd-HH"
log4j_appender_kafkaappender_file: "${kafka.logs.dir}/server.log"
log4j_appender_kafkaappender_layout: "org.apache.log4j.PatternLayout"
log4j_appender_kafkaappender_layout_conversionpattern: "[%d] %p %m (%c)%n"

log4j_appender_statechangeappender: "org.apache.log4j.DailyRollingFileAppender"
log4j_appender_statechangeappender_datepattern: "'.'yyyy-MM-dd-HH"
log4j_appender_statechangeappender_file: "${kafka.logs.dir}/state-change.log"
log4j_appender_statechangeappender_layout: "org.apache.log4j.PatternLayout"
log4j_appender_statechangeappender_layout_conversionpattern: "[%d] %p %m (%c)%n"

log4j_appender_requestappender: "org.apache.log4j.DailyRollingFileAppender"
log4j_appender_requestappender_datepattern: "'.'yyyy-MM-dd-HH"
log4j_appender_requestappender_file: "${kafka.logs.dir}/kafka-request.log"
log4j_appender_requestappender_layout: "org.apache.log4j.PatternLayout"
log4j_appender_requestappender_layout_conversionpattern: "[%d] %p %m (%c)%n"

log4j_appender_cleanerappender: "org.apache.log4j.DailyRollingFileAppender"
log4j_appender_cleanerappender_datepattern: "'.'yyyy-MM-dd-HH"
log4j_appender_cleanerappender_file: "${kafka.logs.dir}/log-cleaner.log"
log4j_appender_cleanerappender_layout: "org.apache.log4j.PatternLayout"
log4j_appender_cleanerappender_layout_conversionpattern: "[%d] %p %m (%c)%n"

log4j_appender_controllerappender: "org.apache.log4j.DailyRollingFileAppender"
log4j_appender_controllerappender_datepattern: "'.'yyyy-MM-dd-HH"
log4j_appender_controllerappender_file: "${kafka.logs.dir}/controller.log"
log4j_appender_controllerappender_layout: "org.apache.log4j.PatternLayout"
log4j_appender_controllerappender_layout_conversionpattern: "[%d] %p %m (%c)%n"

log4j_appender_authorizerappender: "org.apache.log4j.DailyRollingFileAppender"
log4j_appender_authorizerappender_datepattern: "'.'yyyy-MM-dd-HH"
log4j_appender_authorizerappender_file: "${kafka.logs.dir}/kafka-authorizer.log"
log4j_appender_authorizerappender_layout: "org.apache.log4j.PatternLayout"
log4j_appender_authorizerappender_layout_conversionpattern: "[%d] %p %m (%c)%n"

# Change the line below to adjust ZK client logging
log4j_logger_org_apache_zookeeper: "INFO"

# Change the two lines below to adjust the general broker logging level (output to server.log and stdout)
log4j_logger_kafka: "INFO"
log4j_logger_org_apache_kafka: "INFO"

# Change to DEBUG or TRACE to enable request logging
log4j_logger_kafka_request_logger: "WARN, requestAppender"
log4j_additivity_kafka_request_logger: false

# Uncomment the lines below and change log4j_logger_kafka_network_requestchannel$ to TRACE for additional output
# related to the handling of requests
#log4j_logger_kafka_network_processor: TRACE, requestAppender
#log4j_logger_kafka_server_kafkaapis: TRACE, requestAppender
#log4j_additivity_kafka_server_kafkaapis: false
log4j_logger_kafka_network_requestchannel: "WARN, requestAppender"
log4j_additivity_kafka_network_requestchannel: false

log4j_logger_kafka_controller: "TRACE, controllerAppender"
log4j_additivity_kafka_controller: false

log4j_logger_kafka_log_logcleaner: "INFO, cleanerAppender"
log4j_additivity_kafka_log_logcleaner: false

log4j_logger_state_change_logger: "INFO, stateChangeAppender"
log4j_additivity_state_change_logger: false

# Access denials are logged at INFO level, change to DEBUG to also log allowed accesses
log4j_logger_kafka_authorizer_logger: "INFO, authorizerAppender"
log4j_additivity_kafka_authorizer_logger: false
2 changes: 2 additions & 0 deletions meta/main.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
galaxy_info:
author: Simon Leigh
namespace: sleighzy
role_name: kafka
description: Apache Kafka installation for RHEL/CentOS and Debian/Ubuntu
license: MIT
min_ansible_version: 2.10.4
Expand Down
22 changes: 0 additions & 22 deletions molecule/default/Dockerfile.j2

This file was deleted.

22 changes: 0 additions & 22 deletions molecule/default/INSTALL.rst

This file was deleted.

2 changes: 1 addition & 1 deletion molecule/default/molecule.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ provisioner:
lint: |
set -e
yamllint -c ./.yamllint.yaml .
ansible-lint -c ./.ansible-lint .
ansible-lint -c ./.ansible-lint
verifier:
name: ansible
scenario:
Expand Down
3 changes: 2 additions & 1 deletion molecule/default/requirements.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
roles:
- sleighzy.zookeeper

collections:
- community.docker
- community.docker
8 changes: 4 additions & 4 deletions molecule/default/verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@
- "'kafka' in getent_passwd"
- "'kafka' in getent_group"

- name: Register '/opt/kafka_2.13-2.8.0' installation directory status
- name: Register '/opt/kafka_2.13-2.8.1' installation directory status
stat:
path: '/opt/kafka_2.13-2.8.0'
path: '/opt/kafka_2.13-2.8.1'
register: install_dir

- name: Assert that '/opt/kafka_2.13-2.8.0' directory is created
- name: Assert that '/opt/kafka_2.13-2.8.1' directory is created
assert:
that:
- install_dir.stat.exists
Expand All @@ -39,7 +39,7 @@
that:
- kafka_dir.stat.exists
- kafka_dir.stat.islnk
- kafka_dir.stat.lnk_target == '/opt/kafka_2.13-2.8.0'
- kafka_dir.stat.lnk_target == '/opt/kafka_2.13-2.8.1'

- name: Register '/var/log/kafka' directory status
stat:
Expand Down
20 changes: 20 additions & 0 deletions tasks/main.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,16 @@
tags:
- kafka_config

- name: Template configuration file to log4j.properties
template:
src: log4j.properties.j2
dest: '{{ kafka_dir }}/config/log4j.properties'
group: '{{ kafka_group }}'
owner: '{{ kafka_user }}'
mode: 0644
tags:
- kafka_config

- name: Create symlink to kafka server properties file
file:
src: '{{ kafka_dir }}/config/server.properties'
Expand Down Expand Up @@ -222,6 +232,16 @@
tags:
- kafka_config

- name: Create symlink to kafka log4j properties file
file:
src: '{{ kafka_dir }}/config/log4j.properties'
dest: /etc/kafka/log4j.properties
state: link
group: '{{ kafka_group }}'
owner: '{{ kafka_user }}'
tags:
- kafka_config

- name: Template Kafka init.d service file
template:
src: kafka.initd.j2
Expand Down
Loading

0 comments on commit 92c7187

Please sign in to comment.