Skip to content

Commit

Permalink
Added information about kafka log retention policies (#2052)
Browse files Browse the repository at this point in the history
* Added info about kafka log retention policies

* Update docs/home/howto/RETENTION.md

Co-authored-by: to-bar <46519524+to-bar@users.noreply.github.com>

* Update docs/home/howto/RETENTION.md

Co-authored-by: to-bar <46519524+to-bar@users.noreply.github.com>

* Update docs/home/howto/RETENTION.md

Co-authored-by: to-bar <46519524+to-bar@users.noreply.github.com>

Co-authored-by: atsikham <atsikham@users.noreply.github.com>
Co-authored-by: to-bar <46519524+to-bar@users.noreply.github.com>
  • Loading branch information
3 people committed Feb 12, 2021
1 parent ee5d303 commit 89f0b6d
Showing 1 changed file with 40 additions and 2 deletions.
42 changes: 40 additions & 2 deletions docs/home/howto/RETENTION.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
An Epiphany cluster has a number of components which log, collect and retain data. To make sure that these do not exceed the usable storage of the machines there running on the following configurations are available.
An Epiphany cluster has a number of components which log, collect and retain data. To make sure that these do not exceed
the usable storage of the machines they running on, the following configurations are available.

## Elasticsearch

Expand All @@ -10,7 +11,44 @@ TODO

## Kafka

TODO
There are two types of retention policies that can be configured at the broker or topic levels: based on time or size.
Epiphany defines the same default value for broker size retention policy as Kafka, -1, which means that no
size limit is applied.

To define new log retention values following configuration can be used:

```yaml
kind: configuration/kafka
title: "Kafka"
name: default
specification:
kafka_var:
partitions: 8
log_retention_hours: 168
log_retention_bytes: -1
```
### Configuration parameters
#### specification.kafka_var.partitions
Sets [num.partitions](https://kafka.apache.org/documentation/#brokerconfigs_num.partitions) parameter
#### specification.kafka_var.log_retention_hours
Sets [log.retention.hours](https://kafka.apache.org/documentation/#brokerconfigs_log.retention.bytes) parameter
#### specification.kafka_var.log_retention_bytes
Sets [log.retention.bytes](https://kafka.apache.org/documentation/#brokerconfigs_log.retention.bytes) parameter
---
**NOTE**
Since this limit is enforced at the partition level, multiply it by the number of partitions to compute the topic
retention in bytes.
---
## Kibana
Expand Down

0 comments on commit 89f0b6d

Please sign in to comment.