Skip to content

Commit

Permalink
Add support for Sentry monitoring
Browse files Browse the repository at this point in the history
Includes addition to documentation.
  • Loading branch information
pvannierop committed Dec 5, 2024
1 parent 701409c commit ec081cb
Show file tree
Hide file tree
Showing 9 changed files with 154 additions and 9 deletions.
55 changes: 47 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,19 @@
# Kafka Connect REST Source and Fitbit Source

This project contains a Kafka Connect source connector for a general REST API, and one for
Fitbit in particular. The documentation of the Kafka Connect REST source still needs to be done.
This project contains a Kafka Connect source connector for a general REST API, for
specific Fitbit and Oura devices. The documentation of the Kafka Connect REST source still needs to
be done.

<!-- TOC -->

* [Kafka Connect REST Source and Fitbit Source](#kafka-connect-rest-source-and-fitbit-source)
* [Fitbit source connector](#fitbit-source-connector)
* [Installation](#installation)
* [Usage](#usage)
* [Sentry monitoring](#sentry-monitoring)
* [Contributing](#contributing)

<!-- TOC -->

## Fitbit source connector

Expand All @@ -12,7 +24,9 @@ of Java 17 or later.

### Usage

Generally, this component is installed with [RADAR-Kubernetes](https://github.com/RADAR-base/RADAR-Kubernetes). It uses Docker image [radarbase/kafka-connect-rest-fitbit-source](https://hub.docker.com/r/radarbase/kafka-connect-rest-fitbit-source).
Generally, this component is installed
with [RADAR-Kubernetes](https://github.com/RADAR-base/RADAR-Kubernetes). It uses Docker
image [radarbase/kafka-connect-rest-fitbit-source](https://hub.docker.com/r/radarbase/kafka-connect-rest-fitbit-source).

First, [register a Fitbit App](https://dev.fitbit.com/apps) with Fitbit. It should be either a
server app, for multiple users, or a personal app for a single user. With the server app, you need
Expand All @@ -22,7 +36,8 @@ For every Fitbit user you want access to, copy `docker/fitbit-user.yml.template`
`docker/users/`. Get an access token and refresh token for the user using for example the
[Fitbit OAuth 2.0 tutorial page](https://dev.fitbit.com/apps/oauthinteractivetutorial).

For automatic configuration for multiple users, please take a look at `scripts/REDCAP-FITBIT-AUTH-AUTO/README.md`.
For automatic configuration for multiple users, please take a look at
`scripts/REDCAP-FITBIT-AUTH-AUTO/README.md`.

Copy `docker/source-fitbit.properties.template` to `docker/source-fitbit.properties` and enter
your Fitbit App client ID and client secret. The following tables shows the possible properties.
Expand Down Expand Up @@ -94,7 +109,8 @@ your Fitbit App client ID and client secret. The following tables shows the poss
<td>fitbit.user.firebase.collection.user.name</td><td>Firestore Collection for retrieving User details. Only used when a Firebase based user repository is used.</td><td>string</td><td>users</td><td></td><td>low</td></tr>
</tbody></table>

If the ManagementPortal is used to authenticate against the user repository, please add an OAuth client to ManagementPortal with the following properties:
If the ManagementPortal is used to authenticate against the user repository, please add an OAuth
client to ManagementPortal with the following properties:

```
Client ID: fitbit.user.repository.client.id
Expand All @@ -106,7 +122,8 @@ Access Token validity: 600
Refresh Token validity: 0
```

Finally set the `fitbit.user.repository.oauth.token.url` to `http://managementportal-app:8080/managementportal/oauth/token`.
Finally set the `fitbit.user.repository.oauth.token.url` to
`http://managementportal-app:8080/managementportal/oauth/token`.

Now you can run a full Kafka stack using

Expand Down Expand Up @@ -163,7 +180,29 @@ sequenceDiagram
connector ->> connector: Update offset times
```

## Sentry monitoring

To enable Sentry monitoring for the generic REST, Fitbit, or Oura source connector service:

1. Set a `SENTRY_DSN` environment variable that points to the desired Sentry DSN.
2. (Optional) Set the `SENTRY_LOG_LEVEL` environment variable to control the minimum log level of
events sent to Sentry.
The default log level for Sentry is `WARN`. Possible values are `TRACE`, `DEBUG`, `INFO`, `WARN`,
and `ERROR`.

For further configuration of Sentry via environmental variables see [here](https://docs.sentry.io/platforms/java/configuration/#configuration-via-the-runtime-environment). For instance:

```
SENTRY_LOG_LEVEL: 'ERROR'
SENTRY_DSN: 'https://000000000000.ingest.de.sentry.io/000000000000'
SENTRY_ATTACHSTACKTRACE: true
SENTRY_STACKTRACE_APP_PACKAGES: io.confluent.connect,org.radarbase.connect.rest
```

## Contributing

Code should be formatted using the [Google Java Code Style Guide](https://google.github.io/styleguide/javaguide.html).
If you want to contribute a feature or fix browse our [issues](https://github.com/RADAR-base/RADAR-REST-Connector/issues), and please make a pull request.
Code should be formatted using
the [Google Java Code Style Guide](https://google.github.io/styleguide/javaguide.html).
If you want to contribute a feature or fix browse
our [issues](https://github.com/RADAR-base/RADAR-REST-Connector/issues), and please make a pull
request.
3 changes: 2 additions & 1 deletion buildSrc/src/main/kotlin/Versions.kt
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ object Versions {
const val kotlin = "1.9.22"
const val wrapper = "8.4"

const val radarCommons = "1.1.2"
const val radarCommons = "1.1.3"
const val confluent = "7.7.0"
const val kafka = "$confluent-ce"
const val avro = "1.12.0"
Expand All @@ -18,6 +18,7 @@ object Versions {

const val log4j2 = "2.23.1"
const val slf4j = "2.0.13"
const val sentryLog4j = "1.7.30"

const val okhttp = "4.12.0"

Expand Down
8 changes: 8 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,10 @@ services:
KAFKA_HEAP_OPTS: "-Xms256m -Xmx768m"
KAFKA_BROKERS: 3
CONNECT_LOG4J_LOGGERS: "org.reflections=ERROR"
# SENTRY_LOG_LEVEL: 'ERROR'
# SENTRY_DSN: 'https://000000000000.ingest.de.sentry.io/000000000000'
# SENTRY_ATTACHSTACKTRACE: true
# SENTRY_STACKTRACE_APP_PACKAGES: io.confluent.connect,org.radarbase.connect.rest

#---------------------------------------------------------------------------#
# RADAR Oura connector #
Expand Down Expand Up @@ -221,3 +225,7 @@ services:
KAFKA_HEAP_OPTS: "-Xms256m -Xmx768m"
KAFKA_BROKERS: 3
CONNECT_LOG4J_LOGGERS: "org.reflections=ERROR"
# SENTRY_LOG_LEVEL: 'ERROR'
# SENTRY_DSN: 'https://000000000000.ingest.de.sentry.io/000000000000'
# SENTRY_ATTACHSTACKTRACE: true
# SENTRY_STACKTRACE_APP_PACKAGES: io.confluent.connect,org.radarbase.connect.rest
32 changes: 32 additions & 0 deletions docker/log4j.properties.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# This template file was taken from the Confluent Platform distribution and modified to add Sentry support in Docker images.
# See: https://docs.confluent.io/platform/current/installation/docker/development.html#log-to-external-volumes

log4j.rootLogger={{ env["CONNECT_LOG4J_ROOT_LOGLEVEL"] | default('INFO') }}, stdout{% if env['SENTRY_DSN'] %}, sentryAppender{% endif %}


log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern ={{ env["CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN"] | default('[%d] %p %m (%c)%n') }}

# Appender for Sentry monitoring
{% if env['SENTRY_DSN'] %}
log4j.appender.sentryAppender=io.sentry.log4j.SentryAppender
log4j.appender.sentryAppender.threshold={{ env['SENTRY_LOG_LEVEL'] | default('ERROR') }}
{% endif %}

{% set default_loggers = {
'org.reflections': 'ERROR',
'org.apache.zookeeper': 'ERROR',
'org.I0Itec.zkclient': 'ERROR'
} -%}

{% if env['CONNECT_LOG4J_LOGGERS'] %}
# loggers from CONNECT_LOG4J_LOGGERS env variable
{% set loggers = parse_log4j_loggers(env['CONNECT_LOG4J_LOGGERS']) %}
{% else %}
# default log levels
{% set loggers = default_loggers %}
{% endif %}
{% for logger,loglevel in loggers.items() %}
log4j.logger.{{logger}}={{loglevel}}
{% endfor %}
6 changes: 6 additions & 0 deletions kafka-connect-fitbit-source/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,9 @@ COPY --chown=appuser:appuser ./docker/kafka-wait /usr/bin/kafka-wait

# Load modified launcher
COPY --chown=appuser:appuser ./docker/launch /etc/confluent/docker/launch

# Overwrite the log4j configuration to include Sentry monitoring.
COPY ./docker/log4j.properties.template /etc/confluent/docker/log4j.properties.template
# Copy Sentry monitoring jars.
COPY --from=builder /code/kafka-connect-fitbit-source/build/third-party/sentry-* /etc/kafka-connect/jars

7 changes: 7 additions & 0 deletions kafka-connect-fitbit-source/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,12 @@ dependencies {
compileOnly(platform("com.fasterxml.jackson:jackson-bom:${Versions.jackson}"))
compileOnly("com.fasterxml.jackson.core:jackson-databind")

// Application monitoring
// This dependency is not used by the REST connector, but copied into the Docker image (Dockerfile)
compileOnly("io.sentry:sentry-log4j:${Versions.sentryLog4j}") {
// Exclude log4j with security vulnerability (safe version is provided by docker image).
exclude(group = "log4j", module = "log4j")
}

testImplementation("org.apache.kafka:connect-api:${Versions.kafka}")
}
6 changes: 6 additions & 0 deletions kafka-connect-oura-source/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,9 @@ COPY --chown=appuser:appuser ./docker/kafka-wait /usr/bin/kafka-wait

# Load modified launcher
COPY --chown=appuser:appuser ./docker/launch /etc/confluent/docker/launch

# Overwrite the log4j configuration to include Sentry monitoring.
COPY ./docker/log4j.properties.template /etc/confluent/docker/log4j.properties.template
# Copy Sentry monitoring jars.
COPY --from=builder /code/kafka-connect-oura-source/build/third-party/sentry-* /etc/kafka-connect/jars

7 changes: 7 additions & 0 deletions kafka-connect-oura-source/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,12 @@ dependencies {
compileOnly(platform("com.fasterxml.jackson:jackson-bom:${Versions.jackson}"))
compileOnly("com.fasterxml.jackson.core:jackson-databind")

// Application monitoring
// This dependency is not used by the REST connector, but copied into the Docker image (Dockerfile)
compileOnly("io.sentry:sentry-log4j:${Versions.sentryLog4j}") {
// Exclude log4j with security vulnerability (safe version is provided by docker image).
exclude(group = "log4j", module = "log4j")
}

testImplementation("org.apache.kafka:connect-api:${Versions.kafka}")
}
39 changes: 39 additions & 0 deletions oura-library/src/main/resources/log4j2.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
<?xml version="1.0" encoding="UTF-8" ?>
<!--
~ /*
~ * Copyright 2024 The Hyve
~ *
~ * Licensed under the Apache License, Version 2.0 (the "License");
~ * you may not use this file except in compliance with the License.
~ * You may obtain a copy of the License at
~ *
~ * http://www.apache.org/licenses/LICENSE-2.0
~ *
~ * Unless required by applicable law or agreed to in writing, software
~ * distributed under the License is distributed on an "AS IS" BASIS,
~ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ * See the License for the specific language governing permissions and
~ * limitations under the License.
~ */
-->

<configuration status="INFO">
<appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout
pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"
/>
</Console>
<!-- For Sentry to work the DSN must be set via SENTRY_DSN environment variable
When SENTRY_DSN is empty string, the Sentry SDK is disabled -->
<Sentry name="Sentry" debug="false"/>
</appenders>

<loggers>
<root level="INFO">
<appender-ref ref="Console" />
<!-- Note that the Sentry logging threshold is at WARN level by default -->
<appender-ref ref="Sentry" level="${env:SENTRY_LOG_LEVEL:-WARN}" />
</root>
</loggers>
</configuration>

0 comments on commit ec081cb

Please sign in to comment.