Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce debug exporter to stream records out of a broker #289

Merged
merged 14 commits into from
May 27, 2022

Conversation

npepinpe
Copy link
Collaborator

Description

This PR introduces a new DebugExporter and its paired DebugReceiver to stream records out of a broker. The exporter is focused on correctness and simplicity, not performance.

As mentioned, it's made of two components, an exporter and a receiver. The receiver is an HTTP 1.1 server with a single endpoint, POST /records. The endpoint expects a list of records (for forwards-compatibility), all of which it assumes come from the same partition. It will then answer with a 200 and the highest known acknowledged position for that partition in the form of { "position": long }, or simply a 204 and no body if there is no known acknowledged position.

To auto acknowledge, you could simply immediately acknowledge Long.MAX_VALUE for all known partitions before starting your test.

The exporter will export each record as a single POST. This is inefficient, but extremely simple and accurate, as any failure immediately causes the record to be retried.

At the moment, the exporter is dynamically packaged as a standalone JAR and copied over to the container on demand. The alternative was packaging it separately and adding it to the resources, which is less than ideal (and also makes testing flakier, i.e. did you rebuild the JAR for the exporter based tests?). I will anyway propose to move the exporter to the Zeebe project itself, so I hope we can simply get rid of this limitation in the future.

Due to this, however, it means any work on the exporter should be done carefully. The class must remain standalone, and any custom types it requires must be inner classes (to simplify packaging logic).

Pull Request Checklist

  • All commit messages match our commit message guidelines
  • The submitting code follows our code style
  • If submitting code, please run mvn clean install -DskipTests locally before committing
  • Ensure all PR checks are green

@npepinpe npepinpe force-pushed the np-debug-exporter branch from 3ece542 to 7f5d745 Compare May 25, 2022 12:21
@npepinpe npepinpe force-pushed the np-debug-exporter branch from 74928da to 6596bc0 Compare May 25, 2022 14:48
@npepinpe npepinpe force-pushed the np-debug-exporter branch from 1091f44 to 34f3692 Compare May 25, 2022 15:40
@npepinpe npepinpe force-pushed the np-debug-exporter branch from ae019a2 to 9e760a0 Compare May 25, 2022 15:43
@npepinpe npepinpe merged commit 74a8e4e into main May 27, 2022
@npepinpe npepinpe deleted the np-debug-exporter branch May 27, 2022 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant