-
Notifications
You must be signed in to change notification settings - Fork 831
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create wrappers for Java based models #137
Comments
Merged
agrski
added a commit
that referenced
this issue
Dec 2, 2022
* Remove Kafka Protobuf Serdes dependency as unused * Add retry library for pipeline subscriptions, etc. * Add retry policy for pipeline subscription attempts * Formatting * Refactor Kafka admin client to instance variable * Make pipeline creation method more idiomatic Avoid mutable state. Improve formatting. Rename variables for clarity and to avoid shadowing. * Rename method for clarity * Refactor Kafka admin-client interactions to new class This avoids leaking Kafka-specific knowledge into what is essentially a gRPC-wrapper class. It also avoids scope creep and increasing file size for the pipeline subscriber class. * Formatting * Check pipeline steps are valid before creating Kafka topics (if not already extant) Interactions with Kafka are out-of-process, hence slower and less reliable. We may also decide there is nothing to do, if the pipeline is not valid. * Refactor Kafka topic creation into a single pipeline of transformations * Await creation of Kafka topics in parallel * Refactor Kafka result waits to private method * Formatting * Refactor Joiner serde definitions to companion object These are stateless - essentially type definitions - so there is no need to redefine them for evey instance. * Use when statement instead of if-else for clarity * Formatting * Replace call to deprecated Kafka API The existing 'of' call has been deprecated. As we already allow a significant join window, this does not add a wall-clock grace period. * Refactor pipeline header filtering to extension method on KStreams * Refactor inference v2 type conversion to extension method for chainer * Replace inference v2 conversion logic in joiner with new extension method * Add CLI options for Kafka partition count & replication factor These will be used when creating topics that do not yet exist. * Rename CLI var for consistency & group CLI vars * Add CLI var for use of clean Kafka state on startup * Add local properties for new Kafka CLI vars * Formatting * Refactor inference v2 (un)marshalling to KStream extension methods * Remove type constraint on KStream key type in extension method * Remove TODO as only intend to support inference v2 internally Internally means within the context of a pipeline. * Remove unused import * Add tail-recursive version of joiner logic * Replace recursive joiner builder with tail rec equivalent * Make bootstrapping param to method nullable This simplifies the API to only provide pending joins when they exist, making the caller's task easier to read and understand. * Rename private method for clarity * Fix bug using wrong var in joiner topology construction * Remove redundant non-null filtering in joiner * Rename data class to disambiguate from upcoming new class * Add CLI var for KStream join window duration * Refactor domain params for KStreams into data class Rather than passing an increasingly large number of configuration options, it is cleaner to pass a single data class which wraps up that set of options/config. * Rename var for clarity & consistency * Refactor Kafka (Streams) type aliases & serdes to separate file These definitions were scattered around and duplicated in a few places. * Formatting & removal of unused imports * Remove done TODOs * Rename method & vars for clarity The method name was noun-like rather than verb-like, which is counterintuitive. The variable names were cryptic. * Formatting for consistency * Fix broken tests by adding Kafka domain params instance * Formatting * Remove stale comment * Formatting + fix incorrect type alias * Remove redundant parameter from source projection subtype * Formatting * Add comparison logic for joiners in transformer tests * Fix joiner cases in transformer tests + rename param for clarity * Rename param for clarity * remake operator to get resource yaml changes Co-authored-by: Clive Cox <cc@seldon.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
To cover
The text was updated successfully, but these errors were encountered: