Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add !mermaid command #32

Merged
merged 2 commits into from
Jun 21, 2023
Merged

Add !mermaid command #32

merged 2 commits into from
Jun 21, 2023

Conversation

ryannedolan
Copy link
Collaborator

Added a new CLI command !mermaid which renders pipelines as mermaid flowcharts. In the resulting graph, resources are grouped by "kind".

Added a builder DSL to ResourceProvider to enable expressing source->sink relationships between resources. These relationships are not meaningful to the planner or operator.

Testing

The existing adapters are too simple to adequately test this against, so I modified the Kafka adapter to include additional "hops" (not included in the PR):

ResourceProvider.empty()
  .with(x -> new MySqlTable(x))
  .to(x -> new BrooklinCDC(x))
  .to(x -> new KafkaTopic(x, numPartitions, topicConfigProvider.config(x)))

Using this somewhat contrived adapter, I tested a simple query across two tables:

0: hoptimator> !mermaid SELECT * FROM RAWKAFKA."products", RAWKAFKA."test-topic"

... which produces the following flowchart:

flowchart
  subgraph SqlJob
  R663951423["sql: CREATE DATABASE I..."]
  end
  subgraph BrooklinCDC
  R1206753277["name: products"]
  R-354296883["name: test-topic"]
  end
  subgraph MySqlTable
  R-623545925["name: products"]
  R1664761355["name: test-topic"]
  end
  subgraph KafkaTopic
  R-41078963["clientOverrides: group.id: hoptima...
name: products
numPartitions: null"]
  R1728328189["clientOverrides: group.id: hoptima...
name: test-topic
numPartitions: null"]
  end
  subgraph SqlJob
    R-41078963 --> R663951423
    R1728328189 --> R663951423
  end
  subgraph BrooklinCDC
    R-623545925 --> R1206753277
    R1664761355 --> R-354296883
  end
  subgraph MySqlTable
  end
  subgraph KafkaTopic
    R1206753277 --> R-41078963
    R-354296883 --> R1728328189
  end
Loading

@ryannedolan ryannedolan enabled auto-merge (squash) June 12, 2023 15:32
Copy link
Collaborator

@vmaheshw vmaheshw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the mermaid drawing, at the datastream level, it uses the name of the MYSQL table. Does this mean the name of the data stream created and the MYSQL table is the same? Since, auto-generation is involved, it will probably make more sense to prefix the name with the object type.

Similarly, it shows overrides at the next step. What does it mean and how is it useful? Is it showing the properties of the object at that level?

Lastly, it shows SQL job, rather than Flink SQL job. It is important since the planner is specifically generating a FlinkSQL job.

@ryannedolan ryannedolan merged commit 0285d7b into main Jun 21, 2023
@ryannedolan ryannedolan deleted the mermaid branch June 21, 2023 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants